Interview with Stacy Konkiel about altmetrics

2018-04-18

Martin Giesler talked with Stacy Konkiel – developer of the metrics-toolkit and working at altmetric.com – about metrics in general, the perspectives of altmetrics and the activity of the *metrics-project.

Nowadays we have more information online than ever before. There is more news articles in the world, there is more basic information like what people are eating for breakfast and more information in general. The same goes for research and the sciences. There are a lot of peer-reviewed papers being published in the world – now more than ever before and therefore it can be harder to find out what research is relevant to you, e.g. if you are a researcher, if you want to know what other people in your field are researching, that could be one reason. Or if you are a member of the public and you want to find out about a certain disease and you need to get access to the most up-to-date high quality information out there is another reason. It is really important nowadays to go through all the information that is out there online.

So, traditionally in terms of measurement people used citation data. So when scientist or researchers would describe other studies that are happening in this space they would do so by looking for peer-reviewed papers. Traditionally people looked through those citations and they count those citations – the more citations there are the higher quality the work is. And there are some challenges with that approach. It is not only that the most cited stuff is the highest quality but more or less it tells you what you need to pay attention to. But before these metrics existed, people knew what to pay attention to by simply knowing what was available in their field or by knowing the people in your field. The academia used to be a small world. You simply knew what was happening. But for reasons we just described, that is not necessarily possible anymore. Because there is too much research out there. So people have to find signals like citations to figure out what is the highest quality or at least the most important, most influential research in a subject area.

There is a lot. And the number seems to be growing every day. Not only in terms of citation metrics. There is simple citation counts. But then there is also other types of derivative metrics like the journal impact factor, the eigenfactor – lots of different ways of interpretation of citation data to help you understand most influential journals, most influential papers. And beyond citations there are these signals that are left online, traces of interactions with research articles – we call these traces altmetrics. And it is tracking when somebody is tweeting about an article, if an article has been cited in wikipedia, if an articles has been mentioned in the news or in a public policy document – these are all ways to understand not only traditional scholarly influence of a piece of research but that also members of the public are talking about scientific research. Is it having cultural relevance? Is it important to patient advisory group in biomedicine? So there are a lot of different signals.

So there is not a standard and I don’t necessarily think that that is bad thing. I think people get kind of nervous when there is not something like „Here is how you do X“, some kind of authoritative way to measure something but I would point out that there are not necessarily standards in citation either. You have different databases. So popular databases to track citations include Google Scholar, there is Web of Science, there is Scopus, and each of these databases tracks the same basic thing, they track when research is cited in other research but they all approach it in a different way. So for Google Scholar that means they pick up a lot because they have much much stringent requirements on what a citation should look like in order to appear in Google Scholar and Web of Science they only include certain valid journals in their database. So you are going to get a smaller number of citations but in some ways you could argue higher quality citations. Similarly there are a few altmetrics aggregators out there right now. So I work for a company called altmetric.com and we are one of them. We track mentions of research across 17 different types of sources, social media, there is blogs, public policy, we now track patents, and the way we track it is slightly different from other aggregators out there - Plum Analytics is one, CrossRef is now another. We each have our own way of doing things, there is not necessarily a standard - I see we all try to obey by best practices. Those are tracking things that are somewhat audible. So if I tweet about a piece of research, for the most part, everyone who is an altmetrics aggregator, will only count that tweet if it can be assured to have existed in some way. What that means is either Twitter has to say, we know that a user called X exists and she tweeted about this or we as a general public have to be able to look back at the tweet to verify that it existed. This is necessary in order to keep people from gaming metrics, it is to make sure that the numbers of how many times something has been cited in the news or listed in Wikipedia that it actually existed because everytime you introduce any kind of a metric there can be an incentive to game it for some people. So basically we all strive to some sort of audibility and accountability for the data that we collect.

I think I can understand it to a certain extend. Social Media in general has only been around for ten or fifteen years. We are still trying to figure out what it means when somebody tweets about something, when somebody blogs about something. What are their motivations for doing so? And in the field of altmetrics there is scientists around the world and also humanity researchers who are trying to figure out what are those motivations for tweets and why do people even cite each other. This is still a topic of great debate. We are trying to understand the influence of research and we got these metrics that we can fall back on, what do they actually mean is important. And so I think it makes sense that there is some skepticism from scientists in particular, folks that used to understand for many, many years that there is a particular way that you understand the influence of research whether it is by peer-reviewing other people’s research, which is still the gold standard and I think it should remain that way, informed human judgement is always going to be the way. I think we should understand the quality of research. I think it makes sense. But on the other hand I would encourage those who are skeptical about altmetrics to maybe broaden their understanding of how we might understand influence. So it is not just about what is high quality, what is likely to make an impact in a field, research should also be about how people lifes are touched, whether someone becomes a new patreon of the arts because a particular piece of writing has really moved them that they read and so they decided to talk about it with their friends and get other people reading. Research touches peoples lifes on a daily basis. And altmetrics as it stands right now is the best data that we have available to understand those types of connections that are made between the general public and researchers. So I think it is not a zero sum game, it is not either peer-review or citations or altmetrics, and you cannot have any of the other. We should look at all of the information we now have available in order to find a greater texture to these stories that people tell about the impact of research.

I think the work you are doing with the metrics project is really important because it is helping people to understand all of the possible signals we have out there for understanding the influence of research and categorizing what they are and what they mean. This is an interesting field to be working in right now because there is a lot conjecture around what it means when someone e.g. cites a public policy. There is been fields of study to understand the dissemination of research and so-called knowledge mobilization and that has existed for many, many years and it is this intersection of philosophy and science and other aspects of the humanities. Where we need to come back together is the kind of the work the metrics project is doing - taking a step back and saying what is all the information available? How well situated is it in existing theory, in existing scientific research and then going from there.

You can follow Stacy Konkiel on Twitter or visit her personal website