Assistant Professor Ian Hutchins and PhD student Salsabil Arabi are researchers researching research. Yes, you read that right.
Hutchins and Arabi, members of the Metascience Research Lab at the iSchool, are engaged in the “science of science,” using large datasets and machine learning to reveal trends and expose inequities in the complex landscape of academic research. Hutchins and Arabi’s newest study challenges traditional notions of success in academia by scrutinizing how the impact of scientific publications is measured and why some exceptional work remains virtually “invisible,” as Hutchins said.
The story of their latest research begins more than a decade before Hutchins became an iSchool faculty member, when he was a biomedical researcher at the National Institutes of Health (NIH).
Impact factors as gatekeepers
As a neuroscientist at NIH, Hutchins noticed that in the biomedical realm, decisions about hiring and promotion often came down to which candidates had published in so-called “high-impact journals” such as Science, Cell, or The New England Journal of Medicine. These journals are set apart by a high “impact factor,” a single number measuring how often an average paper from that journal is cited during a two-year period.
Impact factors can vary widely. To offer a few examples, Science boasts an impact factor of 47.7 and Nature Immunology has a 25.6, while the impact factor of Annals of Medicine is 4.4. Hutchins and colleagues consider any journal with an impact factor over 15 to be a “high-impact” journal.
When institutions hire tenure-track faculty and research scientists in biomedicine, impact factors loom large, Hutchins said: “They narrow it down from a long list of candidates to a shorter one by looking only at people who have published in a journal with an impact factor above a certain number.” As a result, otherwise qualified candidates for roles as research scientists and tenure-track faculty members are overlooked solely because their work has not been published in a few extremely selective academic journals. This raises larger questions about the role of journal impact factors in assessing research and the people who do it.
“I think that the value of research is really multifaceted,” Hutchins said. “There are many things that an article can do to advance the frontier of science,” he added, even without being published in a journal with a high impact factor. For instance, a study may end up leading to a new patent for a medical technology that directly benefits human health – but impact factors don’t capture outcomes like patents.
Trailblazing research often overlooked
In a preprint paper currently under peer review, Hutchins and Arabi, along with iSchool Assistant Professor Chaoqun Ni, show just how few biomedical researchers have ever published an article in a high-impact journal, and just how many studies published in other outlets do end up making a real impact.
The research team finds that about half of biomedical researchers have never published in a journal with an impact factor above 15. In addition, they conclude the vast majority of papers that end up being widely cited are published outside high-impact-factor outlets. By implication, many scientists with highly influential work never publish in high impact-factor journals, and they may thus be denied access to professional opportunities on the basis of a single journal metric. These results confirmed what the researchers had suspected.
“Many researchers in the biomedical community feel that they are unfairly judged in research assessment; that they do not receive enough credit given the influence on the research community of their published work,” Hutchins, Arabi and Ni write. “We find strong empirical evidence that this is the case.”
And as Arabi told Science magazine, for early career researchers like herself, “this matters to us more than to senior researchers. There should be [a] better way to evaluate scientists.”
Improving the practice of science
The end goal for practitioners of the science of science is the development of tools and policies that have the potential to improve and accelerate research.
In the case of their latest project, Hutchins and Arabi’s work challenges the disproportionate influence of the impact factor in shaping biomedical scientists’ career trajectories. They argue that looking ahead, decision-makers in biomedicine should weigh article-level metrics, such as number of citations or likelihood of stimulating future innovation or patents, more heavily alongside journal-level metrics like the impact factor.
“Even if it’s not broadly recognized by the scientific community, a paper may still inform later clinical research, which then moves the needle on human health,” Hutchins said. “I want those measures to also be visible so that people have multiple ways to have their research recognized,” he said.
When asked why they decided to pursue science-of-science research in the first place, Hutchins and Arabi both pointed to the direct effects their work could have on real policies. “I saw it as a way of improving legacy decision systems by analyzing decision-making structures and finding ways to improve upon them,” Hutchins said.
Arabi added, “The questions we address go directly to policy decisions and funding decisions.”
Moving forward, the researchers hope their work will prompt biomedical institutions like universities and hospital systems to reconsider how much weight they give journal impact factors in hiring and promotion policies. A broader goal of Hutchins and Arabi’s work, though, is to help create a more equitable research enterprise — one that values scientists’ work holistically, rather than leaning so heavily on a single indicator like the impact factor.
To learn more about the Metascience Research Lab at the iSchool, visit its website.
For information about the iSchool PhD program, contact the program director, Professor Rebekah Willett, at rwillett@wisc.edu.