At last month’s CHORUS Forum on Real World Impacts on Faculty Publications speakers tackled the issues about what real-world impact means and how it might be measured. Panelists touched on a variety of issues within the topic and highlighted tools and systems designed to answer the hard questions around the challenges of finding ways to measure scholarly impact.
Jason Owen Smith, a Professor of Sociology at the University of Michigan, spoke about his work with the Institute for Research on Innovation and Science (IRIS), a consortium of 40 universities that is developing ways to trace individuals within data systems, including publication, patents, and particularly, university administrative data.
Jason called attention to a quote by J. Robert Opppenheimer: “The best way to send information is to wrap it up in a person.” In a paper published by Science (2015) 350(6266): 1367-71, researcher data from the university of Michigan was coupled with data from the US census bureau to understand where they were working, what they were earning, and their contributions.
Data from IRIS was also used in a study titled “Women are credited less in science than men,” published in Nature. The authors gathered data about the full teams of people who were employed on research grants and data about the publications produced by those researchers to demonstrate that women in all roles on research grants are systematically less likely to be credited as authors on publication.
As Jason observed, “if publications and other data are to be used as a way to start tracing the movement of people, we also need to really understand some of things about scientific work and credit it and the transition from contributing to being named on an outcome. Shifting the level of analysis from dollars and documents to people, careers, and teams is a must in order not to miss the picture of impact that can be extracted from data.”
Hear and see more from Jason Owen Smith and the mechanics of IRIS at 02:42
Arthur Ellis, a Senior Advisor to Elsevier, provided an overview of some analysis he performed using SciVal data. Data Analytics can be used for a variety of storytelling – in particular, to communicate to various stakeholder groups the importance of research and the positive impact it can have on society. The SciVal framework tries to take a holistic approach in evaluating the academic ecosystem, and in doing so draws on the United Nations Sustainable Development Goals (UNSDG).
Art explained that SDGs were rolled out in 2015 and endorsed by all UN member nations with the idea that by the year 2030 they would make the world more fair and sustainable. Approaching the mid-point of that time frame, there is a lot of work remaining. However, some institutions have taken steps to review their sustainability efforts, as illustrated by one story from the University of California, Davis.
Using data from SciVal for Historically Black Colleges and Universities (HBCU) between 2013 – 2022, Art found there have been approximately 50,000 scholarly outputs from the roughly 100 HBCUs, involving some 26,000 authors and co-authors. Using citation metrics, one can assess what the citation value might be; these metrics can be mapped to the UNSDG. Policy documents are another example of published material that can have societal impact and influence on policy.
Hear more from Art about additional analysis using data to tell stories at 15:50
Alison Denby, Vice President of Journals at Oxford University Press (OUP) gave an overview of the work being done at Sensus Impact, a collaboration between OUP and Silverchair. Alison pointed out that many in the community have been moving away from some traditional standards and shifting their focus to demonstrating quality in academic research and education, focusing on what quality means and how it shapes science and future innovation and impact. The reporting landscape is changing for everyone; OUP identified key areas for their stakeholders and ways they can demonstrate the impact of the content produced in OUP journals. OUP’s key areas areas of focus are:
- Demonstrating value
- Protecting research integrity
- Preserving author choice
- Supporting APC-funded gold OA
- Interrogating impact data
OUP seeks to show the impact of funded publications on society, an author, a journal, an institution and provide the impact data to stakeholders in dashboards. As there is a lot of data collected you need to make sure to know where that data is, how to link the data, and tag the data. The initial model is dashboarding, focusing on how to demonstrate the value of funded content. The next step is to build on what CHORUS provides in terms of monitoring compliance with funder mandates, showing the value being demonstrated by collecting and presenting data points in combination.
Hear more from Alison at 28:24
Make sure to watch the Q&A session at 38:50
A common thread between the tools referenced during the presentations was that data can be used to develop stories and demonstrate impact around the persons, organizations, and other measures in scholarly communications, whether they be research articles, patents, or other outputs and publications.
Thank you to our speakers for their insights, our sponsors for their contributions, and the many attendees who participated and helped make this CHORUS Forum a success.
