Performance Figures in Research Assessment

Based on the institutional publication databases for GFZ, PIK and IASS the library provides performance figures for internal and external research assessments.

Citation Analysis

"Citations in the actual sense are common practice in scientific literature since the middle of the 19th century. Metaphorically speaking, the citation network of scientific articles is the glue that links publications with related content. The number of citations is seen as a direct measure for the (documented) resonance or the impact an article, a journal or a scientist has generated so far."

From: The impact of Solid State Communications in view of the ISI Citation data, DOI: dx.doi.org/10.1016/S0038-1098(03)00442-3

Science Citation Index:

The concept of citation analysis was developed in the late 1950's. The central tool is the database Science Citation Index (SCI), established by the Institute for Scientific Information (ISI), Philadelphia, USA, now part of the Thomson-Group. Its fundamental idea is to use publications of scientists and include references as an assessment criterion to evaluate the relevance of their scientific work.

Over the years the SCI was complemented by the SSCI (Social Sciences Citation Index) and the AHCI (Arts and Humanities Citation Index). During the last years CPCI (Conference Proceedings Citation Index) and BCI (Book Citation Index) were added. The joint platform is known under the name "Web of Science". Under the same platform there are more databases which can be licensed (Web of Knowledge).

SCI only indexes a defined selection of journals. Find here more on the selection process.

Scopus

2004 Elsevier launched a competitive product named Scopus. Scopus covers more journals than SCI and is also based on a defined list.

 

Content Policy: Scopus Journal Selection Criteria

 

Google Scholar Citation

Contrary to this Google Scholar (GS) aggregates metadata via text mining. The idea is to identify as many scientific publications on the net as possible. Consequently different journal articles, book contributions and more is indexed. 

Microsoft Academic Search

A similiar approach is made by Micorsoft Academic Search.

Different sources generate different numbers.

Depending on the sources of the base of course results like the h-index come out with different values. 

When working with figures from citation analysis these points are worth to consider

unique authorincompletenesses and errors in reference lists
negativ citationstemporal delays in the citation process
self citations new research processes are recognized disproportionately late
citation cartelsage of the cited person
who is cited (prestige...)amount of average citations heavily depends on the subject (differs by factor 7) 
undocumented influences (acknowledgements, etc.)majority of all publications is not cited at all, or only a few times

Hirsch-index (h-Index)

A newer way to calculate comparable numbers of citations from different authors: "The h-index is indicated by a green horizontal line in Web of Science. The number of items above this line, which is h, have at least h citations. For example, an h-index of 20 means that there are 20 items that have 20 citations or more."

Developed by J.E. Hirsch and published in Proceedings of the National Academy of Sciences of the United States of America 102 (46): 16569-16572 November 15 2005.

Impact Factor of Journals

Another metric is the calculation of impact factors of journals.

Thomson-Reuters offers such a opportunity with the Journal Citation Report (JCR) (GFZ access only). Here inter alia the Journal Impact Factor (JIF) of individual journals is calculated.

Calculation for 2000

S = number of articles and reviews, which were published 1998 and 1999 in specific journals (Source Items)

R= number of citations in 2000 of all publications, which call the in S evaluated journals year's issues 1998 and 1999 as source (References)

R/S = Journal Impact Factor

SCImago Journal & Country Rank (SJR) is based on Scopus and also offers journal based evaluation.

Such metrics always refer to the entire magazine and can not be used as personal data.

They are an indicator of "relative importance" within a subject area, a comparison across different disciplines doesn't make sense.

Factors to consider: discipline, share of  methodologically articles, review articles proportion, original language, period of data analyzed. All said things about quote analysis has an effect.

The "San Francisco Declaration on Research Assessment (DORA)"  formulated clear criticism of the excesses of the use of the impact factor in research evaluation.

altmetrics

Next to established metrics such as SCI and Scopus, the possibilities of "Altmetrics" are been studied for years.

"altmetrics" is the creation and study of new metrics based on the Social Web for analyzing, and informing scholarship.

J. Priem, D. Taraborelli, P. Groth, C. Neylon (2010), Altmetrics: A manifesto, (v.1.0), 26 October 2010. altmetrics.org/manifesto

Several publishers like Copernicus Publishing now establish a programme called ALM (Article-level metrics) to quantify the usage, impact, saves, and discussion of scholarly work at the article level. 

In detail the usage is collected from:

  • Individual accesses to the Copernicus library servers (download statistics)
  • Impact is counted from CrossRef and Google Scholar citations
  • Saves are counted from reference management tools CiteULike and Mendeley
  • Discussions are represented by Research Blogging, Facebook, ScienceSeeker, Nature Blogs, Wikipedia, Wordpress.com, Reddit, and Google Blogs.

What's the use?

In comparison to the traditional way of measuring impact at the journal level, ALMs offer a more informative way of assessing the overall influence and reach of the articles themselves.

Article-level metrics are available for all articles of journals published by Copernicus Publications. Authors can stay up to date with their published articles and share the information about the impact of their published work with peers, funding institutions, research bodies, and the overall scientific community.

The following figure shows the perception of a publication by people in social media. The publication has not yet been cited but has already been shared 160 times via Twitter.

 

PlumX Metrics in Scopus
Details in PlumX