Ranking journal impact

Last week, looking to evaluate some academic research, I found a handy website called Journal-Ranking.com. (Non-academics may not want to read further).

The site produces rankings of the top journals for given academic field. They have more than 200 discipline-specific lists to choose from (see the Appendix).

Innovation and Entrepreneurship Journals

Journal-Ranking’s methods reproduce the widely accepted list of top “A” management journals, the same list that are among the 40 used by the Financial Times rankings of business schools: Administrative Science Quarterly, Academy of Management Review, Academy of Management Journal, Strategic Management Journal, Management Science and Organization Science.

For those of us who follow a field — or have a specialized field — we prefer the more thoughtful and complete studies by a scholars in the area. The normal way of ranking journals (as Journal-Ranking.com does) is using citation data to find where the most influential articles are.

As an “Associate Professor of Innovation & Entrepreneurship,” I have an interest in the two corresponding management subfields. Finding a (relatively) objective measure for the former is easy to identify, as Linton and Thongpapanl (2004) have used the standard process for citation ranking — including adjustments for self-citations (which are inherently suspect).

For entrepreneurship, the story is a little more complex. Katz and Boal (2002) survey some of the various articles to date that attempted to solve the problem. Fried (2003) updates a classic opinion survey, but it doesn’t use citations.

Busenitz et al (2004) use a citation analysis system, but only for “A” management journals (plus the journal that published the article), not the standard top entrepreneurship journals like Journal of Business Venturing, Entrepreneurship Theory & Practice, Small Business Economics and Journal of Small Business Management.

Normally, for general comparability of business journals I use the citation-based statistics compiled by former Academy of Management president Bill Starbuck. I have also provided the Starbuck list to others seeking to identify (or justify) journal choices.

Limitations of Journal Rankings

The idea of journal ranking is controversial. (For now, I’ll just concern myself with the controversies within the business field — although there are similar controversies in other fields.)

Those who can’t (or don’t) publish in the top ranked journals disagree with either the implementation or premise of the idea. Even among those who winners at the current game, there are some who just object in principle to the idea of “good” journals and “bad” journals. Two influential scholars, Nancy Adler and Anne-Wil Harzing this month published a detailed attack on rankings that fits the latter category (Adler and Harzing 2009).

Some object to the use of citations as a measure of an importance of an article. There are many ways to false positives: a statement such as “Jones (2008) produced a flawed study contradicted by 200 years of research” will generate a citation. The false negatives are generally for research that impacts managerial practice in a way that does not end up being cited in an academic journal.

Still, citations are one of the few objective measures of evaluating the impact of the research of thousands of articles and researchers. Opinion surveys are heavily biased by personal (un)popularity and prior reputation. The survey can also be biased by the choice of who to ask.

The most valid argument against journal ranking is that journal impact is just a lazy way to estimate article impact. There are good articles published in middling journals. There are articles in top journals that never have any impact at all. The best way to measure an article’s impact is to wait a few years to see if it has any impact, i.e., it gets cited.

(However, tenure decisions depend on judging the impact of recently published or even forthcoming articles. The only way to assess such recent research is to read the article and try to guess. As a practical matter, academic scholarship is often so specialized that only a few dozen people in the world can accurate access an article’s future importance by reading it. Rarely do such experts work at the same university, which raises a host of practical problems.)

However, there is one limitation to the argument about good articles should count wherever they appear: no matter how good an article, if no one ever sees it, it will never gets cited. I have seen some middlin review articles that get cited because they appear in a top journal where everyone sees it: a better article in a comparable journal would have been cited instead, but no one saw it. (In my own experience, I have a good article that has been accepted in a B- journal that no one will ever see).

The best example of articles that have influence beyond the impact of their journal are articles that create a new field or line of research, but the author chose to bypass the “A” journal process to get the paper published quickly — normally by publishing in a “B” journal.

One celebrated example is when Jay Barney in 1991 launched what became known as the “resource-based view of the firm” (11,000+ citations thus far). His alternative would have been to spend years fighting “A” journal reviewers to get a controversial new idea published, as David Teece and his colleagues did to publish their 1997 article on “dynamic capabilities”.

In general, what I advise young academics to do is to publish in an “A” or “B” journal. (All of the Starbuck-ranked journals can be considered “B” or higher). Those are the journals that authors normally look at when researching their own articles.

My own most-cited article appeared in such a journal, Research Policy, which is the undeniably top journal in innovation management but only a “B+” journal by measures that span the entire business (or management) field. A believe RP does not rank higher because fewer people publish innovation articles (and thus cite innovation articles) than publish articles in strategy, organizational behavior, or other more popular management fields; the same bias would also impact other smaller subfields, like human resources or ethics.

Using the Site

What I find handy about the Journal-Ranking.com site is that it uses the same metrics across multiple fields.

However, I don’t get the business model. As best I can tell, it’s a relatively cheap to implement and operate website run as an experiment/loss leader by a Hong Kong IT consultant:

The team at Red Jasper is able to harness advanced research from the academia to create innovative products and applications that generate value to our clients. We aspire towards building unparalleled alliances and long term relationships with companies of complementary strengths so that extraordinary value can be extricated and realized.

As with other journal ranking systems, this one is based on how frequently articles from the journal are cited by other articles. The two key problems are what to use as a sampling domain, and how to weight citations from different journals.

Red Jasper claims to have developed a new weighting algorithm. (Several explanatory papers are available). For this, they’ve won an AAAI award, which seems like a fair return on their investment.

References

Nancy J. Adler, Anne-Wil Harzing, “When Knowledge Wins: Transcending the Sense and Nonsense of Academic Rankings,” Academy of Management Leaning and Education, 8, 1 (March 2009).

Lowell W. Busenitz, G. Page West III, Dean Shepherd, Teresa Nelson, Gaylen N. Chandler, Andrew Zacharakis, “Entrepreneurship Research in Emergence: Past Trends and Future Directions,” Journal of Management, 29,3 (2003): 285–308. DOI: 10.1016/S0149-2063_03_00013-8

Vance H. Fried, “Defining a forum for entrepreneurship scholars,” Journal of Business Venturing 18,1 (2003) 1-11. DOI: 10.1016/S0883-9026(02)00117-9

Jerome Katz and Kim Boal, “Entrepreneurship Journal Rankings,” March 2002, URL: http://www.marketingtechie.com/articles/mtart20020307.pdf

Jonathan D. Linton and Narongsak (Tek) Thongpapanl, “Ranking the Technology Innovation Management Journals,” Journal of Product Innovation Management, 21, 2 (March 2004): 123-139. DOI: 10.1111/j.0737-6782.2004.00062.x

Appendix: Supported Disciplines

Not counting some aggregate search criteria, Journal-Rankings.com provides a choice of 218 disciplines:

Acoustics, Agricultural Engineering, Agricultural Economics and Policy, Agriculture Dairy and Animal Science, Agriculture Multidisciplinary, Agriculture Soil Science, Agronomy, Allergy, Anatomy and Morphology, Andrology, Anesthesiology, Anthropology, Applied Linguistics, Area Studies, Astronomy and Astrophysics, Automation and Control Systems, Behavioral Sciences, Biochemical Research Methods, Biochemistry and Molecular Biology, Biodiversity Conservation, Biology, Biophysics, Biotechnology and Applied Microbiology, Business, Business Finance, Cardiac and Cardiovascular Systems, Cell Biology, Chemistry Analytical, Chemistry Applied, Chemistry Inorganic and Nuclear, Chemistry Medicinal, Chemistry Multidisciplinary, Chemistry Organic, Chemistry Physical, Clinical Neurology, Communication, Computer Science Cybernetics, Computer Science Artificial Intelligence, Computer Science Hardware and Architecture, Computer Science Information Systems, Computer Science Interdisciplinary Applications, Computer Science Software Engineering, Computer Science Theory and Methods, Construction and Building Technology, Criminology and Penology, Critical Care Medicine, Crystallography, Demography, Dentistry Oral Surgery and Medicine, Dermatology, Developmental Biology, Ecology, Economics, Education and Educational Research, Education Scientific Disciplines, Education Special, Electrochemistry, Emergency Medicine, Endocrinology and Metabolism, Energy and Fuels, Engineering Aerospace, Engineering Biomedical, Engineering Chemical, Engineering Civil, Engineering Environmental, Engineering Geological, Engineering Industrial, Engineering Manufacturing, Engineering Marine, Engineering Mechanical, Engineering Multidisciplinary, Engineering Ocean, Engineering Petroleum, Engineering Electrical and Electronic, Entomology, Environmental Sciences, Environmental Studies, Ergonomics, Ethics, Ethnic Studies, Evolutionary Biology, Family Studies, Fisheries, Food Science and Technology, Forestry, Gastroenterology and Hepatology, Genetics and Heredity, Geochemistry and Geophysics, Geography, Geography Physical, Geology, Geosciences Multidisciplinary, Geriatrics and Gerontology, Gerontology, Health Care Sciences and Services, Health Policy and Services, Hematology, History, History and Philosophy Of Science, History Of Social Sciences, Horticulture, Imaging Science and Photographic Technology, Immunology, Industrial Relations and Labor, Infectious Diseases, Information Science and Library Science, Instruments and Instrumentation, Integrative and Complementary Medicine, International Relations, Law, Limnology, Management, Marine and Freshwater Biology, Materials Science Biomaterials, Materials Science Ceramics, Materials Science Composites, Materials Science Multidisciplinary, Materials Science Textiles, Materials Science Characterization and Testing, Materials Science Coatings and Films, Materials Science Paper and Wood, Mathematics, Mathematics Applied, Mathematics Interdisciplinary Applications, Mechanics, Medical Ethics, Medical Informatics, Medical Laboratory Technology, Medicine General and Internal, Medicine Legal, Medicine Research and Experimental, Metallurgy and Metallurgical Engineering, Meteorology and Atmospheric Sciences, Microbiology, Microscopy, Mineralogy, Mining and Mineral Processing, Multidisciplinary Sciences, Mycology, Neuroimaging, Neurosciences, Nuclear Science and Technology, Nursing, Nutrition and Dietetics, Obstetrics and Gynecology, Oceanography, Oncology, Operations Research and Management Science, Ophthalmology, Optics, Ornithology, Orthopedics, Otorhinolaryngology, Paleontology, Parasitology, Pathology, Pediatrics, Peripheral Vascular Disease, Pharmacology and Pharmacy, Physics Applied, Physics Atomic Molecular and Chemical, Physics Condensed Matter, Physics Fluids and Plasmas, Physics Mathematical, Physics Multidisciplinary, Physics Nuclear, Physics Particles and Fields, Physiology, Planning and Development, Plant Sciences, Political Science, Polymer Science, Psychiatry, Psychology, Psychology Applied, Psychology Biological, Psychology Clinical, Psychology Developmental, Psychology Educational, Psychology Experimental, Psychology Mathematical, Psychology Multidisciplinary, Psychology Psychoanalysis, Psychology Social, Public Administration, Public Environmental and Occupational Health, Radiology Nuclear Medicine and Medical Imaging, Rehabilitation, Remote Sensing, Reproductive Biology, Respiratory System, Rheumatology, Robotics, Social Issues, Social Sciences Biomedical, Social Sciences Interdisciplinary, Social Sciences Mathematical Methods, Social Work, Sociology, Spectroscopy, Sport Sciences, Statistics and Probability, Substance Abuse, Surgery, Telecommunications, Thermodynamics, Toxicology, Transplantation, Transportation, Transportation Science and Technology, Tropical Medicine, Urban Studies, Urology and Nephrology, Veterinary Sciences, Virology, Water Resources, Women’s Studies, Zoology