"Who's Afraid of Peer Review?" is an article written by ''Science'' correspondent

John Bohannon John Bohannon is an American science journalist and scientist who is Director of Science at Primer, an artificial intelligence company headquartered in San Francisco, California. He is known for his career prior to Primer as a science journalist ...

that describes his investigation of

peer review Peer review is the evaluation of work by one or more people with similar competencies as the producers of the work ( peers). It functions as a form of self-regulation by qualified members of a profession within the relevant field. Peer revie ...

among fee-charging open-access journals. Between January and August 2013, Bohannon submitted fake scientific papers to 304 journals owned by fee-charging open access publishers. The papers, writes Bohannon, "were designed with such grave and obvious scientific flaws that they should have been rejected immediately by editors and peer reviewers", but 60% of the journals accepted them. The article and associated data were published in the 4 October 2013 issue of ''Science'' as open access.

Background

The first fee-charging open access scientific journals began appearing in 2000 with the creation of

BioMed Central BioMed Central (BMC) is a United Kingdom-based, for-profit scientific open access publisher that produces over 250 scientific journals. All its journals are published online only. BioMed Central describes itself as the first and largest open ...

and then the Public Library of Science. Rather than deriving at least some of their revenue from subscription fees, fee-charging open access journals only charge the authors (or their funders) a

publication fee An article processing charge (APC), also known as a publication fee, is a fee which is sometimes charged to authors. Most commonly, it is involved in making a work available as open access (OA), in either a full OA journal or in a hybrid journal. ...

. The published papers are then freely available on the internet. This business model, gold open access, is one of several solutions devised to make open access publishing sustainable. The number of articles published open access, or made freely available after some time behind a paywall ( delayed open access), has grown rapidly. In 2013, more than half of the scientific papers published in 2011 were available for free. In part because of the low barrier to entry into this market, as well as the fast and potentially large return on investment, many so-called " predatory publishers" have created low-quality journals that provide little to no peer review or editorial control, essentially publishing every submitted article as long as the publication fee is paid. Some of these publishers additionally deceive authors about publication fees, use the names of scientists as editors and reviewers without their knowledge, and/or obfuscate the true location and identity of the publishers. The prevalence of these deceptive publishers, and what the

scientific community The scientific community is a diverse network of interacting scientists. It includes many " sub-communities" working on particular scientific fields, and within particular institutions; interdisciplinary and cross-institutional activities are als ...

should do about them, has been hotly debated.

Methods

Fake papers

Bohannon used Python to create a "scientific version of Mad Libs". The paper's template is "Molecule X from

lichen A lichen ( , ) is a composite organism that arises from algae or cyanobacteria living among filaments of multiple fungi species in a mutualistic relationship.

Publisher targets

To build a comprehensive list of fee-charging open access publishers, Bohannon relied on two sources:

Beall's List Beall's List was a prominent list of predatory open-access publishers that was maintained by University of Colorado librarian Jeffrey Beall on his blog ''Scholarly Open Access''. The list aimed to document open-access publishers who did not pe ...

of predatory publishers and the Directory of Open Access Journals (DOAJ). After filtering both lists for open access journals published in English, that charge authors a publication fee, and that have at least one medical, biological, or chemical journal, the list of targets included 304 publishers: 167 from the DOAJ, 121 from Beall's list, and 16 that were listed by both. The investigation focused entirely on fee-charging open access journals. Bohannon did not include other types of open access journals or subscription journals for comparison because the turnaround time for reviews in traditional journals is too long. The study consequently makes no claim about the relative quality of the different types of journals.

Results

Acceptance versus rejection

In total, 157 of the journals accepted the paper and 98 rejected it, with the other 49 not having completed their evaluation by the time Bohannon wrote his article. Of the 255 papers that underwent the entire peer review process to acceptance or rejection, about 60% of the final decisions occurred with no sign of actual peer review. For rejections, that may possibly have reflected filtering at the editorial level, but for acceptance can only reflect a flawed process. Only 36 submissions generated review comments recognizing any of the paper's scientific problems. 16 of those 36 papers were nonetheless accepted, in spite of poor to damning reviews. Many of the journals that accepted the paper are published by prestigious institutions and publishing companies, including

Elsevier Elsevier () is a Dutch academic publishing company specializing in scientific, technical, and medical content. Its products include journals such as '' The Lancet'', '' Cell'', the ScienceDirect collection of electronic journals, '' Trends'', ...

, Sage,

Wolters Kluwer Wolters Kluwer N.V. () is a Dutch information services company. The company is headquartered in Alphen aan den Rijn, Netherlands (Global) and Philadelphia, United States (corporate). Wolters Kluwer in its current form was founded in 1987 with a m ...

- through its subsidiary

Medknow Medknow Publications also known as Wolters Kluwer Medknow or simply Medknow, is a publisher of academic journals on behalf of learned societies and associations. Previously an independent Indian publisher, Medknow is now part of within Wolters ...

''(Journal of Natural Pharmaceuticals''), and several universities. Among those that rejected the paper are journals published by PLOS ''(PLOS ONE)'', MDPI ''(Cancer)'' and Hindawi ''(Chemotherapy Research and Practice, ISRN Oncology)''. The peer review provided by '' PLOS ONE'' was reported to be the most rigorous of all, and it was the only journal that identified the paper's ethical problems, for example the lack of documentation of how animals were treated in the creation of the cancer cell lines.

DOAJ versus Beall's list

Among the publishers on

Beall's list Beall's List was a prominent list of predatory open-access publishers that was maintained by University of Colorado librarian Jeffrey Beall on his blog ''Scholarly Open Access''. The list aimed to document open-access publishers who did not pe ...

that completed the review process, 82% accepted the paper. Bohannon stated "the results show that Beall is good at spotting publishers with poor quality control." According to Jeffrey Beall, who created the list, this supports his claim to be identifying "predatory" publishers. However, the remaining 18% of publishers identified by Beall as predatory rejected the fake paper, causing science communicator Phil Davis to state, "That means that Beall is falsely accusing nearly one in five." Among the

DOAJ The Directory of Open Access Journals (DOAJ) is a website that hosts a community-curated list of open access journals, maintained by Infrastructure Services for Open Access (IS4OA). It was launched in 2003 with 300 open access journals. The proj ...

publishers that completed the review process, 45% accepted the paper. According to a statement published on the DOAJ website, new criteria for inclusion in the DOAJ are being implemented.

Global map of journal fraud

Along with the report, ''Science'' published a map that shows the location of publishers, editors, and their bank accounts, color-coded by acceptance or rejection of the paper. The locations were derived from IP address traces within the raw headers of e-mails, WHOIS registrations, and bank invoices for publication fees. India emerged as the world's largest base for fee-charging open-access publishing, with 64 accepting the fatally flawed papers and only 15 rejecting it. The United States is the next largest base, with 29 publishers accepting the paper and 26 rejecting it. In Africa, Nigeria has the largest number, of which 100% accepted the paper.

Responses

Responses from open-access academic publishers

Since the story was released, publishers of three journals have stated that they are shutting them down. The DOAJ is reviewing its list and instituting tighter criteria for inclusion. The

Open Access Scholarly Publishers Association The Open Access Scholarly Publishing Association (OASPA) is a non-profit trade association of open access journal and book publishers. Having started with an exclusive focus on open access journals, it has since expanded its activities to include ...

(OASPA) formed a committee to investigate the circumstances that led to the acceptance of the fake paper by three of its members. On 11 November 2013, OASPA terminated the membership of two publishers (

Dove Medical Press Dove Medical Press is an academic publisher of open access peer-reviewed scientific and medical journals, with offices in Macclesfield, London (United Kingdom), Princeton, New Jersey (United States), and Auckland (New Zealand). In September 2017, ...

and Hikari Ltd.) who accepted the fake paper. Sage Publications, which also accepted a fake paper, was put "under review" for 6 months. Sage announced in a statement that it was reviewing the journal that accepted the fake paper, but that it would not shut it down. Sage's membership was reinstated at the end of the review period following changes to the journal's editorial processes.

Responses from the scientific community

Within hours of its publication, the ''Science'' investigation came under intense criticism by some supporters of the open-access movement. The first substantial critique was posted by PLOS cofounder Michael Eisen on his blog. "To suggest – as ''Science'' (though not Bohannon) are trying to do – that the problem with scientific publishing is that open access enables internet scamming is like saying that the problem with the international finance system is that it enables Nigerian wire transfer scams. There are deep problems with science publishing. But the way to fix this is not to curtail open-access publishing. It is to fix peer review." Eisen pointed out the irony of a subscription-based journal like ''Science'' publishing this report when its own peer review has failed so badly before, as in the 2010 publication of the arsenic DNA paper. In an exchange between Eisen and Bohannon in a discussion hosted by

Peter Suber Peter Dain Suber (born November 8, 1951) is a philosopher specializing in the philosophy of law and open access to knowledge. He is a Senior Researcher at the Berkman Klein Center for Internet & Society, Director of the Harvard Office for Scholarl ...

, director of the Harvard Open Access Project, Eisen criticized the investigation for the bad publicity it generated for the open-access movement. "Your study exclusively targeted open access journals – hichstrongly suggested, whether you meant to suggest this or not, that open access journals are more likely to engage in shoddy peer review and therefore more deserving of scrutiny." Bohannon responded that this critique was equivalent to "

shooting the messenger "Shooting the messenger" is a metaphoric phrase used to describe the act of blaming the bearer of bad news. Until the advent of modern telecommunication, messages were usually delivered by human envoys. For example, in war, a messenger would ...

". There have also been many statements of support for the investigation, and statements of concern about the publishing fraud that it revealed. The Committee on Publication Ethics has responded that "There is no doubt that this 'sting' raises a number of issues ... though I'd argue they are not necessarily the ones that ''Science'' thinks are top priorities."

Implications

Some scientists have discussed a number of options for making peer review more transparent. Doing so would make it harder to maintain a predatory journal that does no peer review, because the record of peer review would be lacking or would need to be faked. Another option is to more rigorously vet journals, for example by further empowering DOAJ or OASPA. DOAJ has recently tightened up its inclusion criteria, with the purpose of serving as a whitelist, very much like

has been a

blacklist Blacklisting is the action of a group or authority compiling a blacklist (or black list) of people, countries or other entities to be avoided or distrusted as being deemed unacceptable to those making the list. If someone is on a blacklist, ...

References

{{reflist, 30em, refs= {{cite news , last=Basken , first=Paul , title=Critics Say Sting on Open-Access Journals Misses Larger Point , url=http://chronicle.com/blogs/percolator/critics-say-sting-on-open-access-journals-misses-larger-point/33559 , access-date=20 October 2013 , newspaper=The Chronicle of Higher Education , date=4 October 2013 {{cite news , last=Mudur , first=G S , title=Dubious journal fear stalks India , url=http://www.telegraphindia.com/1131004/jsp/nation/story_17423175.jsp#.UmQ4nCSE7Wg , access-date=20 October 2013 , newspaper=The Telegraph , date=3 October 2013 {{cite news , last=Vergano , first=Dan , title=Fake Cancer Study Spotlights Bogus Science Journals , url=http://news.nationalgeographic.com/news/2013/10/131003-bohannon-science-spoof-open-access-peer-review-cancer/ , access-date=22 October 2013 , newspaper=National Geographic , date=3 October 2013 {{cite news , last=Oransky , first=Ivan , title=Fallout from Science's publisher sting: Journal closes in Croatia , url=http://retractionwatch.wordpress.com/2013/10/17/fallout-from-sciences-publisher-sting-journal-closes-in-croatia/ , newspaper=Retraction Watch , date=17 October 2013 {{Cite journal , doi = 10.1038/500386a , issn = 0028-0836 , volume = 500 , issue = 7463 , pages = 386–387 , last = Van Noorden , first = Richard , title = Half of 2011 papers now free to read , journal = Nature , date = 20 August 2013 , pmid = 23969438 , bibcode = 2013Natur.500..386V , doi-access = free {{cite web , last=Eisen , first=Michael , title=I confess, I wrote the Arsenic DNA paper to expose flaws in peer-review at subscription based journals , url=http://www.michaeleisen.org/blog/?p=1439 , access-date=21 October 2013 {{cite web , last=Koebler , first=Jason , title=Inside Science Magazine's 'Sting' of Open Access Journals , url=http://motherboard.vice.com/blog/inside-science-magazines-sting-of-open-access-journals-1 , work=Motherboard , access-date=22 October 2013 {{Cite journal , doi = 10.1038/495433a , issn = 0028-0836 , volume = 495 , issue = 7442 , pages = 433–435 , last = Butler , first = Declan , title = Investigating journals: The dark side of publishing , journal = Nature , date = 27 March 2013 , pmid = 23538810 , bibcode = 2013Natur.495..433B , doi-access = free {{cite web , last=Shieber , first=Stuart , date=October 15, 2013 , title=Lessons from the faux journal investigation , url=http://blogs.law.harvard.edu/pamphlet/2013/10/15/lessons-from-the-faux-journal-investigation/ , work=The Occasional Pamphlet , publisher=Harvard Law School , access-date=20 October 2013 {{cite web , last=Suber , first=Peter , title=Open Access Overview , url=http://legacy.earlham.edu/~peters/fos/overview.htm , publisher=Earlham College , access-date=21 October 2013 {{cite web , title=Peer review map , url=http://sciencestatic.aws.aaas.org.s3.amazonaws.com/article-resources/who-does-peer-review/index.html?_ga=1.167695550.1210254765.1488811582 , publisher=AAAS , work=Science , access-date=21 October 2013 {{cite web , last=Davis , first=Phil , title=Post Open Access Sting: An Interview With John Bohannon , url=http://scholarlykitchen.sspnet.org/2013/11/12/post-open-access-sting-an-interview-with-john-bohannon/ , date=November 12, 2013 , work= The Scholarly Kitchen , access-date=12 November 2013 {{cite news , last=Yirka , first=Bob , date=October 4, 2013 , title=Research paper publishing sting reveals lax standards of many open-access journals , url=http://phys.org/news/2013-10-paper-publishing-reveals-lax-standards.html , work=Phys.org , access-date=20 October 2013 {{cite web , last=Beall , first=Jeffrey , title=Science Magazine Conducts Sting Operation on OA Publishers , url=http://scholarlyoa.com/2013/10/03/science/ , work=Scholarly Open Access , access-date=22 October 2013 , archive-url=https://web.archive.org/web/20131012052520/http://scholarlyoa.com/2013/10/03/science/ , archive-date=12 October 2013 , url-status=dead {{cite web , last=Oransky , first=Ivan , date=October 3, 2013 , title=Science reporter spoofs hundreds of open access journals with fake papers , url=http://retractionwatch.wordpress.com/2013/10/03/science-reporter-spoofs-hundreds-of-journals-with-a-fake-paper/ , work=Retraction Watch , access-date=22 October 2013 {{cite news , title=Science's Sokal moment , url=https://www.economist.com/news/science-and-technology/21587197-it-seems-dangerously-easy-get-scientific-nonsense-published-sciences-sokal , access-date=20 October 2013 , url-access=registration , newspaper=The Economist , date=5 October 2013 {{cite web, title=Second response to the Bohannon article , date=2013-10-18 , url=http://doaj.org/doaj?func=news&nId=317&uiLanguage=en , work=Directory of Open Access Journals , access-date=21 October 2013 , url-status=dead , archive-url=https://web.archive.org/web/20131110163125/http://doaj.org/doaj?func=news&nId=317&uiLanguage=en , archive-date=10 November 2013 {{cite news , last=Knox , first=Richard , title=Some Online Journals Will Publish Fake Science, For A Fee , url=https://www.npr.org/blogs/health/2013/10/03/228859954/some-online-journals-will-publish-fake-science-for-a-fee , access-date=20 October 2013 , newspaper=NPR , date=3 October 2013 {{cite web , last=Gamboa , first=Camille , title=Statement by SAGE on the Journal of International Medical Research , publisher=Sage , url=http://www.sagepub.com/press/2013/october/SAGE_statementSAGEJIMR.sp , access-date=22 November 2013 , archive-url=https://web.archive.org/web/20131014073843/http://www.sagepub.com/press/2013/october/SAGE_statementSAGEJIMR.sp , archive-date=2013-10-14 , url-status=dead {{cite web , last=Tatalović , first=Mićo , date=2013-03-10 , title=Sting exposes 'wild west' of open-access publishing , url=http://www.scidev.net/global/publishing/news/sting-exposes-wild-west-of-open-access-publishing.html , work=Scidev.net , access-date=20 October 2013 {{cite journal , doi=10.1038/nmat2952 , title=Transparency in peer review , year=2011 , journal=Nature Materials , volume=10 , issue=2 , pages=81 , pmid=21258345 , bibcode=2011NatMa..10...81. , doi-access=free {{cite journal , last=Bohannon , first=John , title=Data and Documents; Who's Afraid of Peer Review , journal=Science , year=2013a , volume=342 , issue=6154 , pages=60–65 , doi=10.1126/science.342.6154.60 , pmid=24092725 , doi-access=free Paper 1 in the data supplement for {{harvnb , Bohannon , 2013. {{cite journal , last=Bohannon , first=John , title=Who's Afraid of Peer Review? , journal=Science , year=2013 , volume=342 , issue=6154 , pages=60–65 , doi=10.1126/science.342.6154.60 , pmid=24092725 , doi-access=free , bibcode=2013Sci...342...60B {{cite web , last=Davis , first=Phil , date=October 4, 2013 , title=Open Access "Sting" Reveals Deception, Missed Opportunities , url=http://scholarlykitchen.sspnet.org/2013/10/04/open-access-sting-reveals-deception-missed-opportunities/ , work= The Scholarly Kitchen , access-date=20 October 2013 {{cite web, title=DOAJ's response to the recent article in Science entitled "Who's Afraid of Peer Review?" , date=2013-10-04 , url=http://www.doaj.org/doaj?func=news&nId=315&uiLanguage=en , work=Directory of Open Access Journals , access-date=22 October 2013 , url-status=dead , archive-url=https://web.archive.org/web/20131023061516/http://www.doaj.org/doaj?func=news&nId=315&uiLanguage=en , archive-date=23 October 2013 {{cite web , url=https://plus.google.com/u/0/109377556796183035206/posts/CRHeCAtQqGq , author=Peter Suber , title=New "sting" of weak open-access journals. {{cite web , last=Redhead , first=Claire , date=October 4, 2013 , title=OASPA's second statement following the article in Science entitled "Who's Afraid of Peer Review?" , url=http://oaspa.org/oaspas-second-statement-following-the-article-in-science-entitled-whos-afraid-of-peer-review/ , publisher=Open Access Scholarly Publishers Association , access-date=11 November 2013 {{cite web , last=Redhead , first=Claire , date=November 11, 2013 , title=OASPA's response to the recent article in Science entitled "Who's Afraid of Peer Review?" , url=http://oaspa.org/response-to-the-recent-article-in-science/ , publisher=Open Access Scholarly Publishers Association , access-date=21 October 2013

External links

''Who's Afraid of Peer Review?''

Full Data Set
Open access (publishing) 2013 in science Works originally published in Science (journal) Ethically disputed business practices Hoaxes in science Peer review