CommentaryIndustry & ResearchTop Story
Opinion: Close ERA loopholes now
Quality ratings are meaningless if the process for evaluating research is easily subverted.
By Peter Drummond
Please login below to view content or subscribe now.
Quality ratings are meaningless if the process for evaluating research is easily subverted.
By Peter Drummond
Please login below to view content or subscribe now.
To continue onto Campus Review, please select your institution.
Points 1-3 seem valid. Regarding point 1, it takes a lot of effort to code journal articles to FOR codes. Taking publication classification out of the hands of universities by automating the process might actually make the process more efficient.
I’m not sure the number of fake papers being submitted for ERA evaluation is sufficiently large that an administrative system to weed them out would be efficient. It would be possible to post-hoc evaluate the magnitude of any problem, but I suspect weeding out fakes will make no material difference to the outcomes.
Good points although the issues are sometimes are more complicated and there are other games not mentioned.
The FOR code issues is more complicated in instances were there are multi-disciplinary teams that span multiple codes and articles can only appear in one journal. In such instances sharing FOR codes or moving works between codes may not be gaming.
Staff- the use of hire people (often outside Australia) on fractional appointments to count their research is another issue that needs to be examined, especially if the fractional staff are never based in the Australia institution for any part of the year.
As for fake journals, relying on impact factors is clearly one way that this can be addressed, new metrics list the quartile of the journal’s impact factor within a discipline, which also allows for new journals to be assessed as they arise without them having to be ‘assessed’.
Great contribution Peter, thank you. The issue of hundred+ author papers has always bothered me, but I’m not sure how best to handle it. Simple normalisation by author number will make those papers worthless, but we all know that some authors on those papers have contributed more than others. My gut instinct is to forget about citations, and get back to journal rankings. Publishing in good journals (high rejection rate) means you have gone through a thorough peer-review process, and were judged to be producing work of sufficiently high quality. If that is good-enough for the journal it ought to be good-enough for ERA. The problem with citations is that capacity for self-citation increases with the number of authors. This is understandable and even desirable, but has nothing to do with quality.
As far as FOR gaming is concerned, the only way to fix it is to code at the six-digit level.
Simple – as computer people know – “Garbage IN, Gospel OUT!” – yes, I meant that variation on the classical admonition in relation to ERA.
However, in the nationally and internationally vital area of research in the broad and vital aspects of “cybersecurity”, from business/government risk assessment and policy to data networks to computer systems to trusted applications to cryptography and on, the ERA situation appears to be a TOTAL MESS! Cybersecurity just “falls between the stools”! AND it has actually had the effect, I submit, of massively downsizing the critical area of cybersecurity education and research in universities here in Australia at a time when it is sorely, no DESPERATELY, needed!
Moreover, just how is the ERA ( ranking journals and so on) relevant to the vital needs of Australia in making our research activity even more relevant to industry, new technologies, new companies, and on, as our Prime Minister has emphasized. Is it really serving to even further move our universities into “ivory tower” mode?
We need a far better way of judging the significance of our university research, particularly that large publicly funded component via ARC / NHMRC, etc., to our nation, our people and to the international advancement of scholarship.
I think you will find that most of these loopholes aren’t really loopholes. The ARC has processes in place to detect much of the dodgy business that can take place. Their staff have done a lot of presentations to explain these processes.
Fake journals don’t count: there is an ERA approved list (and lists for publishers and for conferences) that was out for consultation before the ERA exercise. The final version is not available yet, which is a shame, but I think it will be due out soon after the exercise finishes. The panels and the peer reviewers (the latter exist for some disciplines only) will pick up if a university’s output is in less-respected journals.
it is true that papers follow the author, but grants don’t.
It’s not an easy task to devise a system that works well in every respect. Whatever the ARC does, seems to attract opprobrium; for example when it had spent a huge amount of effort on journal rankings, some academics and organisations made such a fuss that the rankings were dropped. In my view, people should focus their energy on doing good research that matters, and the metrics will by and large reflect this good effort.
Good thought-provoking piece Peter. Like others, I have nuanced reservations about point 4. Dismissing the journals on the Beall list of predatory journals is common sense, but not sure about bringing back the ERA journal rankings. For some disciplines at least, the latter also saw its count of easy routes being taken and strategic choices being made (in some cases it was even pure ignorance). In most disciplines everyone knows to which tier a journal belongs, and this knowledge is far superior to rankings exercises, which invite the type of manipulative games you describe above. Personally, I was happy to see the ERA list go, only lamenting the waste of resources along the way.
How about a new metric say…
Journal Publications by Contribution (JPC) Index = sum of (a journal paper / number of authors).
This means that a single author on a journal paper gets a value of 1-JPC while a paper with 5 authors provides each author with 1/5 JPC.
This would mean a single author publishing one paper can have the same index as one author who works in a group of 10 publishing 10-papers with those 10 authors!