Research in the 21st Century: legacy forms, ‘new’ media and ERA

As Australian universities prepare their submissions for the 2023 round of Excellence in Research Australia (ERA), the national research evaluation framework, it is a good moment to reflect and review this thing we call ‘research’.

Indeed the ERA exercise, which is run every three years to evaluate how different disciplines across the Australian research landscape are performing against an ambiguous ‘world standard’ (although the rankings out of 5 are often used by universities to compare with each other), is currently seeking feedback-submissions on the recently released draft ERA23 submission guidelines.

We, at Macquarie University’s Creative Documentary Research Centre, take this request for feedback as an opportunity to highlight a significant anomaly in research assessment that seems increasingly anachronistic in the 21^st Century.

The specific context is that much of the major research that our members produce is in a form that is quaintly described as ‘Non-Traditional Research Outputs’, or NTROs defined in distinction to the more common ‘Traditional Research Outputs’. While NTROs within universities are produced across many Fields of Research (FoR) codes, and can also take the form of commissioned reports, the main subject of our attention here is Creative Arts and Writing (now Field of Research code 36, previously FoR19).

Approximately half of the outputs submitted in Creative Arts and Writing (then FoR 19) in the ERA15 and ERA18 exercises were NTROs. They include works such as films, novels, short stories, poetry, paintings, photography, videos, dance and music compositions. The argument has been decades in the making, with the combined work of numerous leading scholars, but there can be no doubt now that creative-arts based research methods and forms are capable of uncovering important new knowledge that contributes to scholarship across many fields of research (for some historical context see Jenny Wilson, 2011).

In the case of recent work produced by CDRC members, these contributions cover fields including psychology, cognition, anthropology, geography, feminism and gender-studies, Australian history, Indigenous studies, early childhood learning, the criminal justice system, and various creative-arts disciplines. This deeply researched work, sometimes taking nearly a decade to produce, has engaged large audiences in film festivals and through television broadcast screenings at the Sydney Film Festival, Melbourne Film Festival, Antenna Festival, and SBS TV. 

Our members have also written major fiction works, and non-fiction works such as this, and this.

In each case these research outputs were funded (in whole or part) through the Australian Research Council, university post-doctoral research funds, and internal university research funds. Yet, due to 'legacy' understandings that currently underpin the idea of ‘research’, these works –when submitted during the ERA process–are evaluated as worth only a fraction of what an equivalent 'scholarly' book is worth.

Both the ERA18 Guidelines and ERA23 Draft Guidelines state that ‘books have a weighting of 5:1 compared to other research outputs’ (ERA23 Guidelines, p.12). By contrast, all ‘original creative works’ (i.e. NTROs), no matter the size, have a weighting of 1 point each. (There are cases too where a portfolio of works is submitted together as a single output, attracting 1 point.) In other words, while a scholarly book attracts a weighting of 5 points, research delivered in a feature-length screen documentary or a 100,000-word novel or work of literary non-fiction, can only attract 1 point.

In the 21^st Century it is odd that research delivered in audio-visual media, for example, can never be equivalent to research delivered in words on a page. Even though the former may have more to say.

Is this important?

After ERA18, Canberra University scholars Jen Webb and Ross Gibson usefully reflected on the consistent ‘lacklustre’ ERA results in the creative arts (NiTRO, 2019). While every other FoR had increased its average score in each ERA round, FoR19 (now 36) was the only code in which the average score across the sector had decreased (McKee 2018).

Webb and Gibson wondered why this was the case: Australian creative practice-led research is, after all, recognised as world-leading. They concluded that the underwhelming ERA results in this FoR were due to a failure of creative arts researchers to sufficiently address why their work constituted research; that evaluation measures in ERA remain science focused (‘espoused knowledge is more positively valued than a sensed understanding’); and that members of the creative academic community are, perhaps, inordinately tough on one another. This may also be one reason why peer-reviewed disciplines have generally not improved over previous ERA rounds at the same rate as the citation disciplines, which are predominantly in STEM (science, technology, engineering and mathematics).

We suggest that a further reason for these poor ERA results in the creative arts has to do with the way that creative arts outputs have been consistently (and concretely) under-valued as research, in both quantitative and qualitative terms. At the same time, evidence (both UK and Australian data) indicates that the numbers of NTROs being produced by universities are rising. It is therefore more urgent than ever that this inequity be understood and redressed.

How does this impact the Australian research landscape?

This inequity has the potential to significantly skew the status of ERA submissions in FoR codes, and in research institutions, where large-scale NTROs are a substantial element of the submission. NTROs currently appear in a number of two-digit FoRs: e.g. Anthropology or History, for example, might, as part of their submissions, include outputs such as a documentary podcast or an ethnographic film. Such outputs might solely fall under ‘Anthropology’ or ‘History’ because their knowledge-contributions sits squarely in these areas. In these two-digit FoRs, NTROs are well and truly in the minority. Not so in Creative Arts and Writing: in ERA2015, 52% of outputs were NTROs; in ERA2018, 49% were NTROs (data: ERA18 report).

Currently in the ERA assessment process, there is a low volume threshold for submissions: for FoRs that are evaluated using peer review, the threshold is the equivalent of 50 weighted apportioned research outputs. ‘If the number of weighted apportioned research outputs over the six-year output reference period is less than 50 in any four-digit or two-digit FoR at an institution, then ERA will not evaluate that FoR at that institution’ (ERA23 Guidelines, p.12). For a FoR to reach that threshold, it might submit 10 scholarly books. Alternatively it might submit 50 journal articles and scholarly book chapters. Or perhaps a combination of all three.

However, in FoR codes where major creative works feature prominently in the submission, how might a threshold be reached? Significantly more labour is required to do so. Is it possible to produce 50 feature films in a six-year period? Currently, by awarding a single point for major creative works, the time a researcher has devoted to the development and creation of the work, the effort that the researcher has put in, and the achievement of the work, are not commensurately recognised. Moreover, the labour required to produce such large works takes researchers away from producing other work – which in turn suppresses the profile of the FoR outputs. As a result, those FoR codes can stagnate.

In other words, by awarding 5 points for scholarly books and only 1 point for major works delivered in other forms, the research contributions (and associated labour) of the latter are seriously misrepresented.

There is no logical argument to justify why a major creative work should be valued at 1 while a scholarly book is valued at 5. It is becoming increasingly clear that major works of scholarship, no matter what the media-form of their delivery, should be weighted at 5 points if they contribute an equivalent amount of new knowledge as would a scholarly book.

The phrase ‘amount of new knowledge’ is a metric one. It deals with measurability – which brings us back to an issue that continues to plague creative arts researchers: the capacity to articulate the research contribution (its weight and significance) of each creative output; and the capacity of evaluators to recognise that contribution. In our opinion, the potential of creative practice techniques to access understandings not available via traditional approaches remains under-examined. There is a dearth of discourse about what, precisely, creative practice methods can do.

Towards assessment criteria to measure NTRO scale, impact and success

Australian universities differ when it comes to measuring the research contribution of NTROs. This measuring process is done so as to ascertain whether a researcher is meeting their requisite internal institutional KPIs – hence each specific research ‘output’ is assessed and valued to determine the long-term performance and productivity of individual academics.

To measure this performance most universities have committees that assess each NTRO according to an internal scale. Some universities use a 1 or 5 scale that accords with the metric of ‘traditional’ research; other institutions have used a 1, 3, 5 scale; and a few employ a 1, 2, 3, 4, 5 system.

We think that a survey of these various attempts to calibrate the research contribution of specific NTROs could be useful in developing a more equitable ERA exercise – potentially addressing the comparative ease of rating an output against a notional ‘world standard’ by citation measures as opposed to peer review evaluation measures, and the documented tendency of creative practitioners to judge harshly in their own fields.

For example, Dr Karen Pearlman, senior lecturer and a director of the Creative Documentary Research Centre, has developed the following list of criteria as a starting point for discussion. This list is intended to provide clear but not rigid ways of thinking about creative practice research. A work may have one or more of these things, it will not have all, nor does it need to. 

The first two points allow assessors to evaluate how many research points to allocate against the research time and effort a researcher has devoted to a work.

1. Duration or size of output
2. Effort and duration of process (e.g. fieldwork, archival research, or production processes)

On the 5 point/1 point equivalency model, a 5 under criterion #1 might be: 40 minutes or longer in duration for a film, theatre, dance, sound or music work; a substantial book of poems; a novel or book-length work of creative nonfiction; or a large-scale exhibition. Under criterion #2 we note that it would unlikely for a researcher to be able to produce more than one work of a ‘5-scale’ within a two-year period.

Principles three and four, below, move beyond metrics.

3. Research question clarity: identification of context and kind of question. Is the research seeking knowledge of creative practice, i.e. knowledge of process, form, technologies or techniques? Or is the research question seeking knowledge of something other than creative practice which can only be secured through creative practice, i.e. knowledge in history, anthropology, gender studies, law, medical or natural sciences? Is the context for this clearly articulated and has the researcher identified what s/he/they are addressing?
4. Research investigation depth: Depth of investigation of a question/sub-questions. This is helpful to consider if, say, the length of work does not seem commensurate with content, or when different forms express themselves in radically different durations. 

The clarity of research question, including clarity about the kind of knowledge it is aiming to uncover, also makes evaluation of the research contribution possible. 

Research contribution is identified/described in the research statements to guide the evaluator’s eye towards the knowledge contributions that are not explicitly stated as such. (A documentary generating new knowledge of indigenous perspectives on colonial history, for example, will not state: ‘this is new knowledge of Indigenous perspectives on colonial history’ it will simply provide perspectives, as a documentary can uniquely do.)  Therefore, qualitative considerations of contribution might include:

5. Research’s creative contribution to knowledge: How the creative work as a creative work generates new knowledge.
6. Aesthetic achievement in form, expressiveness, creating and conveying of ideas through means other than direct statement.

Clear articulation of question and contribution also makes possible an evaluation of innovation or whether the knowledge generated is new knowledge. Here it is once again important to determine whether the research is asking a question about creative practice or through creative practice (though it could be argued that ‘well above world standard’ creative practice research will generally do both.)

7. Innovation in creative practice technology; creation or refinement of a method or process; or innovation in form

8. Innovation, insights, apprehension, or comprehension generated through achievement in form

Finally, measures of significance are provided in the research statements.  Except for point nine, which requires some disciplinary expertise to discern, these are more clear-cut evaluative measures. Like a Scimago ranking, these culturally determined measures of significance are used by ARC evaluators to augment their own views with the expertise of the curators, judges, critics or publishers that have evaluated the work publicly.   

9. Significance of the new knowledge to the field
10. Significance of venue, publisher, broadcaster, or funding source
11. Recognition by experts or position of work in the field.

Conclusion

We have given a brief analysis of the apparent underperformance of Creative Arts and Writing (FoR code 36) within the various ERA assessment cycles. Further analysis of how Creative Arts and Writing have performed against other peer-review fields is necessary to check whether this is an isolated case. We suggest that the criteria of evaluation need to be updated in light of the continued evolution in these disciplines as modes of research. We also offer some thoughts on addressing this issue into the 21^st Century, where an increasing amount of research will be produced through, and delivered in, forms other than written words in printed texts.

Addressing the current inequity of research assessment that privileges legacy forms of research at the expense of new, engaging, and innovative forms of research delivery, will also provide a major stimulant to the morale and research capacity-building of creative-arts based enquiry across the tertiary sphere.

We present this as a discussion paper with the aim of developing a consensus across Australian universities for the next ERA.

This article represents the individual authors' views and not those of Macquarie University.

Bronwen Neil FAHA is professor of Ancient History at Macquarie University and immediate past Executive Director for Humanities and Creative Arts at the Australian Research Council.

Karen Pearlman is a senior lecturer in Screen Practice and Production at Macquarie University, and an artist/scholar whose last film won two ATOM awards, an Australian Director’s Guild award and 17 other guild and film festival awards.  

Kate Rossmanith is associate professor of Media and Creative Practice at Macquarie University, and is an ARC Future Fellow (2022 - 2026).

Tom Murray is associate professor of Screen Media and Creative Practice at Macquarie University. He is a current ARC Future Fellow and Director of the Creative Documentary Research Centre at Macquarie University.

Do you have an idea for a story?
Email [email protected]