MEASURING EXCELLENCE IN RESEARCH AND RESEARCH TRAINING
Canberra, 22 June 2004

Excellence in the humanities, creative arts and media
Professor Iain McCalman

What is excellent research in the humanities and arts?

Innovative or original work that moves a field of knowledge forward, breaks fresh ground, stimulates further developments. It may do this by finding new areas to explore or alternative ways of examining existing problems, for example by discovering new data, interpreting new or existing bodies of knowledge in a new way; by systematically challenging the existing canons or ideologies; by experimenting with materials and ideas in search of new formulations; by questioning methodologies and forming new hypotheses and theories, by testing and implementing models and paradigms; or by acts of creative practice or performance.

Work that makes a substantive contribution to an existing field of study or discipline. Important to stress that this can and does include scholarship which is sometimes and erroneously denigrated by comparison with what is deemed research. Scholarship, as defined in the latest British RAE as, ‘the creation, development and maintenance of the intellectual infrastructure of subjects and disciplines, in forms such as dictionaries, scholarly and critical editions, catalogues and contributions to major research data-bases.

Work that feeds the development of new technologies or the application of technologies that bring benefit to peoples.

Work that is wealth creating or that represents good value for the money invested.

Excellent research in the humanities and arts may have an immediate and recognisable use-value in terms of wealth generation or social benefit or it may be ostensibly non-utilitarian. It may be pure, or academically driven investigative research, or applied research which is policy driven. It may have an ultimate significance that lies not in its applicability to any given current set of circumstances, but in its enhancement of the cultural or spiritual dimensions of the collective life of peoples. Its significance and effects may be long-term and subtle, that is in contributing to shifts of thought and values or long-term and speculative whose results may not reach immediate fruition.

How do we achieve it?

We must identify and reward research of a good standard as well as the best possible research.

Excellent research must be tested and evaluated in the public domain, through publication or other comparable means of dissemination ranging from print, through electronic and digital forms, to performance or exhibition. It needs to be assessed both by national and international peers, according to clear, fair and agreed benchmarks.

If it includes a substantial component of practice or is embedded in creative practice, as in the visual, creative and performing arts (and similar to law, engineering and parts of medicine), it must bring enhancements of knowledge and understanding in the discipline or related disciplines and incorporate a scholarly apparatus and record of research activity that enables other researchers in the discipline or related disciplines to assess its value and significance, and its methods.

We need to foster a research culture or environment that is self-reflexive, accountable and innovative. It should encourage and allow individual scholars to evolve, formulate, publish and disseminate their research both for a national and international scholarly community of specialists and for general publics, as well as research which bridges the gap between scholarly communities and publics.

It should, where relevant, support and encourage team-based and collaborative research, interdisciplinary and multi-disciplinary as well as single discipline research, and cross-institutional as well as single institutional collaborations.

It must attract postgraduate research students and encourage their completions, and attracts and fosters the work of postdoctoral students and of early career teaching staff. Such an environment must have incentives to improve future research performance for existing staff and improve the supply and demand of researchers. Assessments of research excellence must be dynamic, rewarding past performance and prospective plans or projects. It must be flexible so as to enable new fields to develop.

How do we measure such excellence?

There are four broad methods which have been variously used around the world to measure research excellence in the humanities and creative and performing arts: peer review, such in the British RAE system: self-assessment; historical ratings and algorithmic or quantitative measures. Want to focus particularly on the last because it has been the key system in Australia and because both the British and United States higher education systems are moving more and more in that direction.

Algorithmic assessment.

I am not going to waste much time on the crudely quantitative system currently employed by DEST except to say that by failing to incorporate any real component of qualitative assessment, it is worse than useless in that it skews research practices in directions that are almost uniformly calculated to prevent a healthy and innovative research environment. It rewards mediocrity than quality. I will give a few examples from my own research oeuvre.

Quantitative indicators.

One quantitative indicator commonly employed is the size of research grant income, but this is not necessarily a measure of research quality in a humanities and arts context where work is less dependent than science, engineering and technology on factors such as expensive equipment. In the arts and humanities, the number and competitive bases of research grants may be a better indicator than their size.

Another quantitative measure which has gained fairly wide acceptance is the numbers of postgraduate students attracted, and their completion rates, and the numbers of postdoctoral students attracted.

A third quantitative measure which has not been widely developed but which is being tested by various of the Research Councils in Britain and by the RAE assessors is a series of esteem indicators, such as membership of learned societies; invited national and national keynote lectures; editorial boards of scholarly journals; membership of learned boards such as the ARC, government inquiries, etc. These can be adjusted according to the particular shape of disciplines, to include performances, exhibitions, consultancies, etc.

Citation indices.

This system was invented more than 30 years ago by Eugene Garfield who set up the Institute for Scientific Information based at Philadelphia in the USA, and which has published three citation indices: in science, in arts and humanities and social sciences. These are alphabetical lists of scholars whose writings are mentioned in a body of scholarly periodicals. By counting the number of times a particular researcher and their work has been mentioned in this literature you arrive at a number which is interpreted as proxy of research excellence. A series of studies, especially in librarian studies field, conducted by researchers at Loughborough University, have claimed a close correlation between such indices and other systems of assessment, including peer evaluation. These studies have argued that the correlation holds across a very wide field of subjects and disciplines, including those in