MEASURING EXCELLENCE IN RESEARCH AND RESEARCH TRAINING
Canberra, 22 June 2004

Changing research practices in the digital information and communication environment
Mr Colin Steele

Colin Steele organised a National Scholarly Communication Forum, which was held on 1 June. He hosted Sir Gareth Roberts from the UK, the author of the recent review of the UK Research Assessment Exercise. Colin is going to summarise the outcomes of that forum.

Colin Steele

National Scholarly Communication Forum
(Click on image for a larger version)

When I originally was asked to talk I thought it was for 30 minutes, so you are going to get 32 slides in 20 minutes. I would also like to make reference to the NSCF conference, which is extremely difficult to summarise in about 20 minutes for a whole day.

NCSF Forum Presentations
(Click on image for a larger version)

But the presentations by the participants are up on the Australian Academy of the Humanities website (http://www.humanities.org.au/NSCF/current.htm), and I really would refer you to, particularly, Dr Evan Arthur’s presentation at the end, which I will come to. I am taking a few slides from each of them, trying to pick up some of the issues in relation to this conference. So do go to the Australian Academy of the Humanities website.

Sir Gareth Roberts is the Chair of the RAE for the UK, and his so-called light-touch research assessment would be extremely relevant to discussions here that will take place with DEST and as part of this forum. We were involved with this as an outcome of a DEST study, Changing research practice in the digital information and communication environment. The URL is there [on slide] and the full text is on the DEST website. There is a printed copy, and there is also a copy in the ANU Eprints. So it is available for everyone there.

Basically, the key questions from John Houghton were: how do researchers conduct research, what are the major information sources, how do you access and manage information? The whole issue is in scholarly communication: how is it changing in mode to science, interdisciplinary, inter-institutional and global cooperation, and what are the outcomes in scholarly communication and the implications for research infrastructure?

Houghton Responses: Sample Comments
(Click on image for a larger version)

These are just some very quick quotes. John went and interviewed 75 people around Australia; we had focus groups as well. These were leading researchers and some of you in the audience may even have been interviewed by John. All we are trying to indicate there is the changing nature of communication in the sense of the importance of the publications.

And, in defence of DEST, as it understands it the quantitative basis and the publications rating for the DEST points, which are really quite an inhibitor, as Iain McCalman said, was put in by the AVCC and not by DEST. Certainly that quantitative analysis has to go.

Key points: Houghton
(Click on image for a larger version)

Here we have the new modes of knowledge production emerging, new information access and dissemination, the new opportunities for research. What we are arguing is that the system should be viewed holistically: look at the whole process of creation of knowledge, distribution of knowledge and then access to knowledge. Don’t let’s keep breaking it down into small items of peer review, publication, communication; look at the whole system, what is the copyright, et cetera.

Sir Gareth then came in, in terms of his particular emphasis, and the main aspects of research performance involved quantity, quality, impact and utility.

Robin Batterham picked up the three points this morning that Sir Gareth mentioned in the introduction to the National Scholarly Communication Forum. What is the purpose? To allow the funders to assess the quality of research, arising from the investment of public money. I would certainly echo those commentators who said that we do need to get some practical outcomes. And if you have followed, as many of us have, the previous research assessments in the UK, you would know that the way they are moving at the moment does have a lighter touch. You have had the scenarios in New Zealand and Germany coming up as well.

It enables the academic sector to assess its success, informs future strategy and informs funding models. The strategies that have been used historically or prospectively are expert review – and we have seen that expert/peer review, what does that mean? – the metrics, which is going to be extremely important in this presentation of the implications in the publishing industry, self-assessment, mentioned by Iain McCalman, and historical ratings.

The next UK RAE will be an expert/peer review. The census date will be 31 October 2007 with results published in late 2008. Then subsequent exercises will be on a six-year cycle.

You will see the difference from the previous RAE in the 20 to 25 main panels and some 65 to 70 sub-panels. The questions that were addressed to Sir Gareth included how you can do subjects of departments, which are a composite of individuals, across disciplines – the multidiscipline approach, the multi-collaboration approach and the assessment of standard metrics within that.

Within the two-tier panel structure, arrangements need to be put in place to ensure consistency and the sub-panels need to be encouraged to specify metrics appropriate to their discipline. Some of these wordings are almost like mission statements, but in fact there is quite a lot of detail going to have to go into those, and of course the bureaucracy of the panels that will support them. And there will be papers coming out on metrics and others, later this year.

UK Key Decisions 3: Roberts
(Click on image for a larger version)

Other matters were the quality profiles and the four-star assignations, as opposed to five before; the nature of the quality profiles, criterion-referenced. And Sir Gareth worked through some of the issues of how he did that from engineering and the sub-disciplines within that.

We have transcribed his transparencies into PowerPoints where we can, and they are all on the Australian Academy [of the Humanities] website (http://www.humanities.org.au/NSCF/current.htm).

Further Guidance to Follow in 2004
(Click on image for a larger version)

Clear guidance is clearly needed on the applied and practice-led research; we have seen the comments about how one does this in terms of industry focus – or, indeed, as he talked about, the ‘third leg’, people who are working very much with the local and regional communities, such as in the Northern Territory, and how you put the assessment in terms of research evaluation on that.

We come to rating scale descriptors [in second part of slide] and then the eligibility of staff to be included. Sir Gareth alluded to the Times ‘Liar’ Education Supplement, because obviously there had been quite a few references in the Times Higher Ed to various universities – Kings College, London et cetera – actually taking people off into full-time teaching so that they would not get evaluated in the research process. And of course some of the New Zealand dialogue came into that: who are the people who will be assessed for their research output?

Recommendation
(Click on image for a larger version)

That [on slide] is their mission statement. You really can’t disagree with that, but you need to work through the issues.

Recommendation 2
(Click on image for a larger version)

This is about funding councils working alongside the subject communities and research councils to develop discipline-specific performance indicators. Again the question is going to be: what are those performance indicators, other than the generic peer review of the panels? Those are going to be very crucial issues. Some of those issues get reflected in the ANU’s current quality RAE exercise, in terms for example of where the overseas assessors are dialling-in to the research publications database of the top five. And there are very interesting spin-offs with respect to that.

One of the issues he mentioned which was quite important was the reliance on STM [scientific, technical and medical] metrics, the peer review of peer review. We might actually not read any of the articles because they have been published in this journal and they have already been indexed by ISI. I think there are quite a lot of dangers in that, certainly between disciplines. It is relatively easy, it is argued, in chemistry and astronomy, less so in computing and engineering, and certainly fraught with issues in Asian studies, Pacific, social sciences et cetera.

You may have seen a recent Economist article on alleged flaws in the British Medical Journal and Nature. Two Spanish researchers found that 38 per cent of the BMJ and 25 per cent of Nature had flaws in the statistical evidence – which only led to 4 per cent of criticism in the entire conclusions, but we are saying again: what is the nature of peer review operating in journals, particularly when that is volunteered, often by hard-pressed academi