MEASURING EXCELLENCE IN RESEARCH AND RESEARCH TRAINING
Canberra, 22 June 2004

Changing research practices in the digital information and communication environment
Mr Colin Steele

Colin Steele organised a National Scholarly Communication Forum, which was held on 1 June. He hosted Sir Gareth Roberts from the UK, the author of the recent review of the UK Research Assessment Exercise. Colin is going to summarise the outcomes of that forum.

Colin Steele

National Scholarly Communication Forum
(Click on image for a larger version)

When I originally was asked to talk I thought it was for 30 minutes, so you are going to get 32 slides in 20 minutes. I would also like to make reference to the NSCF conference, which is extremely difficult to summarise in about 20 minutes for a whole day.

NCSF Forum Presentations
(Click on image for a larger version)

But the presentations by the participants are up on the Australian Academy of the Humanities website (http://www.humanities.org.au/NSCF/current.htm), and I really would refer you to, particularly, Dr Evan Arthur’s presentation at the end, which I will come to. I am taking a few slides from each of them, trying to pick up some of the issues in relation to this conference. So do go to the Australian Academy of the Humanities website.

Sir Gareth Roberts is the Chair of the RAE for the UK, and his so-called light-touch research assessment would be extremely relevant to discussions here that will take place with DEST and as part of this forum. We were involved with this as an outcome of a DEST study, Changing research practice in the digital information and communication environment. The URL is there [on slide] and the full text is on the DEST website. There is a printed copy, and there is also a copy in the ANU Eprints. So it is available for everyone there.

Basically, the key questions from John Houghton were: how do researchers conduct research, what are the major information sources, how do you access and manage information? The whole issue is in scholarly communication: how is it changing in mode to science, interdisciplinary, inter-institutional and global cooperation, and what are the outcomes in scholarly communication and the implications for research infrastructure?

Houghton Responses: Sample Comments
(Click on image for a larger version)

These are just some very quick quotes. John went and interviewed 75 people around Australia; we had focus groups as well. These were leading researchers and some of you in the audience may even have been interviewed by John. All we are trying to indicate there is the changing nature of communication in the sense of the importance of the publications.

And, in defence of DEST, as it understands it the quantitative basis and the publications rating for the DEST points, which are really quite an inhibitor, as Iain McCalman said, was put in by the AVCC and not by DEST. Certainly that quantitative analysis has to go.

Key points: Houghton
(Click on image for a larger version)

Here we have the new modes of knowledge production emerging, new information access and dissemination, the new opportunities for research. What we are arguing is that the system should be viewed holistically: look at the whole process of creation of knowledge, distribution of knowledge and then access to knowledge. Don’t let’s keep breaking it down into small items of peer review, publication, communication; look at the whole system, what is the copyright, et cetera.

Sir Gareth then came in, in terms of his particular emphasis, and the main aspects of research performance involved quantity, quality, impact and utility.

Robin Batterham picked up the three points this morning that Sir Gareth mentioned in the introduction to the National Scholarly Communication Forum. What is the purpose? To allow the funders to assess the quality of research, arising from the investment of public money. I would certainly echo those commentators who said that we do need to get some practical outcomes. And if you have followed, as many of us have, the previous research assessments in the UK, you would know that the way they are moving at the moment does have a lighter touch. You have had the scenarios in New Zealand and Germany coming up as well.

It enables the academic sector to assess its success, informs future strategy and informs funding models. The strategies that have been used historically or prospectively are expert review – and we have seen that expert/peer review, what does that mean? – the metrics, which is going to be extremely important in this presentation of the implications in the publishing industry, self-assessment, mentioned by Iain McCalman, and historical ratings.

The next UK RAE will be an expert/peer review. The census date will be 31 October 2007 with results published in late 2008. Then subsequent exercises will be on a six-year cycle.

You will see the difference from the previous RAE in the 20 to 25 main panels and some 65 to 70 sub-panels. The questions that were addressed to Sir Gareth included how you can do subjects of departments, which are a composite of individuals, across disciplines – the multidiscipline approach, the multi-collaboration approach and the assessment of standard metrics within that.

Within the two-tier panel structure, arrangements need to be put in place to ensure consistency and the sub-panels need to be encouraged to specify metrics appropriate to their discipline. Some of these wordings are almost like mission statements, but in fact there is quite a lot of detail going to have to go into those, and of course the bureaucracy of the panels that will support them. And there will be papers coming out on metrics and others, later this year.

UK Key Decisions 3: Roberts
(Click on image for a larger version)

Other matters were the quality profiles and the four-star assignations, as opposed to five before; the nature of the quality profiles, criterion-referenced. And Sir Gareth worked through some of the issues of how he did that from engineering and the sub-disciplines within that.

We have transcribed his transparencies into PowerPoints where we can, and they are all on the Australian Academy [of the Humanities] website (http://www.humanities.org.au/NSCF/current.htm).

Further Guidance to Follow in 2004
(Click on image for a larger version)

Clear guidance is clearly needed on the applied and practice-led research; we have seen the comments about how one does this in terms of industry focus – or, indeed, as he talked about, the ‘third leg’, people who are working very much with the local and regional communities, such as in the Northern Territory, and how you put the assessment in terms of research evaluation on that.

We come to rating scale descriptors [in second part of slide] and then the eligibility of staff to be included. Sir Gareth alluded to the Times ‘Liar’ Education Supplement, because obviously there had been quite a few references in the Times Higher Ed to various universities – Kings College, London et cetera – actually taking people off into full-time teaching so that they would not get evaluated in the research process. And of course some of the New Zealand dialogue came into that: who are the people who will be assessed for their research output?

Recommendation
(Click on image for a larger version)

That [on slide] is their mission statement. You really can’t disagree with that, but you need to work through the issues.

Recommendation 2
(Click on image for a larger version)

This is about funding councils working alongside the subject communities and research councils to develop discipline-specific performance indicators. Again the question is going to be: what are those performance indicators, other than the generic peer review of the panels? Those are going to be very crucial issues. Some of those issues get reflected in the ANU’s current quality RAE exercise, in terms for example of where the overseas assessors are dialling-in to the research publications database of the top five. And there are very interesting spin-offs with respect to that.

One of the issues he mentioned which was quite important was the reliance on STM [scientific, technical and medical] metrics, the peer review of peer review. We might actually not read any of the articles because they have been published in this journal and they have already been indexed by ISI. I think there are quite a lot of dangers in that, certainly between disciplines. It is relatively easy, it is argued, in chemistry and astronomy, less so in computing and engineering, and certainly fraught with issues in Asian studies, Pacific, social sciences et cetera.

You may have seen a recent Economist article on alleged flaws in the British Medical Journal and Nature. Two Spanish researchers found that 38 per cent of the BMJ and 25 per cent of Nature had flaws in the statistical evidence – which only led to 4 per cent of criticism in the entire conclusions, but we are saying again: what is the nature of peer review operating in journals, particularly when that is volunteered, often by hard-pressed academics?

There could be opportunities for new metrics, especially for social science-humanities, relating to the non-ISI citations, and particularly in relation to repositories.

There is also the contrast that Iain McCalman made about the importance of the monograph and the decline of traditional university presses. Professor Blaise Cronin, of Indiana, has just trawled through all the tenure/promotion credentials at the major universities in America – Stanford, Harvard, Cornell et cetera – and in the humanities the key criterion is the publication of a monograph, which is increasingly difficult for a whole variety of reasons. And then that monograph itself may only sell, on average, 200 copies worldwide. So is that a symbol of tenure, or a communication device?

Professor Oppenheim, when he was here, talked about the publications which lead to tenure – what is called in the scholarly communication arena the ‘Faustian bargain’, where authors give away copyright in return for all the advantages of visibility, citations et cetera – and about citations becoming increasingly important in the assessment of individuals and then, collectively, of departments, and the dominance of ISI, which has really rocketed up in the 1990s since Thomson bought ISI and made it a much more aggressive marketing tool.

ISI is having an enormous influence over the scholarly journal publishing scene. Journal publishers are lobbying hard to have their journals included. For example, only 44 per cent of STM journals, I think, are included in ISI.

There is pressure on ISI to add more open access journals. A new phenomenon called open access journals we will come to later. However, ISI have strict criteria for deciding how a journal gets into their database – again a predominance, as Iain McCalman said, on North America and Europe, and the English language.

Joint authorship in Science Citation Index 1981–1999 [etc.]
(Click on image for a larger version)

This is a graph that John Houghton used in his report, but I just thought I would slip it into the Science Citation Index part in terms of where things are going. You are seeing again, as we saw before, the increase in international collaboration as defined by citations within that period, and also inter-institutional collaboration. And when you split that back down again in the ISI-type things, how do you measure the performance evaluation?

Role of Journal Articles in the Reward System: Oppenheim
(Click on image for a larger version)

I am just putting those straight up. I know I am going across the thing, but in terms of time it is all I can do.

So again we are getting a reinforcement of the traditional journal arena, and those journals are increasingly electronic – 85 per cent of them are electronic. We are not talking necessarily about print or print output.

We are seeing a shift from the metrics to peer judgment. What he was meaning there was that in fact because an article has already been published and peer reviewed, we don’t need to double peer review it. So the metrics are in fact embedded in the peer review.

Publication is now viewed as the objective of research, rather than the dissemination of the knowledge contained within it – which as Houghton showed has often been disseminated well before the publication. The publication is for the accreditation and tenure. And Oppenheim cited that even now, for the 2008, he is getting instructions from his university to start publishing in those journals that get cited. And it may be only two articles, as opposed to five. So people play the game!

Michael Mabe Fiesole 2003
(Click on image for a larger version)

Elsevier, who now are a hundreds of millions of pounds a year profit making multinational, Reed Elsevier, and who publish quite a significant proportion of STM – followed by Springer, Kluwer and others – did a survey in 2003. There were quite considerable differences between the percentages there, in terms of what the academic community wanted. Reputation was the first, refereeing quality, refereeing speed, impact factor, production speed et cetera. And again that presentation is available in a particular website that I can link you to.

It basically said that authors seek out the top journals, and those authors often don’t know by the journal title who the publisher is, whether it is Springer, Kluwer, Reed Elsevier, ANU or whatever. They just know the title of the journal. And then when they become a reader, they change completely from being an author. They rarely want the journal in its totality, but often the article and often free. And none of the financial ramifications of the publishing actually impact on the actual author most of the time – except indirectly, when cancellations have to be made within a university or research organisation. So is this system dysfunctional? And how are we going to cut through those tentacles of dysfunctionality?

We are getting international policy initiatives which you might not be aware of: a declaration in Berlin on open access in October 2003, the Academies of Science in Mexico, the World Summit on the Information Society which had a lot to say about this, an OECD declaration of January 2004, at which Peter McGauran chaired that meeting of the Science Council ministers, largely talking about scientific data but subsequently extrapolated to textual data, and the very, very important UK Science and Technology [Committee] Inquiry into Scientific Publications, which is taking evidence from a huge number of publishers, authors et cetera, research councils, and is going to report in July.

Open access publishing is a bit skeletal at the moment in contrast to the science financial oysters. Elsevier makes a gross profit on its STM publishing of 38 per cent per annum, and we are really talking big business here. When I went to a merchant bank presentation they were saying that investing in Elsevier and some of the bigger multinationals was the best investment around at the moment. But a recent Credit Suisse report said, ‘Well, there may be problems with open access,’ and Elsevier’s share price dipped for quite a bit. It has gone back up again.

A Road Map – The Budapest Open Access Initiative
(Click on image for a larger version)

The Budapest Open Access Initiative: these are the two criteria and within these there are huge subsets of issues, of course – ‘Why should I put it in a repository? I already do in the physics archive,’ et cetera. But there are major issues occurring round the world. Self-archiving is one, depositing refereed journal articles or, indeed, guild literature into their electronic archives. And open access journals are ones which are funded by the granting institution or by other means, particularly the American foundations or others. Part of this was stimulated by the Chair of the Wellcome Trust, in Britain, which has been mentioned: Mark Wolpert funded some major research in Britain, and when he tried to access it last year he couldn’t because his library didn’t subscribe to the Elsevier journal. He got so annoyed at this that he started supporting the open access movement.

The Joint Information Systems Committee, in Britain, has recently done an open access survey across the academic community. About two-thirds were aware of open access, but only by their institutions, 25 per cent. The primary reason they supported open access was belief in the principles of free access to research information, publication fees to come from research grants, then the institution or the library. It is not actually asking the author to pay, like the author pay charges.

There was much ignorance of the institutional eprint archives, but they said if they were told by their funding authority or their university that open access was a condition of grant, most would comply – again the incentives that Houghton identified.

So the NSCF outcomes, before I move on to Evan Arthur’s governmental ones, are to encourage the development of a system built on the principles of open access, recognising considerable differences in research patterns even within the sciences, to raise awareness of international trends, particularly in the context of best practice, open access, copyright et cetera, and to look at the incentive systems. Unless the incentive systems change, the practices probably won’t, if they remain within the traditional ones.

Dr Evan Arthur Concluding Powerpoints
(Click on image for a larger version)

Dr Evan Arthur’s concluding PowerPoints: the next six concluding ones are all his, but his entire presentation is up on the website. This is quite an interesting and very useful presentation, and is probably the most up-to-date presentation of DEST’s thinking at the present time.

Of central importance are Backing Australia’s Ability, national collaborative research, and then most particularly in this area, the Quality and Accessibility Frameworks.

Background to the QAF
(Click on image for a larger version)

To come back to quality: how do we judge quality, and the quality of Australian research, and the evaluation of knowledge and innovation reforms we have heard talked about? There was also the fact that the mechanisms for research fund allocation were often driven by numerical inputs and not by the quality of the outputs.

Background to the QAF 2
(Click on image for a larger version)

The government has not taken any decision to implement QAF at the moment, nor to link it to allocation of funds. However, the consultations that were alluded to and will be taking place are obviously occurring, and will cover the universities and publicly funded research agencies.

Key questions about a QAF
(Click on image for a larger version)

The issues for later on today include: what types of quality could be assessed, what are the units of assessment, how is it going to work, can we develop a richer set of performance data than currently available? John Houghton identified some of the issues that Iain McCalman did, about how you can assess multimedia presentations, the differences in non-book, non-textual, and what should be the role of peer review, as we have indicated before.

Relevance of accessibility
(Click on image for a larger version)

We were also talking about making Australian research more accessible, a variation of the Wellcome Trust. There is a major policy interest in improving the accessibility of the research, making it available to all – this is part of this global movement about making material available to the Third World, the developing world – and making research quality more apparent and research results more accessible in parallel, a very crucial point that he brought out.

Next Steps
(Click on image for a larger version)

As to the next steps by the government, there is a need to address the issues of access and scholarly information – picking up some of those world trends that we mentioned before – and government may be prepared (you can see some Yes, Minister type phrases) to commit significantly more resources, focusing on research quality and access. There is also a consciousness that this agenda needs to be fully shared by the sector.

This National Academies Forum meeting today plays a very important part in that debate, and what I was trying to do today was to say that there is a very large structural area around here, and things like ISI citations, publications, are a key part – rightly or wrongly – of the research assessment. We need to look at the whole issue of scholarly communication in a holistic fashion.

Questions/discussion

Chris Blackall – Thanks, Colin, for a very comprehensive survey of this complex area. I did get to see Evan Arthur’s slides, and he put up a very interesting slide where for a senior bureaucrat he did something very, very unusual: he tried to sketch a scenario where people would be publishing their work in eprint archives or digital libraries, and the metadata from those libraries could be collected by an agency in lieu of, perhaps, ISI data. What do you think of the possibilities of developing such a system in Australia, given the stranglehold of ISI on this critical data and the fact that it is so incomplete for many, many areas, and key strategic areas for Australia?

Colin Steele – There is a group in New York called the Open Society Institute, funded by George Soros. One of the issues that currently a working party that I am on is looking at is alternative metrics to ISI. That is not to say ISI will go out of business, because de facto, Sir Gareth said, they are in fact becoming a metric that is acceptable even though the bibliographical cleansing of their data that is clearly needed, as we saw in the Australian science laureates, needs to be intensively analysed.

So there are a whole series of other metrics that can be used in the institutional repositories, that Professor Stevan Harnad, from Montreal, has actually mentioned and are available. But the problem is that a lot of those repositories are not actually populated in great depth.

Now, ISI as I understand it – and Linda Butler is in the audience, and Vic Elliott is visiting ISI next week, in Philadelphia – are looking at including, as a separate exercise, beta testing of archives, material that is not in the Web of Science et cetera, to develop a new model which they can sell at enormous cost. Reed Elsevier have already announced Scopus, which was beta tested at Oxford University, Lausanne and Toronto and which will be a rival to Web of Science. So next year, irrespective of these other metrics, there will be two very expensive indices that your university will have to buy, or your research institution will need to access.

So there is going to be a huge increase in the number of metrics. The issue will be, though, whether the government agencies that do the research assessment accept the metrics. A lot of the metrics that are being done by Harnad and others, in Southampton University in particular, are actually very good ones in terms of downloads, digital hits, cross-references, the group call counter et cetera, but one of the problems will be whether they are going to be accepted, and whether they are the de facto metric – which doesn’t even need to be analysed – that Sir Gareth Roberts was alluding to. So the UK metrics paper, later this year, is going to be very interesting.

Marie Carroll – You mentioned the Australian National University's 2004 quality review. Those of us who are struggling with this process – I am one of them, and Linda Butler is another – have come to the conclusion that there have got to be better ways of measuring excellence for those who write books. I know that Iain McCalman mentioned this, and you yourself mentioned it just now.

Linda can probably speak for this, but I understand that there are ways in which there is an equivalent of an ISI citation for books, or some institutions are trying to do this now, so that books which, as we heard this morning, might take three or four or five years to have their impact can actually be assessed in a meaningful way, in the same way that the perhaps more peripheral work that comes through journal articles is assessed through ISI now.

Colin Steele –Linda Butler is doing a study on book citation in ISI citations. The book one is a really interesting one, because the average sale – and we are talking about research material here, not teaching and learning or textbooks – of a social science-humanities monograph around the world is 350 copies. So again the time of reviewing those books in the review journals is usually pretty slow.

As we have seen from the American study, the book is the symbol of the tenure/promotion, rather than the dissemination of knowledge. What some of the commercial publishers, like Oxford University Press and a number of others, are doing is going to the electronic book with print-on-demand. Each chapter will have metadata, each chapter will have an abstract written by the author – Oxford Scholarship Online does this already – and they will be citing the chapters in the book rather than the book itself in some of their methodologies.

The other ones would be, ‘Well if we do say that a book is very important, if there is only one copy of it in Australia and it is in print, how is it going to impact too much on your local community?’ So there are major issues to look at here, and one of the things that the Open Society Institute is trying to do is to find the new metrics for social science-humanities.

Linda Butler – I would just give a brief description of one of the things that we are studying in a current project in our group – which is actually an ARC-funded linkage project, with DEST as the linkage partner. It will be familiar to many of you that if you go in and search for a book in the Web of Science you can actually get citation counts to a book or a book chapter or other non-journal sources.

The problem in the past has been that ISI does not aggregate those in any meaningful way. All they concentrate on is journals and journal impact factors and journal citations. However, there is a wealth of data in there, and what we are looking at doing is trying to trawl through that data and see if we can come up with some meaningful benchmarks for books and book chapters in the social sciences and humanities.

We hope within the next 12 months or so to have some idea whether it is possible, and whether it can be used as a performance measure or not.