MEASURING EXCELLENCE IN RESEARCH AND RESEARCH TRAINING
Canberra, 22 June 2004

Assessing excellence in research in terms of private and public benefits
Dr Geoffrey Vaughan

I have got two challenges. One is time, but the other is to discuss issues related to two unrelated organisations, the CRC Committee and the IR&D Board.

I certainly agree with Robin Batterham when he set us off by saying he wants us to ‘integrate’ everything you hear today so that by the end of the workshops we have got some definite parameters that come out of it. Now, I agree with that, but the difficulty I see is, for those mathematically inclined, the limits of the integration – the area under the curve. The limits could either be, let’s say, pure and applied, one end of the integral or the other, public and private, one end of the integral or the other, competitive versus block – it goes on and on – individual versus collaborative. And so that is going to be the biggest difficulty that you face, and indeed it is the biggest difficulty I face in talking about CRCs and the IR&D Board.

Most of my comments will be associated with collaborative research, cooperative research, user-driven research, but even this does have a dependence on individual research at the same time.

Let me just discuss some definitions of what I see being in the public benefits area. I would define this briefly as adoption of research, adoption which will lead to some areas of national benefit – the famous triple bottom line. Generally in Australia it is pretty hard to escape participating under the National Research Priorities. And, even in public benefits, taxation comes into it: taxation at the front end because they are paying for the research, but also at the other end, from outcomes, savings in public expenditure and taxation. A great example of that in public benefit research would be the CRC for Aboriginal Health, where any savings in the Aboriginal health budget have a direct impact on savings in public expenditure down the line.

Employment issues come up, as does export/import replacement. People may not see this as public benefit but it is, because any change that we can have in trade balance allows money for other public benefit activities and that is important as well. Increased skills are very important as a public benefit, and CRCs have done a lot in this, in relation to the research training associated with CRCs. One of the measures, I think, of excellence in research is reflected in excellence in training – increased skills coming out of CRC, training tomorrow’s scientists – and the skills training can’t be avoided in looking at excellence in research.

I guess in looking at public benefits one has got to have performance indicators to measure all of these issues and many more. The important thing I think we have got to see, as others have mentioned, is that we don’t want this to be all-consuming, we don’t want to overdo it and we need to have some performance indicators which do the job without hindering research and research activity. Researchers have still got be left free to roam, there is no question about that in my mind.

In going to another definition, private benefit, under one heading I would put this as commercialisation of innovation. Here we are looking at issues such as new industries, new products, turnover and profit, expansion, efficiency, productivity, competitiveness, technology transfer, income from royalties, licence fees, equity, sale of technology et cetera. And again you have got to look for performance indicators that measure private benefit. To do that in a CRC has some difficulties, but I will try to get to that very shortly.

We are looking at performance indicators right across the board in complex areas. Public indicators are not going to be the same as private indicators, and for CRCs versus the rest, collaborative and cooperative research is not going to necessarily have the same indicators as individual research.

But what can we do with indicators? Well, that is obviously simple and we have all been there before. You can’t think of much else other than inputs, outputs and outcomes. The inputs and outputs are very easy, nevertheless they are valuable because some of the value-add you can get directly by looking at input versus output: let’s say the number of high-quality PhD students coming from a program. That could be directly found from input versus output.

A harder one – we all know it and we have been through it before with the mapping exercise – is the issue of outcomes. On top of performance indicators, I think the measurement of excellence of research and research activity, particularly in CRCs, comes from reviews. These reviews may well be peer review, they may be in-house review, which I hope is going on in every CRC, with the board looking for evaluation of programs and seeing that their strategies, policies and priorities are leading to expectation of outcome, and there are also independent reviews.

Again, we have got to look here at user uptake in the collaborative research end, as well as the other messages that I mentioned of employment opportunity, training et cetera.

Cooperative Research Centres
(Click on image for a larger version)

Unfortunately – or fortunately, whichever way you look at it – the logo I am using there [on slide] I have kept on all of my slides, but let me emphasise that there is no direct relationship between the CRC Committee and the IR&D Board, other than having some members associated with both the committee and the board.

When we look at the Cooperative Research Centres we see the challenge. For ease we have got Cooperative Research Centres across six sectors: manufacturing, information, mining, agriculture, environment and medicine. Obviously, you can’t have a set of indicators which are going to match all of those discipline areas, all of those sectors. There is a large number of CRCs, 71 at the present time, all listed in our current compendium – if you haven’t got one, they have just come out and are available through DEST – so within those 71 you have got a tremendous spectrum of activity, expectation, objectives et cetera, and you have got to be able to have measurement to identify the outcome across those centres.

We have got to be able to measure research excellence at the beginning. In a selection round, when we have many applications, we have got to be able, through the expertise on the CRC Committee – or the expertise, more importantly, on expert panels looking at CRC applications – to look at measuring, identifying and rewarding excellence in research in the outcome of a CRC application.

But it is not the only thing, because we have also got to measure other issues such as participation, user involvement, education, budget – it goes on and on. So although research is important for a CRC, there are certainly other issues that we have got to have.

But we have got a fair bit of experience now. Although there are 71 centres now, if you take every centre that has been established in over eight selection rounds – we are now currently in the ninth selection round – we have in fact started off 144 centres. Some of those are continuing centres, but there have been 144 announcements of a new centre when the selection rounds are added up. They go for seven years; that is 1000 centre-years we will have had by the time the eighth selection round gets through the process. So we are certainly getting the experience. Whether we are getting the mechanism right is yet to be seen.

Cooperative Research Centres
(Click on image for a larger version)

How do we get the mechanism? I have mentioned the CRC compendium, inputs, outputs – there’s a lot of it in there with regard to the funds going in, the people involved, the students involved, the participants involved et cetera. With regard to the inputs, the compendium is excellent for doing that, and certainly at the back it has got not only individual centres but a collection of data and statistics with regard to the input, their research activity et cetera, across the various participants in a centre, whether it be a university, CSIRO or a research institute, government, industry et cetera.

But more importantly, to measure excellence in research of a CRC, we have the annual reports of CRCs. And again, just as we will have had 1000 centre-years, we will have had 1000 annual reports that have come in over the life history of the CRCs, and the ones that are presently up and running. Within these annual reports the centre has to give us a measure of research excellence. Within these reports, under the research programs there are such headings as ‘Targets and Milestones Achieved’, again a reflection of the research excellence; ‘Major Achievements and Outcomes’, a measure of research excellence; and, at the back of each annual report, ‘Performance Indicators’. Now, this is where the performance indicators for any given centre vary enormously, depending on the centre, depending on the sector, depending on the history, stage and evolution of a CRC. But within there every CRC worth its salt has performance indicators with the objectives, the measurement and a score of achievement against them.

So there are mechanisms in place for CRCs which I think do stand the test for many of them to be excellent measures of excellence in research for any given individual centre.

The formal reviews I mentioned before are important. The formal reviews can be at one stage a review across the system as a whole, and within the CRC history of now about 12 years there have been two outstanding reviews: the Myers review known as Changing Research Culture and the Mercer/Stocker report on greater commercialisation in CRCs. But both of those reports looked at excellence of research in CRCs as well as other issues. They looked at excellence of research in a number of ways, looking at the program as a whole, looking at individual centres through case studies, and they came up with overall a good report on research performance in CRCs.

The other reviews are the reviews of individual centres. Within the program, the CRC Committee carries out reviews of centres, currently at the second and fifth years of their seven-year cycle. We are changing that now to one review at the third year of their cycle; we believe that will be adequate and more useful in the system. So we have got that in CRCs.

The other thing that I think is important is case studies, which give outcomes. The only problem with measuring outcomes through case studies is that you might have to wait a while to see the benefit of the research or to measure the excellence of the research that m