MEASURING EXCELLENCE IN RESEARCH AND RESEARCH TRAINING
Canberra, 22 June 2004

Assessing excellence in research in terms of private and public benefits
Dr Geoffrey Vaughan

I have got two challenges. One is time, but the other is to discuss issues related to two unrelated organisations, the CRC Committee and the IR&D Board.

I certainly agree with Robin Batterham when he set us off by saying he wants us to ‘integrate’ everything you hear today so that by the end of the workshops we have got some definite parameters that come out of it. Now, I agree with that, but the difficulty I see is, for those mathematically inclined, the limits of the integration – the area under the curve. The limits could either be, let’s say, pure and applied, one end of the integral or the other, public and private, one end of the integral or the other, competitive versus block – it goes on and on – individual versus collaborative. And so that is going to be the biggest difficulty that you face, and indeed it is the biggest difficulty I face in talking about CRCs and the IR&D Board.

Most of my comments will be associated with collaborative research, cooperative research, user-driven research, but even this does have a dependence on individual research at the same time.

Let me just discuss some definitions of what I see being in the public benefits area. I would define this briefly as adoption of research, adoption which will lead to some areas of national benefit – the famous triple bottom line. Generally in Australia it is pretty hard to escape participating under the National Research Priorities. And, even in public benefits, taxation comes into it: taxation at the front end because they are paying for the research, but also at the other end, from outcomes, savings in public expenditure and taxation. A great example of that in public benefit research would be the CRC for Aboriginal Health, where any savings in the Aboriginal health budget have a direct impact on savings in public expenditure down the line.

Employment issues come up, as does export/import replacement. People may not see this as public benefit but it is, because any change that we can have in trade balance allows money for other public benefit activities and that is important as well. Increased skills are very important as a public benefit, and CRCs have done a lot in this, in relation to the research training associated with CRCs. One of the measures, I think, of excellence in research is reflected in excellence in training – increased skills coming out of CRC, training tomorrow’s scientists – and the skills training can’t be avoided in looking at excellence in research.

I guess in looking at public benefits one has got to have performance indicators to measure all of these issues and many more. The important thing I think we have got to see, as others have mentioned, is that we don’t want this to be all-consuming, we don’t want to overdo it and we need to have some performance indicators which do the job without hindering research and research activity. Researchers have still got be left free to roam, there is no question about that in my mind.

In going to another definition, private benefit, under one heading I would put this as commercialisation of innovation. Here we are looking at issues such as new industries, new products, turnover and profit, expansion, efficiency, productivity, competitiveness, technology transfer, income from royalties, licence fees, equity, sale of technology et cetera. And again you have got to look for performance indicators that measure private benefit. To do that in a CRC has some difficulties, but I will try to get to that very shortly.

We are looking at performance indicators right across the board in complex areas. Public indicators are not going to be the same as private indicators, and for CRCs versus the rest, collaborative and cooperative research is not going to necessarily have the same indicators as individual research.

But what can we do with indicators? Well, that is obviously simple and we have all been there before. You can’t think of much else other than inputs, outputs and outcomes. The inputs and outputs are very easy, nevertheless they are valuable because some of the value-add you can get directly by looking at input versus output: let’s say the number of high-quality PhD students coming from a program. That could be directly found from input versus output.

A harder one – we all know it and we have been through it before with the mapping exercise – is the issue of outcomes. On top of performance indicators, I think the measurement of excellence of research and research activity, particularly in CRCs, comes from reviews. These reviews may well be peer review, they may be in-house review, which I hope is going on in every CRC, with the board looking for evaluation of programs and seeing that their strategies, policies and priorities are leading to expectation of outcome, and there are also independent reviews.

Again, we have got to look here at user uptake in the collaborative research end, as well as the other messages that I mentioned of employment opportunity, training et cetera.

Cooperative Research Centres
(Click on image for a larger version)

Unfortunately – or fortunately, whichever way you look at it – the logo I am using there [on slide] I have kept on all of my slides, but let me emphasise that there is no direct relationship between the CRC Committee and the IR&D Board, other than having some members associated with both the committee and the board.

When we look at the Cooperative Research Centres we see the challenge. For ease we have got Cooperative Research Centres across six sectors: manufacturing, information, mining, agriculture, environment and medicine. Obviously, you can’t have a set of indicators which are going to match all of those discipline areas, all of those sectors. There is a large number of CRCs, 71 at the present time, all listed in our current compendium – if you haven’t got one, they have just come out and are available through DEST – so within those 71 you have got a tremendous spectrum of activity, expectation, objectives et cetera, and you have got to be able to have measurement to identify the outcome across those centres.

We have got to be able to measure research excellence at the beginning. In a selection round, when we have many applications, we have got to be able, through the expertise on the CRC Committee – or the expertise, more importantly, on expert panels looking at CRC applications – to look at measuring, identifying and rewarding excellence in research in the outcome of a CRC application.

But it is not the only thing, because we have also got to measure other issues such as participation, user involvement, education, budget – it goes on and on. So although research is important for a CRC, there are certainly other issues that we have got to have.

But we have got a fair bit of experience now. Although there are 71 centres now, if you take every centre that has been established in over eight selection rounds – we are now currently in the ninth selection round – we have in fact started off 144 centres. Some of those are continuing centres, but there have been 144 announcements of a new centre when the selection rounds are added up. They go for seven years; that is 1000 centre-years we will have had by the time the eighth selection round gets through the process. So we are certainly getting the experience. Whether we are getting the mechanism right is yet to be seen.

Cooperative Research Centres
(Click on image for a larger version)

How do we get the mechanism? I have mentioned the CRC compendium, inputs, outputs – there’s a lot of it in there with regard to the funds going in, the people involved, the students involved, the participants involved et cetera. With regard to the inputs, the compendium is excellent for doing that, and certainly at the back it has got not only individual centres but a collection of data and statistics with regard to the input, their research activity et cetera, across the various participants in a centre, whether it be a university, CSIRO or a research institute, government, industry et cetera.

But more importantly, to measure excellence in research of a CRC, we have the annual reports of CRCs. And again, just as we will have had 1000 centre-years, we will have had 1000 annual reports that have come in over the life history of the CRCs, and the ones that are presently up and running. Within these annual reports the centre has to give us a measure of research excellence. Within these reports, under the research programs there are such headings as ‘Targets and Milestones Achieved’, again a reflection of the research excellence; ‘Major Achievements and Outcomes’, a measure of research excellence; and, at the back of each annual report, ‘Performance Indicators’. Now, this is where the performance indicators for any given centre vary enormously, depending on the centre, depending on the sector, depending on the history, stage and evolution of a CRC. But within there every CRC worth its salt has performance indicators with the objectives, the measurement and a score of achievement against them.

So there are mechanisms in place for CRCs which I think do stand the test for many of them to be excellent measures of excellence in research for any given individual centre.

The formal reviews I mentioned before are important. The formal reviews can be at one stage a review across the system as a whole, and within the CRC history of now about 12 years there have been two outstanding reviews: the Myers review known as Changing Research Culture and the Mercer/Stocker report on greater commercialisation in CRCs. But both of those reports looked at excellence of research in CRCs as well as other issues. They looked at excellence of research in a number of ways, looking at the program as a whole, looking at individual centres through case studies, and they came up with overall a good report on research performance in CRCs.

The other reviews are the reviews of individual centres. Within the program, the CRC Committee carries out reviews of centres, currently at the second and fifth years of their seven-year cycle. We are changing that now to one review at the third year of their cycle; we believe that will be adequate and more useful in the system. So we have got that in CRCs.

The other thing that I think is important is case studies, which give outcomes. The only problem with measuring outcomes through case studies is that you might have to wait a while to see the benefit of the research or to measure the excellence of the research that measures that outcome. Right at the top of the ladder you can see that in the Nobel Prize. Very often, the Nobel Prize is offered years down the track when the excellence of that research has ultimately been evaluated, though it may not have seemed of much use at the time when it was carried out.

Case studies are key and critical. In the CRC program – and this is an attempt to get to outcome measurement of excellence in research – the Cooperative Research Centres Association, along with the Department of Education, Science and Training (or, before that, the Department of Industry) have put out booklets with outcomes of CRC research. It is generally under a heading of ‘Recent Highlights of the Cooperative Research Centre Program’ but it goes through people, ideas, enterprise, research et cetera, and there have been a number of attempts to tell the story of the CRC program.

Another way of doing it, again through the CRC program – and we have just looked at this in very recent times – is another publication, Science in Action. Last year, every week there was a news release of outcomes from CRCs; they have now been put together in one compendium for the year, again telling the story. And from this you can get a measure of excellence in research, as well as excellence in overall performance, of any individual CRC. The big difficulty in telling the story is to know who is going to take it up, who is going to read it. That is one of the problems; we are trying to analyse the benefits of that in looking at telling the story of CRCs.

One outcome was that the rural press certainly took it up. Last year, as outcomes from this document, many items came through the Weekly Times, Land et cetera, the rural newspapers, and that was one area where telling the story came through very loud and clear.

Let me now just quickly go to the IR&D Board. Again I apologise that I have got the CRC logo on the IR&D area, but I just happened to use the same master slide right through.

Like the CRC program, the IR&D program has a whole spectrum of activities. These are the programs which involve research activity run by the IR&D Board: the Start program, the tax concession, the BIF [Biotechnology Innovation Fund], the COMET [Commercialising Emerging Technologies], the IIF [Innovation Investment Fund], the Pre-Seed [Fund], the REEF [Renewable Energy Equity Fund] et cetera. They all have a research component to them, and the IR&D Board is interested in measuring research performance within those components and going about it. But the difficulty is again the difference between the programs that are run by the board, the difference between individual participants in any given program, and the volume of the program.

An example is the R&D Tax Concession. There are close to 5000 companies in Australia getting a benefit at one level or another through tax concession. And that is just an eligibility program; it is not a competitive program at all. It is a straight grant program or eligibility program, and how you measure research excellence across 5000 applicants or participants becomes very difficult. Nevertheless, the IR&D Board is supposed to do something about it.

And it does it through the annual report. As a statutory requirement the IR&D Board is required to publish an annual report, and within the annual report for each and every one of those programs that we have shown there we have got objectives, budget/expenditure, program performance and outcomes. The first three items there – objectives, budget/expenditure and program performance – are basically an evaluation of the IR&D Board’s role, the IR&D as such, but nevertheless we have got to report on programs and the individual participants, and that is where we rely very much on outcomes. Throughout the report, again telling the story, there are case studies showing where successful applications, successful ventures, successful outcomes have come from the IR&D program.

So the one thing I guess I am putting forward is that one of the best measures of outcomes which everyone finds is very difficult is, I think, best given as a case study. But again I warn you that that has got to be done in light of time, sometimes, because there is sometimes a lapse of time before the benefit and outcome of an application is found.

On measuring research excellence I should just say that those investment funds – the IIF funds, the Pre-Seed funds et cetera – have the challenge of measuring excellence. Whereas an application for a CRC or a Start grant or whatever takes a long time, a long evaluation phase et cetera, if you want to go into an investment area and go to a venture capitalist, you have got to sell him your story in 10 minutes. If you haven’t convinced him of the worth of investing in your research in 10 minutes, forget it. All the rest of the stuff won’t count, because he or she makes up their mind in the first 10 minutes of that encounter. So that is another interest issue associated with measuring research excellence.

Anyway, the IR&D Board is certainly interested in outcome research. Out of the recent agenda papers we can dig out agenda after agenda after agenda on performance measurement as far as the IR&D Board programs are concerned. A current agenda item which took a long part of a meeting was ‘Outcome performance measurement’. We looked at issues of what value we were going to get out of that, and such things as executive alert, so that people at the top end of the administration of the program would know if a research grant was going down the drain, if the research wasn’t meeting its objectives, if outcomes were unlikely to be met.

We wanted to make certain that the research being carried out was relevant to the program, meeting the IR&D Board objectives, which in the short term are meeting the National Research Priorities and the national benefits as defined by the board. And we wanted to consider how we can best measure outcomes of the very many programs operated by the IR&D Board.

It goes wider than the board because it goes right across the Department of Industry, Tourism and Resources. The department as a whole, I think, is getting a little bit too heavily directed towards performance, because it is starting to become, in my mind, a little bit all-consuming. But the department itself is looking at the external interest in outcome indicators, and there the audit department, the government itself, individual politicians are hounded the department: ‘Give us outcomes. Let us know what you are measuring. How do we see whether it is effective?’

There is an evaluation culture being established in DITR, from the Secretary downwards. There are corridor messages going out saying, ‘We’ve got to be able to measure performance. We’ve got to be able to measure excellence. We’ve got to be able to measure right across our programs, not just the IR&D program but every program in industry.’

There is an outcome indicator working group. They have got to match their reporting to Backing Australia’s Ability requirements of reporting. They have got indicator workshops, matching the department’s key performance indicators (KPIs), including the IR&D Board KPI, with what is proposed as Backing Australia’s Ability KPIs, so we are not reinventing the wheel – there are no gaps, there is not duplication et cetera. They are setting targets, they are looking at international benchmarking.

So they are going to a hell of a lot of effort to see that they can measure performance across the research activities, in particular, of the DITR programs. I repeat that I am a bit concerned that there is too much energy going into this. There is too much in resources and time being taken up which I think might be able to be spent in better areas.

Looking at it collectively, I reckon the best performance indicator of our research, at any level in Australia but particularly in collaborative research, really comes down to looking at four outcomes.

We are looking at BERD, business expenditure in research and development. That has got to come up in Australia, and it will come up if business will support research – admittedly, applied research, strategic research – in Australia.

We are looking at employment. Through our research we have obviously got to have increased employment across the country and have a good social outcome from research, as could be indicated from employment.

We have got to look at good balance of payments, so that we are not totally consumed with being a Third World country or African republic or whatever they used to call them – that we do have a very high order of economic stability and ability.

And we have got to look at investor confidence. Whether the investors be at the end of the IIF-type funds, the simple investors, or the long-term stock exchange investors, the insurance and superannuation companies et cetera, we have got to have that confidence to maintain a healthy investor confidence across the nation.

I have this theory, that if you look after BERD, employment, balance of payments and investor confidence in the area of collaborative research, cooperative research, industry research, you are doing pretty well.

Questions/discussion

Doug Hilton – You mentioned that the CRC system is very complicated, in that it is funding a lot of areas, a lot of different sectors, with a lot of different management structures and missions. I am wondering how, with the different reviews that you do and then through different selection rounds, you get consistency of evaluation across your different committees that do the reviewing, and then how you integrate the funding choices that you make across the different sectors when you come to give a list to government in your recommendations.

Geoffrey Vaughan – Yes, it has been a challenge, it has been a difficulty, but we depend very heavily on expert committees. One of the chairmen of our expert committees, Max Brennan, who looked after the physical sciences committee, is here. Max picks people with expertise in the field, they go out and visit the centre, the centre itself has a research survey done with external independence, and the systems come together like that as the second and fifth year review.

However, because of some issues and some challenges and some problems we have reviewed it. We are now going to have just one third-year review, and the review is going to be carried out by the centre. The centre board is going to take the role and responsibility of effective evaluation. It will have to appoint an independent outside assessing team, it will have to appoint its terms of reference to determine how best it is going to measure its performance, including research performance, and it will have to report to the CRC Committee when the review has been taken.

In three years they have had long enough to get going. They have got enough time for us, if we want to, to close them down or to tell them that they are on the wrong track and they have got to do this, that and the other, and do it that way.

Doug Hilton – But for funding new centres, how do you then decide and integrate priorities in physical sciences versus IT versus photonics versus medical, when you come up with your funding list?

Geoffrey Vaughan – That is one of the challenges. It depends on how much money is asked for, it depends on how we see that money as value for money to get the job done, it depends on how much support is coming in from the participants. And you get a feel, in looking at a CRC application, that one has got a better package than the other – one is more competitive than the other, one should be further evaluated through some external evaluation, peer evaluation, expert evaluation, going and visiting them, having an interview. So it comes together that way.

Then we are interested in how they are going to measure their performance to tell us what they are getting out of the program.

Michael Barber – Thanks, Geoff, for a good description of the CRC processes. The CRC processes, however, are driven almost exclusively, as you described, by the selection of centres up front. It has been a comment that there has never been a structural consideration of the CRCs towards building the Australian economy in a more holistic or strategic sense. Have you any comment to make to that sort of claim?

Geoffrey Vaughan – Well, the applicants have got to show how they are going to add value, how they are going to participate to the national benefit, how they may be participating towards Australia’s National Research Priorities. And I think if applicants have got to do that, they are seeing how they are going to be participating at the best level in the community at large.

Michael Barber – I guess it is partly linked to the previous question. Would it have worried you if we had ended up with all the CRCs in a particular round focused, say, on the agricultural industry?

Geoffrey Vaughan – It would certainly worry me, because I would think that something was wrong with the other five sectors that they couldn’t get up excellence applications. History shows that we have never had a ration, we have never had a question of how many we are getting here or there; it has all fallen into place by natural events. We ultimately get a league ladder of applications to refer to the minister – we only give advice to the minister, we don’t make the final decisions, by any means, but we give him a league ladder of where they are – and spotted down that league ladder, it so happens, there has been a pretty good spread.

Now, some of them are a bit out of kilter. You would have noticed a lot more in agriculture and environment than, let’s say, IT and mining. But I guess that reflects the number of applications you get in those areas as well.

John Byron – A couple of weeks ago I was at the CRC Association conference in Adelaide, doing a bit of industrial espionage. I saw a lot of good sessions but the best one, I reckon, was chaired by Michael Barber. It was the postgraduate presentations, and there is obviously some really excellent work going on there.

But I was wondering if you could tell us a little bit about how excellence in postgraduate and higher-degree research is assessed or viewed in the CRC model, where there is presumably a bit of push and pull between the two different outcomes: the students are sort of trying to straddle both worlds.

Geoffrey Vaughan – There is a fair bit of effort put into evaluating the educational component of CRCs, and because the biggest component in education is postgraduate research, particularly PhD research, we hope that it is innovative research and we evaluate it accordingly. We get some measurements because we can look at the number of APRA [Australian Postgraduate Research Awards] grants the students have got, the number of external grants they get, the number of publications they participate in et cetera. We get all those as performance indicators, generally, from the annual reports.

The other thing is the students themselves. We meet with them and they tell us – and they are pretty honest – what they are getting out of a CRC. And very often they say, ‘Look, the guys in the lab next door are jealous of us. We have quicker access to equipment, we have supervisors that come and see us, we have generally multi-supervisors from industry or CSIRO participating,’ and that all adds to excellence in research training as well.

As you rightly said, the CRC Association plays a role. One insert in these outcomes of CRCs [holds up some publications] is the synopsis of each of those graduate students that win the awards of the CRC Association. So I think people can look at these and you start to see something about the quality of activities of research students in the CRC program.

The CRC students do get a bit extra. They generally have some stuff on commercialisation, research management, even human resource management, intellectual property et cetera, and all that adds, I think, to a better research program – for those people that want it – than some of the other types of PhD programs. Not everyone wants that sort of stuff.

The other thing is the employment outcome of these students. Very often there is a high transition of CRC students to industry, and we are starting to hear this term time and time again, that the students are starting to become ‘industry-ready’.

Sue Serjeantson – Is that good?

Geoffrey Vaughan – Yes, for those students, not for everyone.