MEASURING EXCELLENCE IN RESEARCH AND RESEARCH TRAINING
Canberra, 22 June 2004
Measuring excellence: A Chief Scientist perspective
Dr Robin Batterham
Thanks
for the opportunity to kick off today’s discussions. It really is
an important topic.
Role of the Chief Scientist
(Click on image for a larger version)
By the way, this is a commercial which you can read whilst I am getting into the opening. I have to give it these days so that people are quite clear what my role is.
I had the benefit of quite some time last week with Dave King, the UK Chief Scientific Advisor. Dave King is quite adamant that the measuring of the excellence of research in the United Kingdom is one of the key reasons that research is now better funded in that country and taken seriously by government and wider afield – quite adamant. Indeed, coming out in Nature in June is a paper by David King, ‘The scientific impact of nations’. It continues Bob May’s earlier work showing the scientific footprint, as it is now quite developed, of different countries.
I have got the data here for Australia versus many other countries, including most of the OECD list, and it makes some interesting reading. I am not going to get into it, other than saying there are starting to be some quite well developed systems for measuring the footprint, and the excellence that sits behind that footprint, of research in countries as a whole. And I would refer you to Dave King’s paper.
The prompt
(Click on image for a larger version)
Let me run through a few things, because there are three points that I would like to make. They are the changing nature of science, the overall funding levels and a few comments about measuring performance.
But firstly, we don’t really have a robust or a consistent way to measure the quality of research conducted in universities and publicly funded research agencies. That is what is regarded as the state of play now, and is of course the background and context for why you are here.
Quality and accessibility frameworks
(Click on image for a larger version)
You would also all be well aware that a small amount of money has been set aside to help develop an appropriate framework.
CAP-xx Pty Ltd
(Click on image for a larger version)
I wanted to emphasise, particularly on the science side, the changing nature of science. I have used here an illustration from Calum Drummond, a Federation Fellow, from a presentation that he was to give to PMSEIC last week. I thought of it when I was sitting in the Institution of Electrical Engineers, in London, in this marvellous wood-panelled room with a picture of Faraday looking down at you, and you sit there and listen to whoever is presenting – it happened to be Sir David King – and doing the mental comparison, ‘Is this person up to Faraday’s standard or not?’
One of the intriguing things about Faraday, and ultrahigh capacity capacitors, which was the subject of Calum Drummond’s talk, is that the scientific direction for how to get really high capacity capacitors was actually set by Faraday and it has taken all this time before it has been realised. And it has been realised because some great science out of CSIRO in the nanotechnology area has allowed the ultrafine carbon particles, which are at the core of these capacitors, to be produced.
The point that I am making here is that the nature of science – and wider endeavours – is changing, in that there is much more intersection across disciplines and even bridging, of course, between science, social sciences and the humanities.
The nature of R&D is changing
(Click on image for a larger version)
Indeed, plenty of you have heard me on this topic before, that the nature of R&D is changing. I am not arguing that all research should have a user-driven focus on generating income, et cetera. I am not arguing that at all. I am simply saying that that is becoming an emerging style of research, and it is one – an appropriate one, I hasten to add – which assessment schemes have to come to grips with.
If you look at the experience of assessment teams, it is absolutely clear that some go down to far too much detail. Gareth Roberts was commenting that that is a bit of a trap with the New Zealand system. And others are limited just to basic research, and as such they miss out on many important aspects. So we are aware of that, but the point I make over and above that is that the nature of R&D is changing and we have got to accommodate that and look at the experience of others in assessing R&D.
Ten emerging technologies
(Click on image for a larger version)
I would make two comments here. There [on slide] are MIT’s current 10 emerging technologies. And so many of them are either cross-cutting or, essentially, disciplines that simply weren’t around, even a few years ago – not all of them, but a good few of them there. So we have to have a system that copes with that.
To move forward
(Click on image for a larger version)
To move forward: central agencies, by which I am talking about Treasury and Finance, want more information. I would remind you that in the UK they have now had four sets of RAEs and are looking at their fifth. I won’t say anything about what it will be, but in the end of June they will be coming out with a position for their 10-year strategy for science skills and innovation. And by ‘science’ they define science – like Alice in Wonderland – to mean what they want it to mean. They quite explicitly, up front, talk about science, engineering, social science and the humanities. No doubt you can juggle the order if you wish, but the point is that when they talk about science skills and innovation they are talking about the full spectrum, as we are here today with the four academies. And their Treasury put science – as defined – skills and innovation up the top of the spending review: an interesting position for a country to be in.
Central agencies demand that they can have outcome measures which allow them to do meaningful comparisons across such things as health services versus R&D versus security and a string of other things. And it is not an unreasonable demand.
Research assessment
(Click on image for a larger version)
On the research assessment scheme, I would suggest that schemes of this nature do allow funders to assess the quality of the research. It enables you to assess your own success, it undoubtedly informs a funding model. That’s the reality of life. And it is reputed to be the prompt for the enhanced performance in the UK.
From the UK: average research impact
across all fields
(Click on image for a larger version)
There is certainly some strong evidence, not just in Sir David King's paper coming out in June but in earlier papers on the topic. In this [graph] we have got Australia, Canada, Finland and the UK. The UK one, which is at the top of the pack there, has shown quite some marked increase over the last 20 years. And that is what they use with their Treasury to justify more rather than less.
I commend the footprint information coming out in the Nature paper – and there are some earlier papers on it – because it pulls about 20-odd measures down into six general measures and then posts them up as a footprint. And it shows some improvement, I might add, over the last 20 years.
In tandem with the National Academies
Forum
(Click on image for a larger version)
So, in tandem with what is going on today, I alert you to a few things that are also happening. On the National Research Priorities, the agencies’ implementation plans are up for review at the moment; CSIRO is spending quite some time on performance measurement, and indeed has to, as part of its triennial funding arrangements and the extra money that it won there; there is a PMSEIC standing committee having a think about the broad topic of return on investment in R&D – and that is the broad return, not just the monetary return to, perhaps, individual companies – and the CCST, the Coordination Committee on Science and Technology, is also grappling with metrics in this area.
The endgame
(Click on image for a larger version)
So there’s the endgame, which is fairly clear. We are all, I think, interested in quality, and the question is: what value can it deliver?
In conclusion
(Click on image for a larger version)
In winding this up, I recite the three broad points that I wanted to make. Firstly, the nature of science as defined in that Alice in Wonderland way that the Brits define it, to be in fact the purview of the four academies, is changing. Funding levels do depend on having plausible metrics of one sort or another that you can discuss with central agencies. And can I suggest for today, in closing: don’t be too abstract. By all means, have the broad discussion, but if at the end of the day there is a nice abstract position that lists the 20 things that need to be measured and the 42 reasons why none of them are perfect and why the integral of all of them is still something which can be anything, that isn’t going to be too helpful.
You might consider – and this is going to be rather challenging, as my job here this morning is to be brief and to be challenging, I think you said, Mr Chairman – why you wouldn’t just start with the latest version of the UK Research Assessment Exercise, and improve on that. Remember, it has had five iterations now, and an awful lot of time and effort have gone into it. So there is a pragmatic way of going, as well as the abstract one, and no doubt you will come out somewhere in between. I look forward to the results.
Questions/discussion
Sue Richardson –You put up a graph from the UK (which was also presented at a similar gathering a few weeks ago) showing how the UK performance had improved so much and attributing cause and effect. Now, to a simple social scientist that didn’t look very compelling to me.
First of all, the growth in the UK seemed to be absolutely paralleled by the growth in Australia, although the absolute levels were a bit lower. Also, the growth for Canada and for Finland looked distinctly more impressive than the growth for the UK. I wonder if you could just say a word or two about how the interpretation that this graph demonstrates the effectiveness of the UK policy has been arrived at.
Robin Batterham – I am not going to try and argue how they have arrived at it. You can read the paper and look at the details, and it is worthwhile looking at. All I would make is the simple point: UK Treasury had science – as defined – skills and innovation at the head of its spending review. And I don’t make any more than that. This is my point. You can be as abstract and try to be as pure as you like with it, and that’s fine, but at the end of the day the prime reason why one enters such an exercise is to convince central agencies that the money they are spending is good value.
So if you can think of a better way to do that, you will have won. That’s a very pragmatic answer. I hope everyone got my message.