HIGH FLYERS THINK TANK

National Research Priorities Strategic Forum
26–27 June 2002

A critique of the status quo, with respect to the operation of research organisations
by Bruce Hobbs

Bruce Hobbs is Deputy Chief Executive, Strategic and Investment Planning, CSIRO. Within CSIRO, he was previously Deputy Chief Executive, Minerals and Energy; Chief, Exploration and Mining; Chief, Geomechanics; and Chief Research Scientist, Geomechanics. He was formerly Foundation Professor of Earth Sciences at Monash University. He is an Honorary Professor at Monash University and the Sir James Foots School of Mineral Resources, University of Queensland. He was awarded the Jaeger Medal by the Australian Academy of Science in 2001, for distinguished contributions to Earth sciences. His dominating interests at present involve the development and implementation of large national R&D projects that address significant problems facing Australia over the next 25 years.

I might say it is a great pleasure to be here, and some of the talks have been absolutely inspirational to me. I want to go through a bit of a journey that is still in progress. I have not taken the topic that was in the paper (I thought it was about investing in science and measuring the outcomes) but a bit of the critique will come in a moment.

This is ‘our’ map of MIT. I think it is a good map of a knowledge infrastructure. It has a whole lot of things in it to do with what universities normally do, and it has a whole lot of things to do with what other kinds of people normally do. What I worry about in the present trend that I see in this country is that people haven’t defined what this is, as far as we as a country are concerned. It would be nice if we did. And, secondly, if it looked like this – and I think it does look something like this – then we would be trying to optimise all the bits without trying to optimise the system. My preference would be that we take a big overview of all this, as in David Strangway’s beautiful example from Canada, and say: how do we actually optimise the system, rather than optimising, say, the outputs of new ventures or the outputs from teaching or something like that? Let’s take a systems view of this, and say what Australia wants to achieve rather than what individual entities want to achieve.

Well, many governments are moving to this knowledge infrastructure type approach, but the trouble is that most governments in Australia – by that I mean state and federal – still try to measure things as though they were a kind of Department of Administration or something, or like a phone-call centre: ‘How long did you spend on each phone call, and what did you achieve from it?’ They therefore regard the R&D organisations, by which I mean the universities, CSIRO, AIMS and so forth, as from a customer/provider point of view. In fact, if you go and talk to DoFA, the Department of Finance and Administration, which I have unfortunately spent a lot of time doing over the last year or so, you find they really have that customer/provider attitude to life. And from that point of view they have enormous difficulty in defining or understanding what the benefits of the R&D organisation are. They want to understand what you actually provide to them as a customer, and that difficulty arises because they cannot readily measure the effectiveness of the R&D organisation’s outputs.

My message today is simple. In order to develop a world-class knowledge infrastructure for Australia, it is essential that both governments and R&D organisations view the funding process as an investment, rather than a cost. That is again a beautiful example from Canada. But implementation of such a view carries responsibilities for both governments and R&D organisations. I am going to continuously refer to Canada, because what has happened there is that the government has made it very clear what they expect, which is an accountability that says, ‘This is what we achieved,’ and that those outcomes are nicely measured by the framework that has been put in place and there is a responsibility on the R&D organisation to do that reporting and to make sure it happens properly. If the funding process is one of investment, then governments must have a way of being convinced of the magnitude of their return on investment – and that their investment is being managed properly.

An essential feature of the knowledge infrastructure of a country is the investment of government in publicly funded R&D organisations that undertake fundamental research or strategic research that is too risky for industry to undertake. This investment culture implies a shift from a customer/provider to an investment/performer relationship. So my talk is all about: how do you measure performance? It necessitates a move from some form of performance-based contracts with, ideally, a quantitative measure of returns to government. You can see again that the Canadians have gone some way down that route towards quantitative measures, but it is very nicely balanced, I thought, between ‘soft’ measurement and ‘hard’ measurement.

This is the way in which many, many companies and personalities would behave if they wanted to invest in, say, BHP. They would look at the quality of management and management processes – that is, the governance, the finance systems, the strategic planning processes, all that kind of stuff; the track record of the organisation, which is the way in which we normally measure science in this country, say by citation indices, but that gives you some kind of hint as to the likelihood that your return on investment might be rather large; and then, lastly, the likely return on investment, measured in some ideally quantitative way. We to some extent measure the track record very well in this country. We hardly ever really think about the quality of management and management process. In fact, most academic communities would say, ‘We don’t want to really have anything to do with that at all,’ until recently. And we are very, very bad, all over the world, at measuring in quantitative terms, the likely return on investment.

Here is a critique of the status quo:

• Vague goals lead to perpetual programs achieving poor results.
• We can rarely show what our R&D investments have produced, and we do not link information about performance to our decisions about funding.
• Many R&D projects have ended up stepping beyond the legitimate purposes of government to compete with – or unnecessarily subsidize – commercial ventures.
• Finally, many R&D projects directly benefit corporations that could fund their own R&D projects without federal assistance.

Who said that? President George Bush. And in The President’s Management Agenda: Fiscal Year 2002, chapter 8, there is a whole discussion on what he – or his advisers – thinks about the way in which R&D is conducted in the United States. They have run a little trial thing with the Department of Energy now to say, ‘How are we going to do this better?’ But the kind of general sentiment that was expressed in that quote is something that I hear continuously from government.

So a widely recognised problem in specifying outcomes is to define quantifiable measures of effectiveness. I don’t know whether you can ever go all the way to getting quantitative measures of effectiveness, but the term ‘effectiveness’ is used to mean the impact of outputs upon specified outcomes. That is real, strict DoFA-ese that I can speak fluently now. And we heard it again from David Strangway this morning: what is the impact of the research, how do you go about measuring it, how do you go about telling a story that convinces the public, the government or whatever that you have actually been effective in the research that you have done?

The move internationally is away from vague anecdotal statements of achievements, which amount to an attempt to demonstrate compliance with what are commonly equally vague policy statements made by gove