MEASURING EXCELLENCE IN RESEARCH AND RESEARCH TRAINING
Canberra, 22 June 2004

Excellence in mission-directed research
Dr Michael Barber

There is a bit of a danger this morning that we are going to cut up the research enterprise into various silos, and I guess that is almost inevitable, but then I hope the afternoon will start to pull the various threads together. So I am going to again adopt that view. I am going to try and sketch out some of the dimensions of excellence in the topic of mission-directed research.

I guess I had three objectives in the next 15 or 20 minutes. I wanted to give you some characteristics of what we regard as mission-directed strategic research; I will then use the CSIRO’s National Research Flagship Program as, if you like, an exemplar of that in the Australian context, although I will draw a little bit on international comparisons and other exemplars; and I will then spend perhaps the last part of the talk talking about what does it mean to be excellent in this sort of research, what are the possible metrics and, in particular, the role of peer review.

As I have already indicated, there is a danger in what I am saying that it will take on a sort of a life of its own, and I think one of the issues clearly before us is the complexity of the system and the fact that, in a way, for Australia we need to have excellence in a whole range of different types of research. The balance of that system is part of what we need to think about and it should not be a consequence, albeit unintended, of any assessment process to unproductively destroy the balance – though we may want to come back and debate the balance a bit.

So if I put that as a precursor, I am going to focus very much on what internationally has been called mission-directed research.

What do we mean by mission-directed research? It is really R&D directed, as the title would suggest, towards some specific application or outcome area. It is not directed towards a scientific or technical endpoint; rather, towards enabling the applications or the mission to be achieved. Its phrase probably goes back to programs like NASA’s man on the moon, clearly aimed at an auditable mission for which the success measure was to land a man on the moon by the end, as you might recall the John Kennedy statement, of 1970. But the outcomes are the critical part of it.

It does not imply that you cannot do excellent and, in fact, Nobel Prize-winning science – if you put it within the science context – in mission-directed research. Teasing out those dimensions is, I think, important.

What are some of the characteristics that you might associate with mission-directed research? Again these are not by any means unique to mission-directed research, and in many ways the difference as one goes across the spectrum from the investigator-initiated research that Bob Graham talked us through to this sort of research, to the research that Bob Watts will talk about later, inspired out of industry, is really a matter of degree rather than obvious cut-offs.

The characteristics of what I will call mission-directed strategic research are a large-scale, long-term focus on national or critical problems, usually performed by transdisciplinary teams. The objective as an outcome is to solve some problem, to deliver some outcome. They tend not to be problems which immediately define themselves as the province of chemistry, the province of biology. Generating a cotton industry for Australia, which was a mission-directed program of CSIRO’s Plant Industry, didn’t require just of plant scientists but also agriculturalists, across that. The disciplines that are needed are the critical part of it.

It is driven, as I have said, very much by outcomes and needs, and therefore very critically actively managed towards adoption. And I will talk a little bit more about that ‘critical’ thing.

The measures that you need to consider are clearly the science excellence – or perhaps more broadly today, since mission-directed research tends to involve societal impact and therefore our social scientists are becoming critical elements of the teams which address the problem – to which must be added operational excellence and in particular the outcomes and how we demonstrate outcomes. In the Australian context and again globally, these tend to be strong in the publicly funded research agencies. In fact, I would argue that without strategic mission-directed research there isn’t really a logical role for a CSIRO. It is similar for ANSTO or AIMS or, in the US, the Department of Energy national laboratories, or the office of NASA’s research, as examples.

DARPA’s ‘end-game’ approach
(Click on image for a larger version)

There also, in a sense, are design features which I have tended to take from this model from DARPA, the Defense Advanced Research Projects Agency of the United States. It distinguishes one of the aspects which I think we do need to distinguish.

In the Defense Advanced Research Projects Agency of the US Department of Defense, they really are mission directed. They are initiating part of the future. What are the new products and processes they need? Can they develop a detection mechanism to see through a few hundred metres of the surface of the ocean? Can you detect infrared signals? Can you specify those end products that they would want? Can you specify new materials? So it is to anticipate the needs, come down to the present and put together the research teams that will actually deliver those outcomes and then, in particular in the mission-directed sense, manage that process to the end.

It is the complete flip of Bob Graham’s description of investigator-driven research, where the ideas flow from the researchers and the research teams, are evaluated by a peer review process and lead to research outcomes which may or may not be picked up by industry or commercialisation and may or may not deliver towards those new processes. This is an attempt to take the processes and the end requirements, and drive very hard towards the end.

Peer review plays a role, and we will come back to where peer review plays a critical role. Excellence plays a critical role, but it is the other end of that dichotomy of whether it is initiated at one end under ‘research teams’ on the left of the slide, or whether it is initiated by the outcomes.

CSIRO’s Flagship Programs, I would contend, are in fact a classical example of mission-directed research, and I want to tease out a couple of the general features of them before I turn to the question of excellence, because again these are very much part of the differentiation of the system.

They are clearly focused on major national issues. They have got ‘big hairy audacious goals’, to use Geoff Garrett’s phrase, and they are a key driver of our alignment, in fact our response, to the National Research Priorities which we were as a Commonwealth agency required to take into account in our research planning. They have scale and impact, and will over five years consume something like 30 or 40 per cent of the appropriation of CSIRO. But they are not just about CSIRO. To build that scale we do need to reach out, to bring together ‘Team Australia’, to use Geoff’s phrasing.

They are very much outcome-focused and outcome-directed. Their success will not just be measured by excellence in the science we do, but will be done in fact by achieving outcomes. And, critical to that, there is an active management throughout the program towards adoption.

The Flagships
(Click on image for a larger version)

I am not going to spend too much time on the Flagships; they have been well publicised. I have just put up their missions, if you like, because I want to pull out of them a couple of key features. For example, the Flagship P-Health is to improve the health and wellbeing of Australians and save $2 billion in annual direct health costs by 2020 through prevention and early detection of chronic diseases. You can read the other ones, but for each one of them you will note that there is a quantifiable indicator of global success at some point in the future – Light Metals: to double export income and generate significant new industries, Food Futures: to transform the international competitiveness and add $3 billion annually to the Australian agrifood system.

The Flagships
(Click on image for a larger version)

The other three – Energy Transformed: to halve greenhouse gas emissions and double the efficiency, Water for a Health Country: to achieve a tenfold increase in the social, economic and environmental benefits of water in 2025, and the third is Wealth from Oceans.

There are two remarks I want to draw from that. One I have already alluded to. Firstly, each of those missions has a quantifiable indicator of what you might actually say is success at some time in the future. Secondly, it is not only science and technology which is going to give those outcomes, and it is not only CSIRO that can clearly deliver them. We cannot give a tenfold increase in social, economic and environmental benefits from water by 2025 with science and technology alone. Therefore, it is critical in that case that we think through the processes of adoption, the issues that need to be addressed in achieving those outcomes, because science and technology alone, and science and technology along with our social sciences that are critical in many of these alone, are not going to achieve it. It could be regulatory, it could be tax arrangements, it could be government policies – a whole slew of issues need to be, in fact, addressed.

I am not going to drill down through the Flagships to show how these objectives are translated into detailed plans; that is a talk in its own right. But again the critical aspect is that flowing through the Flagships are these two dimensions of strategic research: a focus on the research that is going to lead to those outcomes, but secondly a critical and equal weight on what are the steps we need to take to achieve adoption so it is not something left for the future. So Tony Filmer, the director of Light Metals, whose ultimate goal is to develop a titanium industry for Australia, is already beginning to think about and talk to industry partners about what a pilot plant might look like, what a pilot plant might cost, long before the R&D that would be going into that pilot plant is in fact anywhere near fruition. The blending together of science, the research and the outcomes is critical.

Effectiveness/Outcomes
(Click on image for a larger version)

So what are some of the effectiveness/outcomes? This comes back to perhaps the most critical issue facing research assessment or research excellence, and it picks up Robin Batterham’s question earlier. What are the effectiveness/outcomes?

We have in CSIRO, picking up some work we did with the Committee of International Economics, picked up what we call the ‘nine shocks’ – indicators of effectiveness may be lower, more competitive production costs; improved quality of goods and services; new products, services and businesses; et cetera. Again they are difficult to quantify at times, but each one is potentially possible of being quantified in some particular way. But again the link between these things and how we access the underlying science or the underlying research is a critical dimension for us to worry about.

What, then, is excellence in mission-directed research? We need measures of scientific and technical excellence, without a doubt. That picks up again many of the issues. But we also need measures of how we do the projects, technical adoption, use, economic, environmental and social impact. Putting that suite together, we need therefore to begin to tease out, I would contend, three elements which we need to build in to our assessment process.

We need to see evidence of productivity, vitality if you like, of both research results and their transfers to researchers. But also, if we are really bringing together those teams to tackle major national challenges, we need to see connectivity, both to the research community – have we brought together the necessary obvious teams to do that process, has the design of the program enabled the best researchers to in fact to be contributing to the outcome? – and critically, as I have said, in managing the adoption by the users of the research.

Excellence in mission-directed research
(Click on image for a larger version)

Finally, the impact, both in the world of science – as I have said, if we have tackled some of the nation’s national challenges I believe those will in fact have pushed the paradigms of science, the limits of science, the limits of the disciplines other than science, that are necessary – but also, critically, in the intended application of the research.

Productivity (vitality)
(Click on image for a larger version)

So let me take each of those dimensions, if you like, and briefly – since time is of the essence – put up on the board a few of the indicators that one might look at. In doing so, I want to pick up a couple of them.

In the scientific/technical area it is relatively straightforward: clearly, where you are publishing, whether there have been invention disclosures, new patents issued. That’s the scientific output, a fairly regular part of the scientific effort.

On technology transfer, again we have probably got to the point where we know something: customer reports, the artifacts transferred in software designs et cetera, the licences executed, the start-up companies formed. They give at least a sense of productivity, probably put best in the sense that if you haven’t got those indicators then it is at least a valid question to ask whether you are actually doing anything. But that is not by any means enough if we are interested in delivering the mission or the significant things.

Connectivity
(Click on image for a larger version)

Turning now to the research connectivity, again the connectivity to the research community is something for which we have probably got a number of metrics: students co-supervised, international visitors, the mobility, conference presentations, the engagement of professionals et cetera – all a measure or a metric that gives some sense of the engagement of the research group or the research organisation. And again customers/end-users: increasingly it is an aspect of this management to adoption that we no longer accept the linear model by which, essentially, we chuck the results over the fence and hope that industry picks them up, but to work very deliberately across organisational teams.

And again the CRCs are a good example.Their success has been to bring together the research providers and the research users to manage those activities.

Impact
(Click on image for a larger version)

And then finally, probably the most critical and perhaps in some ways the most difficult is the impact. On the left hand of the slide, under ‘science/technology’ without in any sense suggesting that we have got these things perfect, and echoing some of Iain McCalman’s comments or concerns about the reliability or the success of citation analysis, there is a sense, at least within the sciences, that our scientific impacts are at least to some extent measurable.

A more critical issue for us is to start to think about how we get economic, social and environmental impact. There are a number of standard methods that abound, none of them perfect and all of them probably issues which could well be worked through. Here if nothing else the engagement of our social science colleagues into the assessment of the innovation system is, I think, a critical issue. The Europeans have done some rather interesting work in this area of trying to think about the complexities of the innovation system and how information is in fact going.

You can actually do fairly standard things – benefit:cost studies, customer statements about impact, income from licences, and the value of start-ups and spin-outs. In Australia we have taken a rather simplistic view of saying, ‘How many spin-offs are coming out of the University of Western Australia?’ for example, or out of any university.

But I think in Australia we can see those companies. The more interesting challenge for us is whether we can generate them the next phase up – and that will be a long-term thing.

The most critical aspect about social or environmental or economic impact studies is that they are a long, long time and it is very difficult to bring together the scale on which you need to look at this research. Research for the University of Western Australia spin-off – Advanced Powder Technology Pty Ltd – goes back probably 15 or 20 years now, and its impact I suspect will be fully recognised in 10 years’ time – a very long time but, I think, a natural feature of most of these impact studies.

Let me begin to draw this to an end by commenting on where the role is of peer review. We have actually said we needed all of these dimensions to be looked at. It is critical, because none of those indicators by themselves, while important, are in fact ones that stand up without interpretation. There need to be experts – I think ‘expert’ is probably better than ‘peer’ here; it includes the traditional peers but it also includes wider expert review. That judgment is necessary, judgment that is able to validate what the data says.

To me the data is critical, because in the absence of data you can at least begin to ask some very, very hard questions. Again it may be perfectly reasonable that there are no citations or there are no spin-offs out of a program that you have wanted to go, but judgment needs to be called on how you do it.

That raises the very critical issue of what we mean by peers and experts. Who are the peers, who are the experts that need to be brought to bear? I would contend again that is something that we need to give perhaps more clarity to, in the assessment process. We tend to assume that there are things called peers; we don’t define very clearly the skills we require in the review committee and, since we require them to make judgment in the context of the mission, we don’t define very clearly what they are going to assess. Is it a research group whose main objective is to produce basic research? That is a pretty straightforward objective. If we are looking at a mission-directed program, it is in fact important to have an appropriate focus on the mission, on the context in which we are actually doing it.

And as I perhaps indicated, in looking at mission-directed research all dimensions of excellence are equally important. So this leaves us, then, with the final part of the puzzle: when you start to put these things together across the whole of the innovation system, you have to be careful that you don’t actually decide to compare an apple with an orange.

There is absolutely no doubt in my mind that by focusing on a global problem, an important national problem, you will do excellent research. You only have to look at the history of the CSIRO Division that my friend and colleague Jim Peacock led for 25 years, Plant Industry: a mission to produce value for Australia in agricultural research and in the cotton program. It has produced significant breakthroughs in many areas of plant science. But in a sense, every one of those developments had the potential to impact upon the cotton industry. You would not, for example, I suppose, Jim, have acknowledged a project in the genomics of coconuts because it had some relationship to the cotton industry. It is more focused. And so the ability to judge across these systems is a critical part of the evaluation.

It comes back again that the reviewers have to be people who have an understanding of the complexity of the science and the complexity of the research that we work in today, in part reflecting the complexity of the world.

So I guess my final message in this debate is that we have to acknowledge the complexity but, in the end, rely critically on the judgment of well-regarded, diligent peers and experts who are able to, in fact, assess on a multiplicity of dimensions.

Questions/discussion

Doreen Clark – Is there any facility for ‘telling the story’ in the system of mission-directed research?

Michael Barber – I certainly think there is a challenge to tell the story. The classic is the story of the man on the moon. I think in the Australian context there are stories which we don’t actually tell: the cotton industry, on one hand; industry that is mission-directed to support the mineral industry; the values that have flowed through it – I think there are stories there that resonate very spectacularly with many of the issues, and address very much the context of those outcomes. Those outcomes have to become outcomes that resonate in a context of people’s experience. I think there is a real story to be told, from industry, from CSIRO, from the wider community. CRCs, to a certain extent, tell a very good story of mission-directed research.

Chris Blackall – You contrasted the mission-driven research model with Bob Graham’s investigator-driven research. Clearly, in Bob’s model the accountability is very much tied to the individual. If the individual succeeds, then the individual enjoys the fruit of their labour – and equally so if they screw up. In your model the lines of accountability are not clear, or at least they would appear to be held by the organisation, and clearly as in all organisations the rewards are distributed asymmetrically.

I was wondering how you could square up this issue of accountability and the individual contributions of researchers as reflected in citation counts and in the literature.

Michael Barber – It is an important issue. To take the Flagship Programs, the accountability of the Flagship Programs, the accountability to the government for the extra money as a headline in Backing Australia’s Ability, is clearly on CSIRO. It is an agency responsible for delivering them, and that impact is critical.

The question of, now, how you actually translate the values, how you get reward systems, how you get motivations of staff et cetera within that framework, is a critical issue of the human resource processes within CSIRO. Individuals will drive and deliver those Flagships, and therefore what are their internal reward systems, how do we recognise excellence et cetera?

I think for the wider science community, though, the danger is that the metrics which are actually used to achieve recognition of someone may not be appropriate. Let me pick a scientist working at the Queensland Centre for Advanced Technologies doing mission-directed research to add values for Australia’s iron ore experts working for BHP Billiton on projects of importance to BHP Billiton. That person will not be publishing as extensively as someone working in investigator-driven research. I would contest that he or she is making a significant value for Australia; he or she is certainly being rewarded within CSIRO. But the potential is that they will not be perceived as being as valuable within the science story or telling as great a story, as Doreen’s comment suggested, as someone working in investigator-driven research, because they are not publishing, they are not being seen so visibly in that community.

So I think those stories are important but the individual’s responsibilities are different in the two situations, as part of that rich fabric of the research enterprise one needs in the 21st century.