MEASURING EXCELLENCE IN RESEARCH AND RESEARCH TRAINING
Canberra, 22 June 2004

Excellence in investigator-driven research
Professor Robert Graham

All I can say after Robin Batterham’s talk is, ‘Oh dear! There are lots of things I disagree with in what he said.’ But anyway we will have to battle on, as you will see in my presentation.

What I chose to do today, rather than getting into specifics and examples of how you measure excellence, or what is excellence, is to explore more the notions of quality, the concepts of what constitutes excellence, and present some principles.

To begin with, let me tell you what I think investigator-driven research is, why I think it is very important and why I think we should resist what Robin is suggesting we do with the new way of funding, which I hope will become apparent. What I would like to do is to explore just four issues: what is investigator-driven research, what is excellence in investigator-driven research, how can we foster excellence in investigator-driven research, and how can we better capture the value of research?

So let me start with the first: what is investigator-driven research? Inherent in the concept of investigator-driven research is that it is a system of research funding, largely pioneered in the United States and adopted in Australia and a few European countries. It is a system not practised in Eastern Bloc countries, Japan or many European countries. In those countries the direction and the control of research are vested in a relatively few senior people, and agendas are determined in a top-down fashion.

In the US, the bulk of research funding goes to individual investigators, thus making them unbeholden to department heads, to deans or to university politics. After applying for a grant, the individual is judged in competition with other applicants by a group of peers outside of their institution, and with the awarding of a grant the investigator becomes his or her own boss, whose success or failure depends entirely on what they can accomplish.

So investigator research is curiosity-driven inquiry initiated by an individual. With respect to biomedical research, it need not be commercial or applied research, not health-outcomes research nor clinical trials research, although it can be. But it can as well just be pure basic research.

I would suggest that, much like a democracy, it is a system that has to be constantly defended but one that we are prepared to fight for – indeed, we are prepared to die for it in some instances – because we can see its profound value.

So too investigator research is not without its problems. It is a system that engenders conflict: the conflicts of freedom versus responsibility, discovery versus utility, what we are able to do versus where we need to go, strategic initiatives versus unencumbered directions, and societal responsibility versus intellectual curiosity.

And because of these conflicts, investigator-driven research, or funding for individual investigators, is continuously under threat. As Robin told you before, there is a move away from this sort of individual research into group and collective research. And, I would suggest, that is a bad thing.

For example, although Japan is to be applauded for instigating the Human Frontier Science Program, such grants are made only to groups from several countries who can devise a project advanced enough to be divided among them. The bulk of the research funding in Japan is not investigator-driven.

In Europe, the EU requires that investigators from three or more countries find a consensus project that can be parcelled up, leaving little room for a scientist to do something utterly original and entirely unpopular.

In the UK, despite what Robin says, I would suggest that in biomedical research, at least, their research has improved because of the great injection of funds by the Wellcome Trust, which was desperately needed after the Thatcher era otherwise it might have fallen completely apart, rather than because of any measures of excellence. Indeed, the Medical Research Council is increasingly consolidating grants along the lines of the EU.

In Italy, the powerful baronial organisation of research granting agencies perpetuates fragmentation and favouritism.

What about in Australia? While the abolition of block grant funding I think is to be applauded, a considerable fraction of the NHMRC budget is already obligated to program grants, in some of which directors can select investigators and projects that might not otherwise withstand peer review. Although most of these projects serve legitimate purposes, the outlay to support them becomes a fixed-entitlement expenditure.

Of greater concern to me personally is the move to direct up to one-third of the ARC budget to ‘strategic initiatives’, an issue that I tackled Vicki Sara on in this very room over a year ago. In fairness, Vicki then explained to me that that was not her idea but an initiative that came from above.

Although the recent NHMRC review, the so-called Wills report, resulted in a much-needed increase in research funding, and for that it should be applauded, I would suggest that it was predicated largely on a business platform, with heavy emphasis on research commercialisation but with little recognition or understanding of the intrinsic value, let alone the riches, of discovery – probably reflecting in part that the head of this review was a business person rather than a scientist.

How do we resolve these conflicts? Here I can offer but three principles. Like democracy, and despite its problems, investigator-driven research has served us well. Let’s not throw out the baby with the bathwater. We need to defend it, and we need to uphold its value.

Secondly, I would like to suggest that scientists, let alone governments, are very bad at looking into the crystal ball and trying to determine directions and where we should be going. Strategic paths, therefore, should provide a compass, not control the journey.

And, thirdly, I think we need to get the balance right. For example, in the United States and other countries that have successful investigator-driven research programs, only 10 to 15 per cent of the total research funding is devoted to programmatic or strategic research.

I would like to digress here a moment and read you an editorial by Arthur Kornberg, Nobel Laureate for discovering DNA polymerase, and published in Science, 12 December 1997. He said:

An oft-stated reason for block grants and collective efforts is the expensive equipment and resources needed to solve the problems of major diseases. A common illusion is that ‘strategic’ objectives are necessary to discover the cures for cancer and AIDS, and that groups of sufficient size need to be mobilised for wars and crusades against the enemies.

Nothing could be more misguided. In the history of triumphs in biomedical science, such wars and crusades have invariably failed because they lacked the necessary weapons: the essential knowledge of basic life processes.

Instead, some of the major advances – X-rays, penicillin, polio vaccine and genetic engineering – have come from the efforts of individual scientists to understand Nature, unrelated to any practical objective. Basic research has been the province of the individual investigator and remains the lifeline of medicine.

Let me move on to what constitutes excellence in investigator-driven research. I agree entirely with Robin Batterham that this is a very difficult issue to quantify, and I am not even going to try here. The best I can say is that when you see it, you know it, but it’s damned hard to describe, let alone to quantify.

If you wanted to push me, I would suggest that, at least in biomedical research, excellence is research that provides essential knowledge into basic life processes and, further, it is research that allows us to make the unimaginable imaginable. And that, I would suggest, is its true and immediate utility.

To quote from a lovely essay given by Richard Klauser, the head of the National Cancer Institute, at the time of the Lasker Awards in 1998:

Our interest in science is so often wrapped up in a need to justify the effort. We justify the scientific pursuit by ‘the discovery’, and we justify the discovery by its ‘utility’. We thus assign to science its value and we too often miss its riches.

So how do we quantify excellence? I would suggest long term it is not that difficult. It is what one discovers, the impact to science and society, who one trains and the recognition by one’s peers. Short term it is much more difficult to quantify. The two things we look to are citations and grant success, but they are also imperfect measures.

The impact to society can either be concrete, in terms of commercial outcomes or biotechnology, or it can – as importantly, I would suggest – be cognitive, in terms of either direct advances: new diagnostics; or indirect: a new understanding. For example, discoveries made on the anatomy of the renal collecting tubule in the desert rodent, which is able to concentrate water so effectively that it doesn’t even need to drink (it just gets water from the atmosphere and from food metabolism), have led to important understandings of how our bodies regulate salt and water, which we use every day. Every clinician uses it. And I suspect that the seminal paper describing that is never cited any more. So you can see that citations would not be a help. It is something that we use all the time and is of enormous, immense importance to us in treating patients, but its discovery is something that has now been forgotten.

How can we foster excellence in investigator-driven research? To me there are just three issues that need to be considered: excellent researchers, excellent management and adequate funding.

How do you get excellent researchers? Paul Ehrlich, the father of immunology, suggested that to be a scientist you need four things (he said them, of course, in German): Geschick, Geduld, Geld und Glück, which roughly translate into skill, patience, money and luck. And so I would suggest it follows that if you want excellence in researchers you need to have excellent mentoring and training, you need tenacity, perseverance and courage, you need sufficient money and you need insight, creativity and critical thinking. One can certainly train people into critical thinking; insight, of course, and creativity are more innate.

For excellence in management I have turned to a model: the model of Max Perutz’s Laboratory of Molecular Biology, which started in 1962. By the time Max died, in 2002, nine Nobel Prizes had been awarded to its faculty, four Orders of Merit, which is the highest order given in the United Kingdom, eight Copley Medals and an astounding 100 Fellowships of the Royal Society. So I would suggest it is a reasonable model of how you manage a research facility.

Perutz’s Management Model is one that I have tried to adopt in my own institution. It is fairly straightforward: choose outstanding people and give them intellectual freedom, show genuine interest in everyone’s work, give younger colleagues public credit, enlist skilled support staff, facilitate the interchange of ideas, have no secrecy – or, as we would say today, be transparent – and engender a happy environment where people’s morale is kept high.

Adequate funding is a topic that for me was very difficult. What is adequate funding? Clearly, if you want to do internationally competitive research, it is expensive and you need internationally competitive funding. But I thought I would tackle this issue again more conceptually, rather than more specifically, and again read to you from Klauser’s wonderful essay. He said:

If we are to continue to support the type of beautiful science we celebrate today –

and he was talking about the discovery of the cell cycle made by Paul Nurse, Lee Hartwell and Yoshio Masui –

and enable the expression of the creativity of the scientists we honour today

– it will take much more than well disposed appropriations committees, even more than vocal advocates;
– it will take a society that truly values science and scientists, much as Renaissance Florence valued its art and its artists;
– it will take a society that sees its own historical narrative as, in part, the narrative of its science.

I would suggest we have a big problem here. We have a society that sees its own historical narrative as in part the narrative of its sports. And there is nothing wrong with that. We have a society that sees its own historical narrative as in part the narrative of its brave and proud fighting men and women. But we do not yet have a society that sees, as its historical narrative, the narrative of its science.

It is a difficult issue to get around, and all I can suggest is that we need to engender an appreciation not only of the value but of the riches of science.

For example, what is the value of a gold medal? What is the value of discovering the theory of relativity? It has no intrinsic value, nor does poetry or music, yet it is very, very important to our lives and I think few of us have trouble seeing that it is important.

So we need to ‘tell the stories of science’. All of us here, as scientists, if we want to get this message out, need to find ways to tell the stories of science and get people excited. The issue of gold medals, of course, is one that everyone can relate to. The fact is we know we can’t run 100 metres in 10 seconds and so we appreciate somebody who can do it. We can understand poetry because it is something we can come to grips with. Most people in the everyday community can’t understand science because we don’t tell the stories in language that is understandable.

Here let me recite, again from Klauser’s article, what I think is the way to tell the stories of science:

'The work of Nurse, Hartwell and Masui told us that the cell cycle, perhaps the most basic and the most central process of life, can be studied and dissected, and its mechanisms elucidated like the gears and springs of life’s cyclic clockworks. For cancer biologists this discovery has, more than most, illuminated the black box of these diseases. Genetic changes alter the mechanisms that govern the cell cycle, and the altered cell cycle allows a cell to accept genetic damage and passes it on to its progeny – an intricate and fatal choreography of life’s cellular dance gone awry'.

I wish I could write and communicate as beautifully as Klauser.

Finally, let me turn to the topic of how we can better capture the value of biomedical research. I agonised over this, and certainly there are a number of suggestions I can make: develop a culture that encourages interaction between basic scientists and clinicians; reward excellence in not only basic but translational research; increase funding for research; encourage, reward and support intellectual property development; increase tax incentives for biotechnology development; provide adequate seed funding for research commercialisation – and I could go on.

But I would like to suggest that we should be thinking of this in terms that can allow us to capture the value more definitively. I would like to suggest that to truly capture the value of science we have to first understand and appreciate the scientific enterprise. And I think that is not done by many people, because they just aren’t exposed to it and don’t understand it.

What do I mean by the scientific enterprise? Again I am going to turn to Max Perutz, who elegantly said:

To be creative in science, as in the arts, cannot be organised, despite what some may think. It arises spontaneously from individual talent. Well-run laboratories can foster it but hierarchical organisations, inflexible, bureaucratic rules and mountains of futile paperwork can kill it. Discoveries cannot be planned; they pop up like Puck [from A Midsummer Night’s Dream] in unexpected corners.

I would suggest that if we fully understand the scientific enterprise, then we can much better capture the value of research.

So let me summarise. What I have tried to do today is to tell you what I think investigator-driven research is – namely, curiosity-driven research by an individual. I have tried to suggest that to get excellence in investigator-driven research we need research that makes the unimaginable imaginable.

How can we foster research excellence? I have tried to suggest that people, management and funding are critical. And, finally, I have tried to suggest that we can better capture the value of research if we can understand and appreciate the scientific enterprise.

I apologise for not being more specific in areas and presenting rather broad overviews, but I would like to leave you with one thought. That is: measuring excellence in science, as in art, is more of an art than a science.

Questions/discussion

Tom Clark – Thank you for your strong and yet humble presentation, Bob. I partly focus on the paradox. You were talking about excellence in research and one of the components of excellent research was adequate funding. I wonder why we are trying to seek ‘adequate’ funding when the object is ‘excellent’ research, rather than excellent funding for adequate research, or excellent funding for excellent research – or maybe adequate funding for adequate research. It seems to me that the Australian higher education system’s policy struggle within our political system in Australia is to achieve the last of those four: adequate funding for adequate research. Maybe we are talking about pipe dreams.

Bob Graham – I agree with you. I think we need adequate funding for excellent research. How we achieve that, as I suggest, is not easy. I think that conferences such as this are very beneficial from that point of view.

It does worry me a little that the reason we have conferences like this is that we are a little insecure here in Australia. It is one of our innate characteristics. In 17 years living and working in the United States I was never asked, ‘Is this research internationally competitive?’ whereas here every second question is, ‘Is it internationally competitive?’ Of course our research is internationally competitive. That doesn’t mean it can’t be improved, and that’s why conferences like this are good.

But of course I do mean adequate funding for excellent research.

Robin Batterham – Bob, I think first of all that we are actually in furious agreement, because as I realise now – and I really can’t give you the excuse that I have been flying for 20 hours, because everyone does that – I was talking primarily about one thing, which is the block funds and how one might move in that direction, whereas you were talking primarily about competitive grants (in the language that we use here). And I would totally agree with you, by the way.

We have a dual system here, of competitive grants and block grants. And when I suggested about a year ago that because there didn’t seem to be too many quality drivers or measures on the block grants, perhaps we ought to put more effort into the competitive grants, the firing line that was lined up was pretty extensive – and Treasury noted it. The way they interpreted it (this is my interpretation of their interpretation) was that, yes, competitive grants are a pretty clever way to go, because you do have some intrinsic measures of excellence associated with them. I might add, however, that the National Science Foundation did point out, in a recent survey – actually, about three years ago – that investigator-driven research requires about 30 per cent of your time in actually working up applications. But don’t take that one too seriously.

The point is that we have got a dual system, and I think for one side of it it is relatively easy to see how excellence bubbles to the surface. That is the competitive side. For the block side, which is what I was addressing, it is not so easy. And that is where I think we have to put a lot more attention in.

You just might, in your considerations, consider whether you are better off with the split that we have, of block versus competitive, or whether some other model, such in as the US, where it is far more down the investigator-driven area, makes more sense. I don’t want to distract you too far by saying that if you are looking for measures, you actually do have to cover both.

Bob Graham – I’m glad we are not that far apart, Robin. The problem I have with strategic initiatives is that it is suggesting that somebody up there knows where we should be going. There is some credence to that, and certainly we can identify the problems, but I would suggest that the real creativity comes from investigator-driven research and we should put most of our funding in that. If we want strategic directions to be undertaken, let’s have more money but not take it away from investigator-driven research that can be put into that strategic direction.

Stephen Walker – Bob, I was fascinated by your presentation and agree with many of the points. There was one phrase that struck me. You were talking about research, and you said ‘because we all see its profound value.’

Now, I think for this audience, for this collection of people in this workshop here today, that is not going to be a difficult thing to achieve. We can probably convince ourselves, and I think the research community in general can convince themselves, of the profound value of research. The challenge in a research assessment exercise, as Robin has alluded to, is not to convince ourselves; it is to convince the public, it is to convince the central agencies, it is to convince the government as a whole of the profound value of research.

I think that when we are talking about research assessment, when we are talking about measures, we have to keep it in mind that we are not trying to convince ourselves, we are not trying to convince our peers; we are trying to convince people who have no background in research, who largely have no understanding of research. And the challenge here is to find the language, to find the measures. And essentially it becomes a cultural problem of how to measure research, how to describe research and how to make that case in the broader community and in the broader government.

Bob Graham – I agree with you. That is why I was trying to suggest that we need to tell the stories of science, because then people can see not only the value but the riches of science. The problem is that you have to speak in language that people can understand. It is easy enough for people to understand why it is great to get a gold medal or to have an Academy Award in the picture industry, but it is much more difficult for them to understand that someone like William Bragg – without being able to stand on any other giants’ shoulders – was able to use X-rays to determine structure, with what was absolute genius. It was just a wonderful discovery, yet most school kids wouldn’t know who William Bragg is. We need to start at the school level with stories that they can relate to, early on, and that is the only way I know of engendering an appreciation of the riches as well as the value of science.

Frank Larkins – Bob, I wanted to ask you whether you consider that peer review in this country is robust enough to deliver adequate funding for excellent research.

Bob Graham – We are a small society and one of the weaknesses of our peer review system is that we don’t have enough people. I think of the system in the United States, for example at the NIH, where you have study sections, where you have people who have got experience in particular areas and can more robustly evaluate whether a particular project merits funding in that particular area, whereas here we have got people evaluating research in one area who have very little experience in the area of the grant. It makes it a little bit weaker, but by and large I think we do a fairly good job.

It’s not a perfect system – peer review never is. I suggest, though, that it is one that we should try and work on, rather than throwing away. It may be difficult, but we have got to try our best. We may have to use more foreign reviewers. That’s also difficult, because they don’t understand our system, but at least they can give us an informed view of whether the science that is being presented is excellent or not.

So, it is not easy, but I would suggest that we don’t throw it away because it is not easy. We have got to try and work on it and continue to laud a system that has peer review.

Jim Peacock – I think we should remember many of the things that Bob said, but particularly, in response to Stephen’s point, Bob made a strong point about understanding the narrative of science, and that we need to be able to do that so that our decision makers, as well as the general public, really understand that. I think if we achieve that, we will be a long way forward.