MEASURING EXCELLENCE IN RESEARCH AND RESEARCH TRAINING
Canberra, 22 June 2004

Excellence in social sciences in the context of impact on public policy
Dr Valerie Braithwaite

Thank you for the invitation to share some thoughts with you today. I am delighted to be here. My message is a simple one. Judgments of excellence in the social sciences in terms of impact on public policy must be based ultimately on the same criterion used in other areas of scholarly activity – publications in refereed journals and research monographs. There may be other aspects of engagement with the public interest that we would like to encourage, recognise and reward because it testifies to or even adds to this excellence, but fundamentally our established and accepted criterion should remain our bottom line – for appointment and promotion of staff at the individual level as well as for the funding and recognition of budgetary units, or universities for that matter, at the aggregate level. I now want to set out my reasons for thinking this way.

First of all let me explain the varieties of ways in which social scientists contribute to the public interest – and I am using public interest instead of public policy (which is on your program) quite deliberately because some of our most important contributions are through private actors and organisations, and not directly through shaping government policy – and I will say more about that at the end. But for the moment I want to focus on the form that our engagement takes and package the many different forms into three images.

The first is the public intellectual, those academics whose class room extends beyond campus as they disseminate knowledge and ideas through the media, through documentaries, through web sites, newsletters, public meetings, chairing public enquiries, or through communication with more elite and closed groups, as is the case when one becomes a member of a board or committee, or provides private briefings or seminars to outside groups. In an era where the controlling of information is accepted as commonplace – even legal - and spin is on everyone’s lips, the role of the public intellectual is more valuable than ever before, and is likely to become increasingly significant.

The second image is the one that I suspect most of you had in mind when you read your program. We might call this the academic guru – someone who has developed expertise in a particular area and is sought out by government to provide an answer to a problem that they have. One might think of the HECS scheme and Bruce Chapman’s contribution in turning an idea about funding universities into a reality – or one might think of the Compliance Model which we developed with the ATO as a basis for their administrative practices and which actually led to the setting up of our tax research centre at ANU.

The third image is one that is relatively new on the scene and one that I expect – indeed hope - is being given a boost through ARC funded partnership schemes. We will call this third image the academic-community partnership. Here community refers to any entity that is outside the university, the emphasis is not really on the term 'community' but rather on the term 'partnership'. While academics are providing knowledge, systematising observations and data, and drawing inferences that will address problems of mutual concern, community partners also are providing knowledge and observations, and testing out the ideas and explanations of the academics for their feasibility and plausibility. Both sides are learning from the other, and in this process, new social science theory is being developed, tested, refined and so on. The partnership is defined by theory building that addresses the issue of concern and problem solving that seeks to implement theoretical ideas in ways that are desirable and practicable. The partnership means that both sides – academics and non-academics – contribute to creating new knowledge, one through theorising, the other through doing, but each using the other to piggy back their next set of objectives.

So much for the form that engagement with the public interest may take. Now let us consider how that engagement occurs, the processes that are involved. I think it is useful to visualise this as a continuum from contributions that are direct extensions of one’s expertise (and therefore require relatively little investment or involve relatively little distraction) to those that involve taking time out or making significant detours from one’s mainstream activities to contribute to the public interest. Attached to the latter are far greater opportunity costs – most of us would prefer to be writing an article than attending a community meeting on the need for more nursing home beds or trying to understand how penalty regimes for tax non-compliance can be more efficiently implemented; but in both these cases, our engagement is vitally important to the overall mission of universities – to create and to disseminate our knowledge.

Given the range of things that social scientists do in their efforts to serve the public interest – and given my clear commitment to the importance of these endeavours - why am I not prepared to present an argument in favour of assessing the excellence of these activities in and of themselves? One reason is justice, the other is integrity.

The issue of justice comes into my argument in this way. Academic culture rests heavily on an unflagging conviction among its members that it is a meritocracy – that those who are best at creating knowledge and disseminating knowledge rise to senior positions and do so more quickly than others. Judgments about the worth of an individual in this regard are made through peer review. Now no-one has been a stronger critic of peer review processes than I, but while I believe there are many ways in which they can be and should be improved, it has never dawned on me that we should deny them the pivotal role that they play in regulating our activities. I am certainly strongly in favour of opening peer review processes up so that diverse voices can be heard and outside stakeholders can persuade or speak on behalf of academics who have gone the extra mile in serving public interests, but ultimately the decision about excellence in research that impacts on public policy has to be a decision that has academic merit as its base – that is, public policy contributions must conform to the standard of creating and disseminating knowledge. Peers can interrogate contributions to the public interest as well as anyone else – they can question whether a policy success story is a case of someone being in the right place at the right time, or reflects ingenuity and creativity in moulding current knowledge to suit a particular situation, or whether the ingenuity and creativity came from the bureaucracy itself rather than being academic input. Of course, they won’t always get it right – but they have more chance of getting it right than an outside body counting the number of media encounters, or adding up dollars in external earnings.

The second issue I mentioned was integrity. My starting point here is that universities are important as defenders of knowledge acquisition and its dissemination. I am suggesting that to downplay publications as the base for evaluating contributions to public interest is to compromise our collective mission too much. The academic guru was a label I chose with tongue-in-cheek. Once social scientists engage with the outside world and share their ideas, the amount of control they have over how those ideas are used and shaped disappears. The best of ideas can be implemented in ways that are absolute disasters, but after a period of time can be re-introduced with enormous success. In this process, it is critically important that ideas and implementation are documented, if you like separately – so that they can be disentangled further down the track. The researcher has the publication route for expressing and justifying the ideas, and perhaps even setting down guidelines for implementation. The outside partner has the opportunity to do the same from the implementation side. Both sources of data are important, and each can reflect well on contributions that the university has made to the public interest - but my point is that the ideas need to take precedence in any evaluation process, there should be no opportunity for covering them up or sidelining them, or dismissing them because they did not produce the outcome desired by the government of the day. The university needs to be unwavering in its support of the publication of such ideas in the public domain, and this standard is most likely to be maintained if publications remain our single most important indicator of academic merit.

Assessing excellence within our universities to our own satisfaction is one thing, being accountable to outside bodies in terms of our excellence is another. Here I think the answer lies at the level of the budgetary unit rather than at the level of the individual scholar; and the methodology should be one of story telling or narratives rather than of counting dollars or media events. Budgetary units that place priority on engaging with public issues should be encouraged to tell their stories. It seems wasteful to demand this of everyone, but targeted evaluation seems both practicable and useful for assessment purposes. The best of these stories can be verified in any number of ways, and if a university is seriously interested in having such contributions assessed as part of their 'excellence package' they can do so through relying on independent panels that can seek verification from stakeholders.

While this process seems relatively straightforward, I have concerns about it happening. In the main this is because it doesn’t fit neatly alongside the direction that evaluation is taking. First there is an assumption that detail is the essence of good evaluation and there is a reluctance to go with more holistic and subjective accounts of a centre’s activities. A second problem relates to the units used for evaluation. And often budgetary units that have a high policy profile fall between the traditional units - the individual at the most micro level and the topic areas defined in much the same way as the ARC defines them at the macro level. The solution is not the setting up another special category – again that perpetuates the same problem of splitting a coherent body of research in artificial ways, and in so doing, renders the contribution meaningless. Let me give you an example here. In the recent review that was held at ANU, our 5 academics sat down together to code our papers – we covered 7 may be more different topic areas – it looked like a dog’s breakfast - yet in reality we have a very focused and coherent research agenda. But this was entirely invisible in the research evaluation exercise. What we needed to do was to tell our story of research informing policy processes and policy processes informing research.

Finally I just want to say something about the label – social sciences and public policy – because although it is just a label it can impose blinkers that do not serve future scholars at all well. These days, we are becoming far more aware of the complexity associated with decision making – we talk about different nodes of governance – places within bureaucracies, in the private sector, in the community, local and international (NGOs for example), where key decisions are made that influence the course of events. In other words, if we are worried about carbon dioxide emissions, government may be the least likely point of leverage for academics wishing to contribute to this debate. They may choose to work directly with multinationals, or international environmental groups. Similarly, social scientists who are concerned about trade agreements that deny generic AIDS drugs to Africa and other poor nations may devote their activities to working with NGOs and serve the public interest in this way. Increasingly, social scientists are making their contributions through other decision makers – they are an interest group in themselves forming alliances with some and opposing others, all intent on shaping law and policy.

This again brings me back to my point about story telling, not just for purposes of demonstrating excellence but for purposes of establishing our credibility in the future. We should be constantly on the look out for our best stories to demonstrate to our colleagues that excellence in research and policy can co-exist and can simultaneously promote better social science theory and better practices of governance. Once the academy can accept that this is so, perhaps our critics outside universities will come to believe that we know what we are doing and can be trusted to both satisfy our intellectual curiosity and support the public interest.

Questions/discussion

Tom Clark – I wonder if it is worth teasing out an aspect of the view of excellence you are portraying. You used the word ‘conservative’. It seems to rest very much on track record. As I see one of the key strategic problems for Australian research, we have the problem of the future of Australian research, the future excellence of research if we are subscribing to an excellence doctrine. Academia is the second oldest work force in Australia after farming, I think, and it is only getting older. Just bearing in mind Bob Watts’ challenge to talk about future value, I wonder how in a track record-oriented, conservative, excellence regime you are appraising the future excellence – how we can be getting towards that.

Valerie Braithwaite – Is it all right if I answer this in the way in which I look after it in my own research group? I think that when we go to the universities as a whole I feel a little out of my depth, frankly. In my research group I have all young academics; they are all fresh out of PhDs. So it was a big ask to get them to work with the Tax Office, which is what we do, as well as to meet what I set as the performance criteria of three publications a year in refereed journals – that is what I asked of them.

It was hard, as I say; some of them struggled. But what we did – and the Tax Office paid for this, as an example of how these things can actually have unusual, positive side benefits – was that the Tax Office introduced a working paper series and it was much easier for them to get working papers out, because they were published as soon as they wrote them. And that became the base for seminars and then for refereed publications. Every working paper is turned into a refereed publication, ideally.

So I think that our young academics actually benefited. There are some things that are hard for them, working with me in this kind of environment, but they benefited because they immediately could see something that they had written in print, and they had a justification for keeping writing. One of the great dangers for young people who are working in that public domain is that they do a lot of talking but they don’t get the chance to write, which is what they need to learn to do.

I think it has worked quite well.

Bob Graham – I just want to congratulate you on a beautiful talk. It is clear that you adopt the Perutz Model of Management and you are very proud of your people, and I would suggest that that is part of the excellence in your track record. It is not just the publications you produce but it is who you produce. That doesn’t need to be a criterion but in your case I think it is and I applaud you for it.

Valerie Braithwaite – Thank you.

Alan Lawson – I was just going to take the opportunity, since you had begun to make a number of distinctions in the course of your talk, to raise the possibility that some of the confusion that we have when we are talking about these things arises from the slippage between excellence, value, quality, impact and so on. Those things are not the same, and I think that you were starting to get towards discussing that. I wondered if you wanted to say more about it.

Valerie Braithwaite – Yes, I think that is right. I may be oversimplifying things here, but I don’t think it is our excellence that is actually at stake, that government is questioning. They are questioning our credibility, our right to have taxpayers’ money, to be excellent in the way in which we all want to be excellent. And I think that’s something quite different.

On the issue of establishing credibility – and I am grouping value and all those other things you mentioned under credibility – I think there are many ways of doing it and when you work in different areas you will express that different ways. There are some areas where you can show dollar value, and if you can, why not. It is just that that standard should not be imposed on everyone. And I think it becomes much harder when you look at the public intellectual and, Iain [McCalman], it is hard to put the dollar value on some of the situations that you were describing this morning. But that is where I think the storytelling comes in. The storytelling encapsulates the excellence, but the real point of the storytelling is to improve our credibility. It is to give what we do credibility.

Jim Peacock – Valerie, we have been talking about the recognition of excellence, and we have kind of been avoiding the withdrawal of support for the not-excellence. I was wondering whether the storytelling has a risk of providing a generic cloak which permits the survival of…

Valerie Braithwaite – Yes, I know the point you are making. It depends at what level that storytelling takes place, I think, Jim. I can imagine a budgetary unit doing a very good job of spinning about their relevance and importance.

I am not sure of the appropriate level. I would be very comfortable telling a story about my research team, for instance, and then providing names for verification. I should also add on this verification issue that our centre is going to be finishing soon, and recently the ATO wanted to do an evaluation to decide whether their money was well spent. And they did do that evaluation. I had nothing to do with it; I decided that it was best to stay out of it.

But it was quite a hard-hitting evaluation. I got a copy of the draft of it, and in it I saw things like, ‘She doesn’t seem to realise that she shouldn’t be biting the hand that feeds her,’ and, ‘All she cares about is publications.’ There were some quotes in there, and of course I stewed on these for days and hours until my husband said, ‘Well, isn’t that exactly right? Wouldn’t you be offended if they were saying anything else?’

Out of that story of conflict and a lot of venting by the Tax Office came a [inaudible] where they were absolutely delighted and decided to continue. So I think sometimes we have got to be a little bit brave and put our credibility, if you like, in the hands of those that we feel aren’t always sympathetic to our cause. Of course, it is going to backfire at some times in some ways, but I think that is the answer to that question.