MEASURING EXCELLENCE IN RESEARCH AND RESEARCH TRAINING
Canberra, 22 June 2004

Excellence in social sciences in the context of impact on public policy
Dr Valerie Braithwaite

Thank you for the invitation to share some thoughts with you today. I am delighted to be here. My message is a simple one. Judgments of excellence in the social sciences in terms of impact on public policy must be based ultimately on the same criterion used in other areas of scholarly activity – publications in refereed journals and research monographs. There may be other aspects of engagement with the public interest that we would like to encourage, recognise and reward because it testifies to or even adds to this excellence, but fundamentally our established and accepted criterion should remain our bottom line – for appointment and promotion of staff at the individual level as well as for the funding and recognition of budgetary units, or universities for that matter, at the aggregate level. I now want to set out my reasons for thinking this way.

First of all let me explain the varieties of ways in which social scientists contribute to the public interest – and I am using public interest instead of public policy (which is on your program) quite deliberately because some of our most important contributions are through private actors and organisations, and not directly through shaping government policy – and I will say more about that at the end. But for the moment I want to focus on the form that our engagement takes and package the many different forms into three images.

The first is the public intellectual, those academics whose class room extends beyond campus as they disseminate knowledge and ideas through the media, through documentaries, through web sites, newsletters, public meetings, chairing public enquiries, or through communication with more elite and closed groups, as is the case when one becomes a member of a board or committee, or provides private briefings or seminars to outside groups. In an era where the controlling of information is accepted as commonplace – even legal - and spin is on everyone’s lips, the role of the public intellectual is more valuable than ever before, and is likely to become increasingly significant.

The second image is the one that I suspect most of you had in mind when you read your program. We might call this the academic guru – someone who has developed expertise in a particular area and is sought out by government to provide an answer to a problem that they have. One might think of the HECS scheme and Bruce Chapman’s contribution in turning an idea about funding universities into a reality – or one might think of the Compliance Model which we developed with the ATO as a basis for their administrative practices and which actually led to the setting up of our tax research centre at ANU.

The third image is one that is relatively new on the scene and one that I expect – indeed hope - is being given a boost through ARC funded partnership schemes. We will call this third image the academic-community partnership. Here community refers to any entity that is outside the university, the emphasis is not really on the term 'community' but rather on the term 'partnership'. While academics are providing knowledge, systematising observations and data, and drawing inferences that will address problems of mutual concern, community partners also are providing knowledge and observations, and testing out the ideas and explanations of the academics for their feasibility and plausibility. Both sides are learning from the other, and in this process, new social science theory is being developed, tested, refined and so on. The partnership is defined by theory building that addresses the issue of concern and problem solving that seeks to implement theoretical ideas in ways that are desirable and practicable. The partnership means that both sides – academics and non-academics – contribute to creating new knowledge, one through theorising, the other through doing, but each using the other to piggy back their next set of objectives.

So much for the form that engagement with the public interest may take. Now let us consider how that engagement occurs, the processes that are involved. I think it is useful to visualise this as a continuum from contributions that are direct extensions of one’s expertise (and therefore require relatively little investment or involve relatively little distraction) to those that involve taking time out or making significant detours from one’s mainstream activities to contribute to the public interest. Attached to the latter are far greater opportunity costs – most of us would prefer to be writing an article than attending a community meeting on the need for more nursing home beds or trying to understand how penalty regimes for tax non-compliance can be more efficiently implemented; but in both these cases, our engagement is vitally important to the overall mission of universities – to create and to disseminate our knowledge.

Given the range of things that social scientists do in their efforts to serve the public interest – and given my clear commitment to the importance of these endeavours - why am I not prepared to present an argument in favour of assessing the excellence of these activities in and of themselves? One reason is justice, the other is integrity.

The issue of justice comes into my argument in this way. Academic culture rests heavily on an unflagging conviction among its members that it is a meritocracy – that those who are best at creating knowledge and disseminating knowledge rise to senior positions and do so more quickly than others. Judgments about the worth of an individual in this regard are made through peer review. Now no-one has been a stronger critic of peer review processes than I, but while I believe there are many ways in which they can be and should be improved, it has never dawned on me that we should deny them the pivotal role that they play in regulating our activities. I am certainly strongly in favour of opening peer review processes up so that diverse voices can be heard and outside stakeholders can persuade or speak on behalf of academics who have gone the extra mile in serving public interests, but ultimately the decision about excellence in research that impacts on public policy has to be a decision that has academic merit as its base – that is, public policy contributions must conform to the standard of creating and disseminating knowledge. Peers can interrogate contributions to the public interest as well as anyone else – they can question whether a policy success story is a case of someone being in the right place at the right time, or reflects ingenuity and creativity in moulding current knowledge to suit a particular situation, or whether the ingenuity and creativity came from the bureaucracy itself rather than being academic input. Of course, they won’t always get it right – but they have more chance of getting it right than an outside body counting the number of media encounters, or adding up dollars in external earnings.

The second issue I mentioned was integrity. My starting point here is that universities are important as defenders of knowledge acquisition and its dissemination. I am suggesting that to downplay publications as the base for evaluating contributions to public interest is to compromise our collective mission too much. The academic guru was a label I chose with tongue-in-cheek. Once social scientists engage with the outside world and share their ideas, the amount of control they have over how those ideas are used and shaped disappears. The best of ideas can be implemented in ways that are absolute disasters, but after a period of time can be re-introduced with enormous success. In this process, it is critically important that ideas and implementation are documented, if you like separately – so that they can be disentangled further down the track. The researcher has the publication route for expressing and justifying the ideas, and perhaps even setting down guidelines for implementation. The outside partner has the opportunity to do the same from the implementation side. Both sources of data are important, and each can reflect well on contributions that the university has made to the public interest - but my point is that the ideas need to take precedence in any evaluation process, there should be no opportunity for covering them up or sidelining them, or dismissing them because they did not produce the outcome desired by the government of the day. The university needs to be unwavering in its support of the publication of such ideas in the public domain, and this standard is most likely to be maintained if publications remain our single most important indicator of academic merit.

Assessing excellence within our universities to our own satisfaction is one thing, being accountable to outside bodies in terms of our excellence is another. Here I think the answer lies at the level of the budgetary unit rather than at the level of the individual scholar; and the methodology should be one of story telling or narratives rather than of counting dollars or media events. Budgetary units that place priority on engaging with public issues should be encouraged to tell their stories. It seems wasteful to demand this of everyone, but targeted evaluation seems both practicable and useful for assessment purposes. The best of these stories can be verified in any number of ways, and if a university is seriously interested in having such contributions assessed as part of their 'excellence package' they can do so through relying on independent panels that can seek verification from stakeholders.

While this process seems relatively straightforward, I have concerns about it happening. In the main this is because it doesn’t fit neatly alongside the direction that evaluation is taking. First there is an assumption that detail is the essence of good evaluation and there is a reluctance to go with more holistic and subjective accounts of a centre’s activities. A second problem relates to the units used for evaluation. And often budgetary units that have a high policy profile fall between the traditional units - the individual at the most micro level and the topic areas defined in much the same way as the ARC defines them at the macro level. The solution is not the setting up another special category – again that perpetuates the same problem of splitting a coherent body of research in artificial ways, and in so doing, renders the contribution meaningless. Let me give you an example here. In the recent review that was held at ANU, our 5 academics sat down together to code our papers – we covered 7 may be more different topic areas – it looked like a dog’s breakfast - yet in reality we have a very focused and coherent research agenda. But this was entirely invisible in the research evaluation exercise. What we needed to do was to tell our story of research informing policy processes and policy processes informing research.

Finally I just want to say something about the label – social sciences and public policy – because although it is just a label it can impose blinkers that do not serve future scholars at all well. These days, we are becoming far