MEASURING EXCELLENCE IN RESEARCH AND RESEARCH TRAINING
Canberra, 22 June 2004

Groups report back
Phil McFadden (Chair)

Jim Angus (Group 1)

We started with research. We believe that what we must achieve is public confidence, and that there is value in our R&D because what we do in our research sector will apply, as we said, across to defence, security and many other things.

We need some clarity when we are talking about quality versus applied research, the application. We need to develop, within our own sector, our scorecard – whether it be university, CSIRO or whatever – and within that scorecard we have to indicate what are the priorities for that sector.

We spent quite a bit of time talking about the concentration of minds. What do we believe that means? Is it at the department level? When it comes to an institution, would centres make up the quality or the concentration of that institution? This is particularly important if we think about block funding: would there be a university, for example, that may only be excellent in one centre, and therefore that would be its baseline for funding, rather than thinking about whether there should be a baseline across the sector in the universities for block funding.

Then in considering our quality we were interested in discussing where we are going to get the expert advice. And we must not forget the international competitiveness of our research, and therefore of course not be at all shameful or worried about going for an international panel.

That was what we must achieve in relation to research. What we must avoid, we thought, was the ‘soft’ or the ‘safe’ research that often is developed from a research fellow who may have one technique and continue to slice that salami for the next 10 years. We want researchers to take risks and develop their own research teams in that way.

We need to avoid the international cringe. If we are publishing in international journals we are doing international research, by that peer group that assessed that research. We have to avoid, in any research assessment exercise, the abuses that occurred after the first phase in the UK. We had a quiet chuckle about whether or not, whatever we bring into Australia, there will be abuses and people will try and avoid (or use) the system. So we need to be preparing version 2 as we start version 1. And we need, of course, to avoid any heavy workload – use the light feather touch – on any research assessment exercise.

As far as research training is concerned, we believe that we must, in our new PhDs, recognise the changed culture of research in Australia and therefore develop in our graduates these broad attributes. Mainly, of course, today it is around the teamwork, particularly in the sciences but also within the humanities. We don’t want the PhD students, because of lack of facilities, to do their PhD at home. They have to come into the university, and one way of warming them up is to get them to teach. So some teaching is terribly important, particularly in the humanities but also in our sciences – to get them into the research labs and the practical teaching labs as we do.

We need, of course, to encourage this capacity to write. It is not only research grants but it is the mini-review that leads on to the important review as it leads into the first chapter of their thesis.

Finally, we really need to look at the better resourcing of our research students. It is not good enough to accept research students in a laboratory at the moment unless you have got other grants, because you are needing those other grants to cross-subsidise. So I believe that is a really important area that Australia needs to look at carefully in resourcing our PhD students.

What we must avoid, I think, is the pressure to complete in four years or less. This is coupled with the 3–3½ years of the APAs, and if we are serious about developing a good PhD graduate we certainly need, in most cases, more than four years to do so. We need to develop opportunities to stop a PhD student, I believe, at the end of the second year and make a very tough decision in many cases, where you might say, ‘Let’s complete a Masters degree and leave the system.’ I think for many students we would be doing them a favour.

Lack of resources again we must avoid. It is not just the wet labs but also spaces for PhD students in the humanities. Many universities are struggling there with the increase in numbers.

And, finally, we must avoid poor supervision. It is all about developing that relationship with a very good supervisor in a good laboratory that builds to that excellent centre which is going to develop so much and be so important in developing the quality of our PhD graduates.

Tom Clark (Group 2)

Most of these points are more or less by consensus within the group, but there might have been more disagreement than time allowed us to explore.

We made a couple of preliminary observations, first noting that a pivotal strategy in achieving Finland’s outstanding performance on the graph we saw was targeting ISI journals; that has boosted their numbers hugely. We also began with a questioning of the notion of so-called research training. Leon Mann put it much more dispassionately than I would when he said, ‘It’s much broader than that.’

I am afraid we kind of flowed from one question into another, but I will try and let you know when that happened.

On Question 1, we thought that postgraduate research education should lead to maximal graduate satisfaction. I hope that is pretty clear. The next point in our summary is about ANU but I will come back to that, because actually the ANU has done a major exercise recently.

We have to account for the diversity of higher degrees by research. That is a diversity of modes, a diversity of disciplines, a diversity of ways of interacting with campus and off-campus experience, of industry versus traditional, et cetera.

We must take account of the quality of research that higher-degree research candidates undertake. This goes back to that question of research training. HDR candidates are researchers, and they are trying to prove something about their ability as researchers at the same time as they are learning.

Probably still on Question 1, but starting to fade: we need to accept the use of proxies if we are taking a metrist or measurement approach. I think most of the points here are a bit even-handed about the question of whether we are taking a metrist approach or a peer review dominated approach – the current numbers system or the mooted possible alternative RAE style system – but this is one point specifically about metrism.

Back on research training, so-called: we need to avoid a fixation on completions for their own sake. I can think of an adviser who had carriage of David Kemp’s research policies, Andrew Norton, who was a very proud non-completion in political science.

This is definitely moving more generally into research. We need to develop measures and expectations that are calibrated finely by discipline. We need to be clear about the difference between measuring quality, which is a relative judgment, and detecting excellence, which is an absolute judgment. I don’t think I was alone in the room, in the sense that if you talk about measuring excellence you really need to deal with a certain paradox of the language there. You can measure quality in ways that you are reduced to detecting or discerning excellence.

We need to support the ethical in research and promote it, and our policy framework should do that.

The ANU’s research exercise gives a lot to learn from. It is a big case study, and I think that all of us who are interested in this need to look at what is going on in the ANU right now and into the future, to get a sense of where that takes us. (I am sure the government will be doing the same.)

In ‘telling the story’ we need to be observant of the needs and aspirations of communities with a stake in our research. Our attention was particularly drawn to research on Indigenous topics and to research with a bearing on disadvantaged communities, with a more general view to the breadth and diversity of the outcomes we report.

We had a few final points. These are getting fairly general, I guess.

We need to avoid simple bean-counting techniques like the bibliometrics – which need not be simple bean-counting but to date have been. We need to promote and assist institutional generational renewal, and we need to pick measures or expectations that highlight this. I think our system should be encouraging departments and institutions to account for how they are preparing for the future of research, not simply how they are standing on their track record of the recent past of their research.

And the last point: for applied research, at least, we should incorporate the end-user feedback better than most of us do. Some of us I think are setting good examples here that we could all learn from.

Neil Furlong (Group 3)

Many of the points that I am going to talk about have already been covered, so I will be brief.

There are some general points at the start. The group talked quite a lot about the system in general terms: it must be fair, equitable and recognise the diversity across the system.

We also had some discussion about the difference between quality, as a spectrum of activity, and excellence, which tends to suggest the top end of that spectrum. And in understanding that what we are doing here is providing a system of discussion around future funding, we probably need to decide where we want that funding to sit within that quality spectrum.

We spent a lot of time talking about research training and thought that should be quite top-of-mind. That probably has something to do with the balance of people in the group.

To move through to research training: the group thinks that whatever the framework is, it ought to deliver effectively to both student and institutional expectations, and be cognisant of the diversity in the student cohort profile and be flexible in its programs. We have heard about that already.

It must deliver in the expectation of quality supervision – that is a clear issue across the sector. And again to pick up the resources issue, it may well want to look at setting some benchmarks for institutions around minimum resource levels for research students, around balances, for example, in staff time and teaching loads, which often impact on the quality of supervision.

And I think as well the group is talking about the research training framework as recognising the research training programs against the National Priorities and in terms of career progression for the graduates.

We also used those magic two words ‘light touch’; we liked that. And I think in looking at research training programs there were a number of comments around where the responsibility for quality lies, whether it should be sitting in single student-supervisor pairs or whether it is actually an institutional responsibility. I think the latter won out.

To move across now to research: there was some discussion about the diversity of the system and the indicators. I think the group came to a view that the diversity in the sector would be looked at through various weightings across a single set of indicators. And there was some discussion under that heading about an area that has not necessarily been covered in detail today: community impact and engagement with community.

The third point is around a framework that recognises that excellence can flourish wherever it occurs – I think I am responsible for that one – whilst at the same time making the plea for adequate funding for wherever that excellence does occur.

I think the framework also needs to ensure that new starters in the system are not locked out and that they are encouraged to get into the research game. At the same time, however, we need to drive efficiency, across the sector. Maybe that occurs through greater collaboration.

To come to the end of our summary: we talked about avoiding a system that drives homogeneity – I think the diversity in the system is one of our major pluses – and avoids exclusion but again uses the light-touch approach.

John Beaton (Group 4)

We second everything that has been said.

Taking research training first: we suggest, in measuring excellence of any kind, the use of every measure that you can find, noting that there will be great differences between institutions and even between programs within an institution.

We think it is important to judge supervisors as well as those who are being supervised. We think institutional infrastructure should be judged, particularly as the capacity is highly variable, and note that there are very wide differences in the sector.

We would like to work toward ‘industry ready’ products where possible, and in order to make it possible we encourage industry to start contributing to the university sector so that we can do a better job of it.

We would like to think that feedback from graduates is something that is of particular value when it is five years post graduation rather than six months afterward, when the bitter taste may still be there.

We think it is important not to make any assessment procedures onerous, either for faculty or for students. We should try to make reviews a means to promote, rather than punish. And whatever you do, do not force all programs to dance to exactly the same tune.

Now with respect to research: we think the description of excellence must be extraordinarily broad, and we also note that it will evolve. Excellence in universities and publicly funded research agencies may be quite different things, particularly when you take into account things such as commercial-in-confidence research, and these need to be judged in such a way as to not disadvantage individual researchers.

We think that the recognition of differences in measurements within and between universities is extraordinarily important. Again we don’t want to see them dancing to the same tune, or trying to. And we think that we should recognise non-traditional means of excellence, not just publication. In other words, people who are developing things which may not reach publication but which have an important impact, either in their university or outside of it, should be rewarded for it.

We think it is important to trust peer review, and I will come to that again in just a moment. Also we think it is important to introduce confidence in the system so that those people who are subject to it are quite happy to be involved in the review process.

We suggest that we do not reduce the complexity in such a way that it makes it terribly convenient for someone sitting in DEST to be able to say, ‘The chemists are worth 2.4 and the poets are worth 2.3.’

We say not to manipulate the metrics in order to suit some presumed outcome that someone else may have in mind – a government-driven policy, perhaps – and not to compare full-time and part-time researchers. We know that some people have full-time research appointments and others do teaching and research, so comparing the research output of these individuals is unfair to certainly one of the two groups.

Finally, we don’t think it is important to rely on citation indices.

OHP: Research Quality Assessment
(Click on image for a larger version)

I will, however, take a moment of promotion here and move this a little bit further. In judging research quality assessment, if the goal is for DEST at the top of this page to be able to provide funding from top to bottom, and if it is the job of people in these various hierarchical structures here to report upwards toward DEST, I would suggest there are a couple of things we want to keep in mind.

First of all, qualitative assessment – that is to say, peer review – is extraordinarily important down here [at bottom of slide] at the review of teaching and research individuals. There is no simple metric, certainly not citation indices, because we know that people quite often have terribly valuable manuscripts that have been accepted for publication, for which there is not yet a citation index, and these things need to be rewarded. There are hundreds of other reasons why you do not want to use those simple metrics.

Similarly, when you are talking about quantitative assessment, where these simple metrics come into play, they are probably more important as we go up the hierarchy here, where measuring quality between universities may be reflected better in things such as total citation indices for the total faculty.

There is one other little aspect about this. We have to move between these levels in the hierarchy, and there is something that we cannot live without here: the concept of trust. If departmental programs, faculties, colleges cannot trust the advice of people who are doing faculty peer reviews, then there is no point having them. If universities can’t trust what their deans are telling them about the quality of their programs, then there is no point asking them. So we do have to, we think, encourage government to accept the fact that if they want a qualitative assessment up here [at top level, DEST] by which they can make wise judgments, they have to accept and trust the qualitative assertions of the other levels of the hierarchy.

Ian McMahon (Group 5)

First of all, talking about research training: the group felt that the assessment should look at the benefits, first of all to the student, then to the government and the community, and then to the discipline. So in the sense of the student, what the assessment would be looking at would include the quality of the supervision, the infrastructure, including everything from the facilities – whether it be at an office, the equipment, the desk – to the research expertise of the staff and the supervisor, and then the employability of the degree from the student’s point of view and also the research excellence of the output and the quality of the research that is being done.

For the government and community, that would include the skills that the students have got and their employability, but you would be looking at assessment perhaps from feedback from employers. And then for the discipline: has the research training influenced the discipline? It comes back again to the quality and the excellence of the research.

In terms of some ways of assessing this, you would certainly look at exit interviews. Another group talked about talking to the graduates five years afterwards, and we felt that that was also appropriate, because you get quite a different view of what the student thinks after they have had a bit of experience in the workplace.

In terms of what to avoid, there should be an emphasis on quality rather than quantity, so you avoid purely quantitative measures. You should avoid treating each discipline the same, but also avoid treating each university, each sector, the same. Different universities and different government-funded research organisations, if we are talking about them, have different objectives.

Avoid training for training’s sake; avoid blunt, simplistic measures; avoid focusing too much on inputs. (Obviously, part of the assessment will be input-based.) Avoid conflict with institutional management and governments; and avoid too great an emphasis on completions for completion’s sake, perhaps looking at the appropriate time instead. At the moment, as was pointed out, students often lose their scholarships after 3½ years and the RTS funding after four years. Rather, you should look at completions in terms of what is actually needed for the degree and set mechanisms and assessments in place that, yes, award quicker completions but only where that is appropriate for the discipline and the work being undertaken.

For research, the emphasis on what we wish to achieve is on excellence. As to how we would measure that, certainly international benchmarking would play a part. That would include the usual measures that we use already, including success in getting grants and continuing to get grants; citations; publications, but with an assessment, perhaps, by an expert group; creativity and innovation – the implications for the discipline, whether it changes the discipline paradigm – industry links and funding.

It was felt that one of the purposes of the assessment would be to make Australia – the Australian community as well as the government – feel good about Australian research and to therefore influence government policy. So you are providing an assessment that enables the government to, in effect, give you more money but to be satisfied that the research it is funding is value for money and meets the needs of government and the community.

In terms of what to avoid, this includes a bias against early career researchers; a bias that would disadvantage emerging fields; something that is overly bureaucratic and too expensive to maintain; unintended consequences; a culture of risk avoidance so that people are not prepared to do research that is of high risk; and complacency.

Mike Sargent (Group 6)

Let’s have a look at research training. I think what we want to do is to achieve a framework which recognises the diversity of the paths that lead to a research degree, say a PhD. That is a diversity that varies in accordance with the types of discipline you are working in or the type of research you are doing. It should also recognise the need in many cases for mobility, which is both mobility after you do your research program and mobility in terms of collaborative supervision arrangements.

Most importantly, what we think is that the framework ought to value the quality of the research training, not just the quantity. That will require some consideration of the program itself and the students, supervisors and examiners. We also felt that a big element of research training is the creation of that independent, innovative thinker and so the framework has to recognise that.

In the framework we should avoid reducing innovation within the system so that we all become very vanilla-shaped, and we should try and avoid bureaucracy, because what we are trying to do is create a dynamic training system rather than a paper-warfare system.

In terms of research, we think that it is important to achieve an integrity of a peer review process that is respected and understood. It needs to be international in context and it needs to be respected not only by the funders but also by the research institutions themselves.

The framework also needs to recognise the diversity of the ways in which research is conducted. So when we are talking about the unit of assessment it may in fact vary in accordance with the discipline or transdisciplinary groups that are being assessed. We also think that the process ought to make sure that we don’t eliminate potentials for collaboration.

The third point there is that fundamentally, by and large in Australia, particularly with our volatile research groupings, it is seen to be essential that we be able to reassess, if required, somewhere in the middle of the extended period in which the assessment will take currency.

Finally, it is important for the framework to recognise potential rather than simply current status. In that respect we need to avoid too much retrospectivity and keep ourselves thinking about what we want to do rather than what we have done.

The framework will have to be careful to not bias against and destroy niche universities and institutions which might have very narrow but excellent research activities. And, finally, if you were lucky you might be able to find a system that avoided Heisenberg’s principle and that would be self-adaptive to avoid the game-playing that tends to fill our systems.