Reforming research assessment top-down bottom-up middle-out (solo)

Geri: 00:00:05

Welcome to Changing Academic Life.

2

: 00:00:08

I'm Geraldine Fitzpatrick, and this is

a podcast series where academics and

3

: 00:00:12

others share their stories, provide

ideas, and provoke discussions about what

4

: 00:00:17

we can do individually and collectively

to change academic life for the better.

5

: 00:00:29

Finally, I'm getting to the episode that

I wanted to do about following on from

6

: 00:00:34

my conversation with Karen Stroobants.

7

: 00:00:38

If you remember, Karen is a

researcher and policy advisor and

8

: 00:00:41

consultant, and in our conversation in

particular, she talked about her work

9

: 00:00:46

contributing to the development of

the European initiative called CoARA.

10

: 00:00:51

, C O A R A.

11

: 00:00:53

The coalition for advancing

research assessment.

12

: 00:00:56

And the agreement that comes out of that.

13

: 00:01:01

If you haven't already listened

to that episode, I encourage you

14

: 00:01:04

to stop listening to me now and

go and listen to that one first.

15

: 00:01:12

As always with these sorts of agreements.

16

: 00:01:16

There's the challenge of moving on

from actually having words on paper.

17

: 00:01:21

And signatories to agreements to actually

putting these agreements in place.

18

: 00:01:26

and that means.

19

: 00:01:27

Really significant change initiatives.

20

: 00:01:31

And in that episode, Karen.

21

: 00:01:33

talked quite compellingly about the

need for both top down and bottom up.

22

: 00:01:39

Buy-in.

23

: 00:01:39

If we're going to create

such research culture change.

24

: 00:01:44

And she also encouraged us to have

both the small conversations in

25

: 00:01:47

our every day situations as well as

at more senior management levels.

26

: 00:01:52

And about the systemic issues as well.

27

: 00:01:54

That need much more coordinated action.

28

: 00:01:59

What I'd thought I'd like to do in this

episode is to complement what Karen talked

29

: 00:02:04

about and flesh it out a little bit more.

30

: 00:02:07

, by providing a little bit

more detail about CoARA.

31

: 00:02:10

First of all, I thought it could

be really useful just to read out.

32

: 00:02:14

And name the 10 commitments of the CoARA

agreement, just because I think they do a

33

: 00:02:19

really nice job of summarizing the core.

34

: 00:02:22

Well, the core commitments, just

exactly as they say on the tin.

35

: 00:02:26

So I'm not going to read every item

fully, but just so you get the sense.

36

: 00:02:31

Number one commitment is to recognize

the diversity of contributions to and

37

: 00:02:36

careers in research, in accordance with

the needs and nature of the research.

38

: 00:02:42

Two.

39

: 00:02:44

To base research assessment

primarily on qualitative evaluation.

40

: 00:02:49

, supported by responsible use

of quantitative indicators.

41

: 00:02:54

Three abandon inappropriate uses

in research assessment of journal

42

: 00:02:59

and publication based metrics.

43

: 00:03:01

Like the general impact factor in H index.

44

: 00:03:05

Four.

45

: 00:03:06

Avoid the use of rankings of research

organizations in research assessment.

46

: 00:03:12

Five.

47

: 00:03:14

Commit resources to reforming

research assessment to achieve

48

: 00:03:18

the organizational changes.

49

: 00:03:22

Six.

50

: 00:03:23

To review and develop research assessment,

criteria, tools, and processes.

51

: 00:03:29

Seven to raise awareness of research,

assessment reform, and provide transparent

52

: 00:03:34

communication, guidance and training.

53

: 00:03:39

Eight to exchange practices and

experiences to enable mutual learning.

54

: 00:03:44

Nine to communicate progress made

on adherence to the principles and

55

: 00:03:49

commitments And ten, to evaluate

practices, criteria, and tools, and so on.

56

: 00:03:56

I imagine that these are all

commitments that we would be

57

: 00:04:01

really keen to sign up to.

58

: 00:04:03

And indeed.

59

: 00:04:04

As at the 31st of July is

reported on the CoARA webpage.

60

: 00:04:10

They've had 608 organizations that

have signed up to these 10 commitments.

61

: 00:04:17

And , as I mentioned around Karen's

conversation last week, they're currently

62

: 00:04:21

setting up lots of the working groups

that are trying to do that work of

63

: 00:04:24

putting the commitments into action.

64

: 00:04:29

And we know of course that this is going

to be a challenge at a European level.

65

: 00:04:32

Since Europe, isn't a homogeneous.

66

: 00:04:36

Region.

67

: 00:04:36

It has lots of different countries

and institutions and cultures.

68

: 00:04:40

And so there is a lot of work, a lot

of work to practically translate.

69

: 00:04:45

, the principles of the

commitments into action.

70

: 00:04:50

But still, I find it really

encouraging to see this happening in.

71

: 00:04:54

And I'm excited to see what I think

is a real momentum for change.

72

: 00:05:02

So it's not just COARA , I can, we

can also point to some evidence of

73

: 00:05:08

more general moves for change that now

are showing up in other initiatives.

74

: 00:05:14

So another initiative at the European

level is something called LERU L.

75

: 00:05:20

E.

76

: 00:05:20

R.

77

: 00:05:20

U.

78

: 00:05:21

The league of European

research universities?

79

: 00:05:25

And that's an initiative that involves

23, what they call themselves as

80

: 00:05:29

leading universities and to paraphrase

directly from their document.

81

: 00:05:35

This league was based upon an exchange of

current practices and recent developments

82

: 00:05:39

at the universities regarding the

assessment of researchers in the context

83

: 00:05:43

of hiring promotion and evaluation.

84

: 00:05:47

in particular, the document I am

looking at is a LERU framework

85

: 00:05:52

for the assessment of researchers

and a position paper from: 2022

86

: 00:05:58

And they talk about developing a common

framework that can inspire and support

87

: 00:06:04

universities in the, in the context of

their hiring promotion and evaluation.

88

: 00:06:08

With an issue.

89

: 00:06:10

With an objective to reward and recognize

again that, that argument about the

90

: 00:06:14

diversity of profiles and contributions

and recognizing that we need all of those

91

: 00:06:19

in order to be a success, whether it's

in research education or in service.

92

: 00:06:25

And in this particular framework

document, they also talk about three

93

: 00:06:29

different perspectives of assessment

that I think are really interesting.

94

: 00:06:32

The first one is the

multi-dimensional perspective.

95

: 00:06:37

And here they're drawing attention

to the diversity of contributions.

96

: 00:06:42

They also talk about the

developmental perspective.

97

: 00:06:46

And here they draw attention

to issues around leadership and

98

: 00:06:51

innovation and collaboration.

99

: 00:06:53

And.

100

: 00:06:54

Point to the necessary transpersonal

skills that are needed to engage in

101

: 00:06:59

those activities and therefore the

need for us to develop ourselves.

102

: 00:07:03

As people in those interpersonal skills.

103

: 00:07:07

And the third perspective

that they draw attention to

104

: 00:07:09

is the contextual perspective.

105

: 00:07:12

And here they're pointing to.

106

: 00:07:14

Both professional circumstances, as

well as personal circumstances and

107

: 00:07:20

contextualizing peoples research.

108

: 00:07:23

In those ways.

109

: 00:07:25

And that this in particular being

very important for inclusivity.

110

: 00:07:29

So this particular research paper then

continues with examples and may also,

111

: 00:07:34

encourage experimentation of different

approaches to assessment and may also

112

: 00:07:38

call on policymakers and funders to

explicitly support such experimentation.

113

: 00:07:46

So just as another

example, similar to CoARA.

114

: 00:07:50

It's again, this thing of pointing to

recognizing and rewarding diversity

115

: 00:07:55

of profiles and contributions, and,

, Recognizing that we need all different

116

: 00:08:00

types for the overall success of

research and in the service of society.

117

: 00:08:06

And there are many other examples

that I won't go into in any detail.

118

: 00:08:10

And we can point to the Leiden

manifesto for research metrics from

119

: 00:08:13

2015, that's been quite influential.

120

: 00:08:16

, I can also point back to the episode

with James Wilsden, where he talks

121

: 00:08:21

about the metric tide report in the UK.

122

: 00:08:24

And.

123

: 00:08:25

Um, we can see that there are lots

of these discussions and initiatives

124

: 00:08:30

happening at national and cross national

levels, which points to some of the

125

: 00:08:34

systemic changes that we might need.

126

: 00:08:38

And of course, many of these

initiatives can also be traced back

127

: 00:08:41

to DORA the San Francisco declaration

on research assessment from: 2013

128

: 00:08:47

And I think DORA was particularly

influential in starting the discussion

129

: 00:08:51

about moving away from the use of

journal based metrics , or at least

130

: 00:08:55

driving the discussion a bit further.

131

: 00:08:58

, And looking at.

132

: 00:08:59

, how we could assess research on its

own merits rather than just on the

133

: 00:09:03

journal in which it was published.

134

: 00:09:07

And even more impressive

is the sign up to DORA.

135

: 00:09:10

So again, on their web page, they

report that since: 2013

136

: 00:09:15

23,936.

137

: 00:09:19

Individuals and organizations in

164 countries that have signed up.

138

: 00:09:27

So these are really indicating some big.

139

: 00:09:34

Initiatives internationally.

140

: 00:09:37

At least on paper for these

top down systemic initiatives.

141

: 00:09:44

And we can also see some top

down initiatives happening.

142

: 00:09:48

At more national levels as well.

143

: 00:09:51

Where at national level, they're starting

to actually think about putting some

144

: 00:09:56

of these initiatives into practice, at

least in terms of policies and funding.

145

: 00:10:01

By funding councils or relevant

government departments.

146

: 00:10:05

I know that the Australian research

councils have recently had a survey

147

: 00:10:08

looking for input on different

forms of research assessment.

148

: 00:10:11

Um, as we've said, the UK has

been doing a lot of work on this.

149

: 00:10:15

In Austria, where I'm

currently located there.

150

: 00:10:18

I've also been involved in discussions

with the relevant ministries in Austrian,

151

: 00:10:22

involved in different workshops.

152

: 00:10:24

Again, looking at how we might

reform research assessment.

153

: 00:10:28

And bouncing off a lot of the work that's

going on in these other initiatives.

154

: 00:10:34

One country though, that I'd

particularly like to draw attention to.

155

: 00:10:38

Is the Netherlands.

156

: 00:10:39

Because I think they're doing

particularly well, at least

157

: 00:10:43

looking in from the outside.

158

: 00:10:45

In setting in place, um, some of

the supporting processes for new

159

: 00:10:51

forms of research assessment.

160

: 00:10:54

A very simple document that I can

point you to is a: 2019

161

: 00:10:59

that involved universities and funding

councils and government agencies.

162

: 00:11:04

Called room for everyone's talent.

163

: 00:11:07

And it's short and sharp

and visually compelling.

164

: 00:11:12

And it's focus.

165

: 00:11:15

Is on how to rethink academic reward

and recognition systems, as it says,

166

: 00:11:22

And again, what I like about it is the

way that it promotes the diversification

167

: 00:11:26

of career paths and enabling people

to define their own career shape.

168

: 00:11:32

And then that shape becomes the shape

against which they are to be evaluated.

169

: 00:11:39

So they recognize the

diverse career paths.

170

: 00:11:42

They also focus on quality rather

than quantitative measures.

171

: 00:11:48

, Stimulating open science is another plank.

172

: 00:11:51

, Also recognizing the balance between

individuals and the collective.

173

: 00:11:55

And stimulating academic leadership.

174

: 00:11:58

We can start to see lots of

common themes across these.

175

: 00:12:03

So in terms of recognizing

that science requires both

176

: 00:12:07

individual and team performance.

177

: 00:12:08

Yeah.

178

: 00:12:09

This reflects some of the conversation

that we had with Tanita, Casci

179

: 00:12:12

and Elizabeth Adams about what

Glasgow university are doing.

180

: 00:12:16

For example, around.

181

: 00:12:18

Rewarding collegiality and their

assessment and promotion criteria.

182

: 00:12:27

Various such university councils

and funding councils have also

183

: 00:12:32

collaborated to produce a document

called strategy evaluation protocol.

184

: 00:12:39

And this sets out aims

and specific methods.

185

: 00:12:44

, as advice to be used to assess

research at Dutch universities.

186

: 00:12:49

And, , they often go through

this process of assessment every

187

: 00:12:52

six years at Institute level.

188

: 00:12:56

So this might be where faculties

bring in an external expert committee.

189

: 00:13:01

, the faculty produces a report on their

activities and performances over the last

190

: 00:13:05

six years on the external panel will, come

in and have meetings with various relevant

191

: 00:13:10

groups, , and discuss , the report with

them and write some sort of assessment.

192

: 00:13:18

And again, what's really interesting

about this strategy evaluation protocol.

193

: 00:13:23

Is that it's really trying to take into

account the issues around diversification

194

: 00:13:27

of career profiles and open science

and the importance of culture in

195

: 00:13:31

producing high quality research.

196

: 00:13:34

And so the assessment committee

is explicitly asked to look at

197

: 00:13:37

things like research, quality,

societal relevance, and viability.

198

: 00:13:44

And the unit that is reporting and

writing the report are encouraged to.

199

: 00:13:50

write their, report it gain in terms of a

more narrative qualitative based argument.

200

: 00:13:57

And wherever possible, still using

factual evidence, but not relying

201

: 00:14:02

on indicators is the report.

202

: 00:14:04

and isn't driven by the indicators.

203

: 00:14:10

And the guideline document.

204

: 00:14:12

Also has in the appendix, some

very specific examples about

205

: 00:14:16

how organizations might go about

reporting against the criteria.

206

: 00:14:20

It sets out.

207

: 00:14:22

So while this document is definitely

targeted and written for the Dutch.

208

: 00:14:29

Audience, I think any institution about

to go undergo a faculty review could pick

209

: 00:14:36

up this document and think about how to

reinterpret it for their own context.

210

: 00:14:42

I think it's really encouraging that

these sorts of initiatives are happening.

211

: 00:14:46

And that people are trying

to put it into place.

212

: 00:14:51

I'll put links to all of these

documents and initiatives that I've

213

: 00:14:54

referred to in the podcast notes.

214

: 00:14:56

So, , you can see, you can

go and follow up yourselves.

215

: 00:15:03

So I think there's lots of really

exciting stuff happening top down.

216

: 00:15:08

But we know that these initiatives.

217

: 00:15:11

Slow to make, to be put in place

that change on the ground takes

218

: 00:15:15

time and can be really difficult.

219

: 00:15:17

And it really does take

significant culture change.

220

: 00:15:23

And while we might have some

really encouraging pointers for

221

: 00:15:27

larger scale systemic change.

222

: 00:15:30

We still need the bottom up change of the

individuals every day, sitting on a panel

223

: 00:15:36

or discussing a promotion case and so on.

224

: 00:15:41

And that's challenging because many

of us have grown up in systems.

225

: 00:15:44

Where there's a certain sense of, I

don't know what we would call it.

226

: 00:15:47

Sort of sense of security or

safety or familiarity with

227

: 00:15:51

our quantitative measures.

228

: 00:15:54

And, you know, the seduction of

numbers that can also seem like these

229

: 00:15:57

quantitative measures are more efficient

and also more reliable and comparable.

230

: 00:16:05

So I know that even if we have

these international agreements, that

231

: 00:16:09

organizations have signed up to, or

if we have national initiatives and

232

: 00:16:12

national guidelines on take more

qualitative approaches in the end,

233

: 00:16:17

it does come down to you and me.

234

: 00:16:20

, as reviewers, as people sitting

on evaluation committees,

235

: 00:16:23

promotion, committees, and so on.

236

: 00:16:28

And that's where I think a lot of

work has to be done in terms of

237

: 00:16:31

how to practically operationalize

the aspirations and commitments

238

: 00:16:35

articulated in these documents.

239

: 00:16:36

And that's one of the

commitments of CoARA.

240

: 00:16:38

Around doing the communication

and guidance and training.

241

: 00:16:46

And ,we need to also feel like, , find

new ways to feel confident that

242

: 00:16:50

we're making good decisions, because

when we're talking about qualitative

243

: 00:16:53

assessments, and we're talking about.

244

: 00:16:56

individuals and institutions being

able to define their own shapes

245

: 00:17:01

of, you know, research profiles

against which they're evaluated.

246

: 00:17:05

And that's going to take a lot

more work because previously in

247

: 00:17:10

the past, we've relied on naive

notions of being able to do direct

248

: 00:17:14

comparisons, comparing and contrasting.

249

: 00:17:20

So I wanted to just share now some

examples that we might call bottom

250

: 00:17:26

up examples from my own attempts at.

251

: 00:17:29

Trying to walk the talk a little bit

around some of these initiatives.

252

: 00:17:37

So I can think about this

in particular, in terms of.

253

: 00:17:42

Being a referee, for example, in writing.

254

: 00:17:47

doing evaluations of someone's case

for promotion to professor, or if

255

: 00:17:51

I'm on an appointment committee and

being asked to evaluate candidates.

256

: 00:17:56

And one of the things that I really first

need to do is my own self reflective.

257

: 00:18:01

Self-awareness about trying

to check in with my own biases

258

: 00:18:05

and just being careful that.

259

: 00:18:07

I'm not being sucked in by H indexes

and looking to compare people,

260

: 00:18:12

just on H indexes is an indicator.

261

: 00:18:16

and we have the discussion recently

with Sarah Davies, where she certainly

262

: 00:18:21

critiqued our notion of excellence that

are based just on these indicators.

263

: 00:18:28

So if I'm a referee , and looking at

these, , guidelines and documents,

264

: 00:18:33

They've really challenged me to

think about how I might write a more

265

: 00:18:36

narrative qualitative referee report.

266

: 00:18:41

I'd like to give some specific

examples, all anonymized of course,

267

: 00:18:45

, of things that I've done when I've

been asked to assess a candidate.

268

: 00:18:49

Often still in terms of impact

factors and personal metrics.

269

: 00:18:54

You know, where people are still using.

270

: 00:18:56

Some of these and institutions are

still using some of these old criteria.

271

: 00:19:02

So one example of what I've done

when I'm writing a references.

272

: 00:19:07

Where I have been asked explicitly

to assess a candidate in terms

273

: 00:19:11

of impact factors and personal

metrics, such as H index.

274

: 00:19:15

I'll often make an explicit statement

that says something like this.

275

: 00:19:22

I'm very surprised to be asked to

explain the channels of dissemination

276

: 00:19:25

in terms of impact factors and

personal metrics such as H index.

277

: 00:19:30

This is out of step with many of the

current research assessment reform

278

: 00:19:33

initiatives, such as DORA CoARA.

279

: 00:19:36

Leiden manifesto, et cetera.

280

: 00:19:38

These initiatives make strong cases

for moving away from such quantitative

281

: 00:19:42

metrics or using them responsibly.

282

: 00:19:45

If they are to be used and focusing

more on quality indicators.

283

: 00:19:50

End of quote.

284

: 00:19:53

So I'll often make that explicit comment.

285

: 00:19:56

Speaking back to the assessment committee.

286

: 00:20:00

Before I then go on and provide some more

qualitative contextualized assessment

287

: 00:20:05

of that person's outputs and impacts.

288

: 00:20:09

Because I know that where committees

are still using these old indicators,

289

: 00:20:14

they will be influential still.

290

: 00:20:20

So one of the things that I do, , in

terms of trying to contextualize

291

: 00:20:25

work is where the case that

I'm reviewing is particularly

292

: 00:20:30

cross-disciplinary or multi-disciplinary.

293

: 00:20:34

And we know from research that

people who work in publish across

294

: 00:20:38

multiple disciplines can often

have challenges getting published.

295

: 00:20:45

So in the abstract, if they were

working with solely within a single

296

: 00:20:49

discipline and on a very well-defined

topic in an area that does tend

297

: 00:20:54

to have high citation rates.

298

: 00:20:57

We could see that this person's H

index could be deemed to be higher.

299

: 00:21:04

And so what I try to do in a referee

report is to contextualize their H index

300

: 00:21:10

relative to their multidisciplinary work.

301

: 00:21:13

So again, just to read an example

of something that I've written.

302

: 00:21:19

It needs to be acknowledged

that the candidates research is

303

: 00:21:22

extremely challenging because

it crosses multiple disciplines.

304

: 00:21:26

This is in terms of both

the knowledge and skills and

305

: 00:21:30

collaborations they need to draw on.

306

: 00:21:33

And the challenges of speaking

back to diverse disciplines.

307

: 00:21:36

Who all have different ways of

doing research and evaluating

308

: 00:21:41

and reporting on research.

309

: 00:21:45

End of quote, from that reference.

310

: 00:21:50

Similarly, , another issue that I

think we can help contextualize as a

311

: 00:21:55

reference referees is to contextualize

different types of research that people

312

: 00:22:01

do that may entail different timeframes

and complexities, not one being better

313

: 00:22:07

than the other, but just being different

and having different implications for.

314

: 00:22:12

Publication outputs and

you know, and the numbers.

315

: 00:22:16

So some of us might do research

where we can work intensively

316

: 00:22:19

for a month on a project.

317

: 00:22:21

I get some really deep,

interesting results and write them.

318

: 00:22:24

Um, up and get them published.

319

: 00:22:26

And other others office might be

working on projects that take a number

320

: 00:22:30

of years that might involve, I don't

know, doing some contextual research

321

: 00:22:34

and developing technologies and

deploying them and running for example.

322

: 00:22:38

Some full randomized control trials.

323

: 00:22:40

Afterwards.

324

: 00:22:42

And these sort of projects are at very

different levels of scale and complexity.

325

: 00:22:46

And so do have different

implications for publication cycles.

326

: 00:22:52

So that means that we really can't

compare one person, seven journal

327

: 00:22:56

papers in a year with another person's

one or two journal papers in a year.

328

: 00:23:01

When they're doing these very

different styles of research.

329

: 00:23:06

Again, as an example, I can read from

a reference for someone who I'm trying

330

: 00:23:11

to, again, trying to contextualize some

work for this latter style of research

331

: 00:23:15

that happens over longer periods of time.

332

: 00:23:19

So to, to read from that reference.

333

: 00:23:23

It's worth noting.

334

: 00:23:24

That the candidates publications are

based on years worth of research.

335

: 00:23:29

When many of their peers

publishe papers based on work

336

: 00:23:32

from a scale of months, if that.

337

: 00:23:35

Evaluation is a case in point.

338

: 00:23:37

The scale and difficulty of conducting

full randomized controlled trials.

339

: 00:23:42

Are exponentially greater

than the scale of evaluations.

340

: 00:23:46

Of many other.

341

: 00:23:46

The high quality publications

in this disciplinary.

342

: 00:23:49

Every.

343

: 00:23:49

Area.

344

: 00:23:50

Yet RCTs are what is needed.

345

: 00:23:53

If this research is to speak to the

clinical community and to impact

346

: 00:23:57

clinical practice and patient outcomes.

347

: 00:24:01

And this candidate is evidently committed

to having this impact and doing these more

348

: 00:24:05

difficult longer-term evaluations for the

sake of impact, not just publications.

349

: 00:24:11

End of quote there.

350

: 00:24:15

And this reflects what.

351

: 00:24:16

Karen talked about in the, the

decisions that we often have

352

: 00:24:20

to make as researchers about.

353

: 00:24:23

Are we going to do research because we

want to have impact or do we want to

354

: 00:24:28

build up or need to build up our CV?

355

: 00:24:31

And I do appreciate that.

356

: 00:24:33

Uh, you know, being at

different stages of our careers.

357

: 00:24:38

These sort of questions may create

pressures in different directions.

358

: 00:24:41

Of course.

359

: 00:24:47

I can give another example of the way

in which initiatives and documents like.

360

: 00:24:53

The CoARA agreement have really

helped me in writing fair

361

: 00:24:56

assessments of someone's performance.

362

: 00:25:00

Again, it has taken extra work, I

think, on, on my part as the reference

363

: 00:25:06

writer and it has required me to be

more considered and reflective and

364

: 00:25:11

to take more of a qualitative lens.

365

: 00:25:16

So one example is, where I was

asked to write an assessment

366

: 00:25:20

for someone's promotion.

367

: 00:25:22

Uh, to professor.

368

: 00:25:24

And this was from a university

that in, in their criteria

369

: 00:25:28

document that they sent to me.

370

: 00:25:30

Set out all the usual very standard

metrics around H index and impact

371

: 00:25:35

factors and level of funding and so on.

372

: 00:25:39

So, what I did at the beginning

of the letter was just clearly

373

: 00:25:43

named the elephant in the room.

374

: 00:25:44

If you like, where I

clearly said that, um, Yeah.

375

: 00:25:48

By some of the criteria that

the university was asking me to

376

: 00:25:52

judge the candidate on, They

weren't going to be very strong.

377

: 00:25:58

But then I could go on and say in

the evaluation report and to stop

378

: 00:26:05

at, to read from that reference.

379

: 00:26:08

However to only view the candidates

achievements through very narrow notions

380

: 00:26:12

of quantitative scientific excellence,

misses the significant and unique

381

: 00:26:17

contribution that this candidate has made.

382

: 00:26:20

The international standing

in the field and the diverse

383

: 00:26:23

outputs practices and impacts.

384

: 00:26:26

And if I may, I will address

these in, in the rest of the.

385

: 00:26:30

Referee report.

386

: 00:26:31

As I qualitatively judged their

promotion application, according

387

: 00:26:36

to the principles from CoARA.

388

: 00:26:38

I consider these

principles more inclusive.

389

: 00:26:41

And they explicitly recognized diverse

career paths and contributions,

390

: 00:26:45

as in the case of this candidate.

391

: 00:26:49

And of quote.

392

: 00:26:51

I was able to go on then and what I did.

393

: 00:26:54

and this was an institution that had

not signed up to CoARA, but what I

394

: 00:26:58

did was it wasn't just me saying are

they're doing great work or they've

395

: 00:27:02

had these other impacts I could

quote directly from the agreement.

396

: 00:27:07

So again, like I might say something

like the CoARA agreement specifically

397

: 00:27:11

promotes recognition of diverse

outputs practices and activities based

398

: 00:27:15

primarily on qualitative judgment.

399

: 00:27:17

, and supported by a responsible

use of quantitative indicators.

400

: 00:27:21

So.

401

: 00:27:21

I would put that in

quotes, um, from the CoARA.

402

: 00:27:24

The agreement.

403

: 00:27:25

And then I would also

point to the fact that.

404

: 00:27:28

The agreement also talked about

recognition for different types of

405

: 00:27:31

impacts beyond scientific measures.

406

: 00:27:33

So I would say.

407

: 00:27:35

, and it was here in particular that

this candidate was outstanding in their

408

: 00:27:39

ability to bridge, research and practice.

409

: 00:27:42

And they've had really

significant impact on practice.

410

: 00:27:48

So what the CoARA agreement gave me

was some specific language and some

411

: 00:27:54

credibility to argue for why I was

choosing to review the candidate

412

: 00:27:59

against a different set of assessment

criteria than what I'd been asked for.

413

: 00:28:04

, And then I could finish off.

414

: 00:28:06

I still need to go on and do the

work to make the arguments for why

415

: 00:28:10

they were significant impacts, but

it did allow me to sort of, again,

416

: 00:28:16

just reinforce and at the end of

that description of how they made the

417

: 00:28:19

impact, I could say, you know, these.

418

: 00:28:22

Valuable contributions that

researchers make to science for the

419

: 00:28:25

benefit of society, including diverse

outputs beyond journal publication.

420

: 00:28:30

So I could explicitly acknowledge

that they might not have had

421

: 00:28:33

the journal publications, but

they had lots of other impacts.

422

: 00:28:41

Another point that I often

get asked to do in writing

423

: 00:28:46

reference letters that I really.

424

: 00:28:48

, don't like at all is when I'm.

425

: 00:28:52

Asked, , and this was, this is from

a specific request that I received.

426

: 00:28:55

Yeah.

427

: 00:28:56

How does the candidate compare with strong

researchers in their or closely related

428

: 00:29:01

fields at similar stages of their careers?

429

: 00:29:05

The mention of specific

individuals would be appreciated.

430

: 00:29:09

So that's what I was asked to do.

431

: 00:29:13

And you may remember that.

432

: 00:29:16

One of the commitments from CoARA

was about avoiding the use of

433

: 00:29:21

rankings of research organizations

in research assessments.

434

: 00:29:26

What I did in response to that

particular question was explicitly to

435

: 00:29:31

say, no, I wasn't going to do that.

436

: 00:29:33

And again, to quote, uh, from the

reference that I wrote in this case,

437

: 00:29:38

I've chosen, not to name specific

individuals as point of comparison.

438

: 00:29:42

As I consider the candidates

achievements to stand on their own.

439

: 00:29:46

It's also very difficult to compare

researchers when this candidate has

440

: 00:29:50

such a unique disciplinary profile.

441

: 00:29:53

A comparison with anyone within

a single disciplinary area

442

: 00:29:56

would not be relevant or fair.

443

: 00:30:00

So even though the CoARA commitment

specifically talks about not using

444

: 00:30:05

rankings of research organizations, I

think we can extend that to interpreting

445

: 00:30:09

it as not doing the comparison of

individuals either specifically, when

446

: 00:30:15

we talk about the fact that we can have.

447

: 00:30:17

Diversification of career paths

and recognition of diverse types

448

: 00:30:21

of outputs and, and in particular

contextualizing people's performances.

449

: 00:30:27

And that makes it really ridiculous

to think we can directly compare

450

: 00:30:31

and contrast individuals.

451

: 00:30:35

We can also see the way in

which this might discriminate

452

: 00:30:38

against particular groups.

453

: 00:30:41

So even if a couple of individuals

might be comparable on some

454

: 00:30:44

measure, like, I don't know.

455

: 00:30:45

Let's just say they're

being five year, five years.

456

: 00:30:48

Post-doc.

457

: 00:30:49

How do you compare a person who's single

unattached at, totally focused on their

458

: 00:30:55

work at a well-funded institution.

459

: 00:30:57

He gets lots of researchers support.

460

: 00:31:00

How do you compare that person?

461

: 00:31:02

With someone who has new parenting

responsibilities, who's on a

462

: 00:31:06

precarious contract and at a

less well-funded institution.

463

: 00:31:12

Or how do you compare the really

selfish researcher who doesn't do

464

: 00:31:16

any service and just expects everyone

else to provide peer service in

465

: 00:31:20

reviewing their papers, et cetera.

466

: 00:31:22

Versus the people who are doing all

the work to organize the conference

467

: 00:31:25

or edit the journal that, that

first person's going to publish in.

468

: 00:31:30

So I just point blank, refuse to do that.

469

: 00:31:33

And again, I appreciate being able to draw

on some of these documents and agreements.

470

: 00:31:38

To argue why I'm doing that.

471

: 00:31:40

It's not just me being

difficult to get along with.

472

: 00:31:46

And in a similar way, another issue

that often comes up in evaluations

473

: 00:31:50

that we're often having to deal with.

474

: 00:31:53

, is around service.

475

: 00:31:56

And we know again from the literature that

women in particular are overrepresented

476

: 00:32:00

in performing service roles in departments

and faculties and in peer communities.

477

: 00:32:05

And often at a cost to

their own promotion chances.

478

: 00:32:11

And this is work that's often

under-recognized and under rewarded

479

: 00:32:14

when it comes to promotion and

appointments, especially when we.

480

: 00:32:19

Implicitly fall back or

explicitly fall back on.

481

: 00:32:22

On the quantitative metrics.

482

: 00:32:26

, so again, I, draw explicitly on

the language of the CoARA agreement.

483

: 00:32:31

And I might say something like, you

know, the CoARA agreement argues

484

: 00:32:34

that while the agreement focuses on

the specific challenges of improving

485

: 00:32:38

research assessment, the principles.

486

: 00:32:41

Should also be extended and reinterpreted

into broader academic service, , that

487

: 00:32:47

includes teaching and service to

society so that we can, and again, like

488

: 00:32:52

the, , recognizing such as the Dutch.

489

: 00:32:55

, Room for everyone's talent you about

achieving, recognizing that the

490

: 00:32:58

importance of collaboration and those

interpersonal skills and the service.

491

: 00:33:03

And , the Glasgow work at

rewarding collegiality.

492

: 00:33:09

So in all of these.

493

: 00:33:11

They're just some examples.

494

: 00:33:13

I've really appreciated having ,the

international national, ,initiatives

495

: 00:33:18

and agreements to draw on.

496

: 00:33:20

And just to try to give some

credibility to taking different stands.

497

: 00:33:24

And recognizing that it does take more

work to write a qualitative argument

498

: 00:33:28

and to contextualize, peoples profiles.

499

: 00:33:33

I'm also not sure, particularly how.

500

: 00:33:35

How the committee's receiving my letters.

501

: 00:33:38

, I read them or accept them.

502

: 00:33:40

I, I do know that, , in a couple

of cases, the candidate did.

503

: 00:33:46

Get their promotion or did get the job.

504

: 00:33:49

, to what extent that may have been down

to my arguments and I'd, I, I don't know,

505

: 00:33:52

but, , it, it has been good to at least

have these initiatives to draw on to, to.

506

: 00:33:58

Contribute.

507

: 00:34:02

So, I guess in some ways the

examples I've just given are

508

: 00:34:05

examples of bottom up, me taking my.

509

: 00:34:10

Opportunity as a reference writer

or a committee member or whatever.

510

: 00:34:15

To influence specific cases.

511

: 00:34:18

And I'm also hoping that they're

also a little bit middle out and

512

: 00:34:23

here I would define middle out as

middle out, being able to try to

513

: 00:34:26

influence both the specific decisions

being made on this specific case.

514

: 00:34:31

As well as trying to influence,

and, raise awareness, change some

515

: 00:34:36

of the practices and influence

some of the broader structural

516

: 00:34:39

issues or institutional, issues.

517

: 00:34:43

So I'm using the opportunity as a

reference writer or committee member

518

: 00:34:48

to raise awareness., because many

people on the committee may not

519

: 00:34:52

have even heard of some of these

initiatives or may still be really.

520

: 00:34:56

,set on applying old criteria.

521

: 00:34:59

Or at the very least they might

stimulate some discussions.

522

: 00:35:03

And that can be done in directly through

the letter, or if I'm actually sitting

523

: 00:35:06

on a panel or a committee I can directly.

524

: 00:35:09

Stimulate some of those discussions.

525

: 00:35:16

What I think is useful if you're

actually on a faculty evaluation panel

526

: 00:35:21

or an advisory board or whatever.

527

: 00:35:23

, is also to go and just check if that

institution has signed up to an agreement.

528

: 00:35:28

Because that can also provide some

very specific hooks when you're

529

: 00:35:31

sitting in a committee meeting to.

530

: 00:35:34

, remind people around the table

that the institution has signed up

531

: 00:35:38

to the agreement , and to promote

discussions about how they might be

532

: 00:35:42

going about actually operationalizing

and implementing the principles.

533

: 00:35:49

I think they might also be some bottom

up, maybe middle out influencing

534

: 00:35:53

by mentoring roles that we play.

535

: 00:35:56

, I know for example, one institution

had signed up to the LERU agreement.

536

: 00:36:01

And, , there were some members

of their faculty were needing to

537

: 00:36:04

write some new policy documents

for the faculty about promotion.

538

: 00:36:09

And they weren't even aware that the

institution signed up to this agreement.

539

: 00:36:12

So I was able to point them.

540

: 00:36:14

As a mentor to the, to that.

541

: 00:36:16

agreement.

542

: 00:36:18

Around, the pathways towards

multi-dimensional academic

543

: 00:36:22

careers that I mentioned earlier.

544

: 00:36:24

I was able to point them to that

document and, uh, that may have

545

: 00:36:30

been able to help them in write.

546

: 00:36:32

Some arguments in their policy,

reflecting the agreement that

547

: 00:36:35

the institution signed up for.

548

: 00:36:37

And they were aware that this might

have been contentious in the faculty.

549

: 00:36:42

You know, where there was still a lot of

emphasis on more quantitative measures.

550

: 00:36:48

And we can also influence others as

well in encouraging people to think

551

: 00:36:52

about how they might frame their CV or

their annual report or their promotion

552

: 00:36:57

cases or their job application cases.

553

: 00:37:00

In more qualitative terms.

554

: 00:37:02

Again, you know, you may be able to

reference some of these initiatives,

555

: 00:37:06

even if they're not directly asked

for even if a qualitative, ,narrative

556

: 00:37:10

CVS and particularly asked for.

557

: 00:37:13

And I know that there's a whole lot.

558

: 00:37:15

We still have to learn about working

with narrative CVS, both in terms

559

: 00:37:20

of how to write them as, as well

as how to evaluate them, because

560

: 00:37:23

they can be more challenging.

561

: 00:37:26

But I had a colleague recently.

562

: 00:37:29

Tell me that they chose to write their

annual report as a narrative CV.

563

: 00:37:34

Even that wasn't asked

for by the institution.

564

: 00:37:36

And they just said what a great experience

it was that it really made them step

565

: 00:37:40

back, and appreciate what they'd actually

achieved and the impact that they had.

566

: 00:37:46

And it made them feel

really proud of their work.

567

: 00:37:49

In a way that they're bullet listed.

568

: 00:37:52

old version of the CV didn't do at all.

569

: 00:37:59

And I'm also aware too, of how I

might role model that and thinking

570

: 00:38:02

about, Walking the talk a bit myself.

571

: 00:38:04

And I realized that on my web

page, you often had that bio

572

: 00:38:09

that just reported on that.

573

: 00:38:13

The numbers and the output

driven metrics driven type CV.

574

: 00:38:18

, So I've, I've tried to reframe.

575

: 00:38:21

At least some of the introduction

to, to put it more in terms of

576

: 00:38:24

what drives me and what I focus on.

577

: 00:38:27

that points to the impact

that I'm trying to have.

578

: 00:38:35

So just wrapping up, what I've tried

to do is just point to both some of

579

: 00:38:39

the, what I think are some interesting

top-down initiatives that are happening

580

: 00:38:43

in creating momentum for change.

581

: 00:38:45

And I'm also talking about some of the

ways that we might influence change.

582

: 00:38:51

From the bottom up and middle out

as individuals and being part of

583

: 00:38:55

trying to make that change happen.

584

: 00:38:57

And of course we can try to

encourage our institutions to sign

585

: 00:38:59

up to these agreements as well.

586

: 00:39:01

And you can also think about getting

involved in working groups, if you want

587

: 00:39:05

to . And I just also want to just say that

in sharing my reference writing examples.

588

: 00:39:10

I'm not asking now for lots of requests

to write references, I'm actually

589

: 00:39:13

going to be doing fewer of these.

590

: 00:39:15

If any, as I move into different roles.

591

: 00:39:18

But I'd encourage you to all think

about how you're writing your reference

592

: 00:39:22

letters or how you're playing out

your role sitting in committees.

593

: 00:39:27

And ways that you might contribute to

enhancing and improving and developing

594

: 00:39:31

our institutional policies and practices.

595

: 00:39:34

So I'd really liked to

encourage us all to do our part.

596

: 00:39:37

And see what we can do.

597

: 00:39:40

I think it will take some years for

it all to play out in real practice.

598

: 00:39:43

I think there's in part some sort of

generational change that's needed.

599

: 00:39:47

But there's momentum now towards this.

600

: 00:39:50

And we all want better forms of

research, evaluation, and assessment

601

: 00:39:54

that more sustainable and more

appropriate for the diversity of

602

: 00:39:58

contributions that we can make.

603

: 00:40:00

And we can all argue in different

ways, in different forms for

604

: 00:40:04

how we enable, um, diversity.

605

: 00:40:08

And recognize the importance of team

collaborations and diverse roles in

606

: 00:40:12

producing good science and recognizing

diverse forms of impacts and outputs.

607

: 00:40:19

And recognizing the value of good

leadership and good interpersonal

608

: 00:40:23

skills in the middle of all of this.

609

: 00:40:25

And in creating research

cultures that are sustainable

610

: 00:40:29

and collaborative and supportive.

611

: 00:40:32

And I'm excited that there are

moves happening in these directions.

612

: 00:40:37

Let's make it happen.

613

: 00:40:44

You can find the summary notes,

a transcript, and related

614

: 00:40:47

links for this podcast on www.

615

: 00:40:50

changingacademiclife.

616

: 00:40:52

com.

617

: 00:40:53

You can also subscribe to

Changing Academic Life on iTunes,

618

: 00:40:58

Spotify, and Google Podcasts.

619

: 00:40:59

And you can follow

ChangeAcadLife on Twitter.

620

: 00:41:04

And I'm really hoping that we can

widen the conversation about how

621

: 00:41:07

we can do academia differently.

622

: 00:41:09

And you can contribute to this by rating

the podcast and also giving feedback.

623

: 00:41:14

And if something connected with you,

please consider sharing this podcast

624

: 00:41:18

with your colleagues together.

625

: 00:41:20

We can make change happen.