#Diversity recap: Single-, Double-blind or open review?

Dear followers,

I am so sorry for the late recap this week, but it was my birthday (yay!) and life got in the way. But here we are. Last week we discussed: Would double-blind or open peer review help with diversity?

We came with some short questions to make our discussion flow easier, but I was not as good as Ian to moderate them in order this time!

So the issue is really complicated because it’s field-specific.Per review process dependent on how large and complex the field of study is. People from your field know your research and would be able to guess who the authors are. But they never really know, so, is it still helpful? So probably there is no answer that would fit all journals. But everybody agreed that it is pretty unfair for one side to know the other and not vice-versa. Single-blinded review protects the reviewers, but also hides them.

In an ideal world everyone should be able to speak up freely, be honest, and make meaningful critiques. But that also involves personalities. Some people may be able (or even more capable of) doing critiques when both parties are know to each other. Also, people would probably be more throughout with the revisions if they were all open. But what if you are a PD, in search for a job? Would you stand-up and critique the pope of your field? There may be some negative impact on reviewers; especially early career women and POC. And retaliation can come in other ways– if I reject your Science paper, will you block my grant? Maybe reviewers should identify themselves post-tenure. Won’t solve vendetta issue, but could protects early career professors.

We didn’t come up with an answer to the question, because probably there is not a single answer than would solve all our Diversity issues on peer-reviewing. But it was a nice discussion! Thanks to all that participated and hope to see you all next Monday (10/05/2015, 2pm EST).

Participants: Doctor_PMS, Emily Klein, Ian Street, Caroline VanSickle, Biochem Belle, Ruthie Birger, Kelsey Jordahl, Sarah Manka, Jaquelyn Gill, Luna Centifanti.

Would double-blind or open peer review help with diversity?

As scientists we all know that a good part of our professional success is “measured” by our publications. Not only the number, but also the quality of the research, as well as the impact factor of the journal it is published. Peer-reviewing is the method of judgment utilized by most of the scientific journals out there. However, everybody knows that there is bias involved in reviewing. This week on #DiversityJC we are going to discuss the following article:

Double Blind Peer Review

Hilda Bastian (@hildabast) also did review of the literature on the different methods of peer review & how there’s still a lot of uncertainty, especially surrounding how it affects diversity of what’s published.

The most common method of peer reviewing is the single-blind review. This type of peer review keeps the identity of the reviewer anonymous, but the name and affiliations of the authors are open to the reviewer. This allows the reviewer to evaluate a paper without any influence from the authors. In a double-blind peer reviewing system, both the identities of the reviewers and the authors remain anonymous. The other alternative would be a total open review, where identities remain public to both sides. But some questions remain:

Q1: Is double-blind peer review a realistic alternative?

Q2: Could open or double blind peer review mitigate bias?

Q3: Would an open peer-review system be more effective than the single-blind one?

Q4: Can an open peer-review system bring valuable critique to the authors?

Q5: Single-, double-blind or open peer review, which one brings more diversity?

Join our #DiversityJC discussion, next Monday Sept 7th, 2pm EST!

Doctor_PMS

Emily S. Klein

Ian Street

Student evaluations – how bias shows up when you’re just trying to get some honest feedback.

This week’s Diversity Journal Club (#DiversityJC), we discussed student evals and bias

What’s in a name? In student reviews of their professors, something that can be important for improving your teaching and potentially for your institution, do they rate men better?

Is this just another way in which bias manifests? Is this study just another example of the way we all bring stereotypes to bear in our everyday lives?

Ian Street (@IHStreet) further wondered if student evaluations of their professors might be a “particularly keen” way to identify implicit bias, because students rarely spend much time on them – they’re answered quickly and on impulse (I remember them being administered generally at the end of class, for instance).

And it’s not along gendered lines just for the students

Agreed. We are all subject to culturally constructed biases. As Ruth Hufbauer (‏@hufbauer ) pointed out, “the bias goes both ways.” Simply because we below to one group or another does not mean that we are immune to them.

Student reviews of teaching may be a clear indicator of bias, but they’re also another example of how that bias affects us. These evaluations are used by professors – and sometimes by schools. While not used in every institution and sometimes only in extreme cases. They still can mean something.

Or…

Biased student evaluations could be harmful beyond just the individual, too.


Are student evaluations really a tool we should be using? Kim Hannula (‏@stressrelated) and Anne Jefferson (‏@highlyanne) noted they didn’t think they were useful to identify good teaching, but perhaps to denote really bad teaching or inappropriate behavior. R. Deborah Overath (‏@scienceknitster) agreed that they weren’t a good metric – but raised the question of what to use in their place? Because even if they aren’t used heavily by a school, student feedback remains very important to professors who want to improve.

Moreover, they do have impacts on professors beyond giving incorrect information instead of helpful feedback. In a profession where we are already critiqued by our peers, negative feedback that has no real basis in teaching ability can be really impactful –

I’d say this is probably even more problematic for someone early in their careers (or is that just me?).


To work on the problem of evaluations, Ruth Hufbauer (@hufbauer) shared some ideas she had on evaluations: giving one to students that she read, and one she said went to her boss.

Others agreed that student evaluations needed to be improved, and some shared ideas on how to do that, such as evals designed specifically for each class (Anne Jefferson, @highlyanne), and as before and after “quizzes” (Ruth Hufbauer, @hufbauer). In response to bias in evals in particular, Laura Williams (@MicroWaveSci) recommended institutions stay informed on the research, and use it to improve the format and content of evals, e.g. to be very careful about wording and phrasing. AnalyticallyFabulous ‏@analyticallyfab noted an easy one: throw out any comment about appearance (this sounds a bit silly, but I can only imagine just how important this small step probably is!). Later, Abhi (@abhichand) noted:


Regardless of the method or how to address the evals themselves, the issue still remained. As Doctor PMS (@Doctor_PMS) reminded us…


In addition – stereotypes go well beyond gender!


And this makes for a really, really complicated picture.

And we’re back to understanding this study, and that similar conclusions would result if you could look along other stereotypes, as another to demonstrate how bias infiltrates how we live our lives and the actions we take.

And this, my dear twitter friends, is what we’re all here for and it’s not a problem we’re going to solve in an hour.


But we can start talking about it – and student evaluations are one way to really start doing that. In my opinion, they actually provide a really clear example of stereotypes that might prove useful for starting the conversation, and starting initiatives that get at the systemic problem…


Even if it’s tough, and even if it will take time, and those initiatives are about way more than student evaluations.

Moreover, as Laura Williams (@MicroWavesSci) further pointed out, faculty collectively need to be on board, and I’d add the administration too. I think this collective effort is critical, and Laura also noted that students need to hear about bias and its implications and importance in many classes. It needs to be a collective and repeated statement. In addition, there can be ways to get students thinking about bias as it pertains specifically to their evaluations.

Although, again, this would likely be far more effective if part of a larger conversation about bias that was happening at that institution. Part of education, and of a message students had heard and discussed before.


For me, this was another reason to have these conversations and to study bias with students as part of their education – as early as possible, certainly, but at the very least in college. We all arrive on campus as children, and leave looking forward towards being adults (or maybe you were one – I still had some ways to go). It’s where we screw around and screw up and learn on our own – where we discover, hopefully, what we want to do with our lives. Why not also start learning about being socially responsible too? At the very least, it might give your professors some more productive and real feedback.

 

Finally, a few articles shared by the group:

Dynamic Ecology on teacher evaluations: “Evaluating Teaching
NPR Blog “If Your Teacher Likes You, You Might Get a Better Grade

Another reference in response to how the RateMyProfessor tool has been reported came out after the convo, but is also of interst!


Thank you all for another great discussion, those that joined and just followed along! As always – hope I got everyone, let me know if I spelled anything wrong or missed any body. Give these kids a follow and hope to see you all at the next one!

Ian Street (@IHStreet)
R. Deborah Overath (@scienceknitster)
Ruth Hufbauer (@hufbauer)
Anne Jefferson (@highlyanne)
Rebecca Pollet (@rmpollet)
Kim Hannula (@stressrelated)
Laura Williams (@MicroWaveSci)
Amy Lossie (@ilovepigenetics)
AnalyticallyFabulous (@analyticallyfab)
Abhi (@abhichand)
V. Siva (@DrVidSiva)
Cassie (@mosquito_chaser)
K.A. Woytonik (@SonicWoytonik)
And even… PinkGlitteryBrain (@aiquintero) just under the wire! 😉

 

Please leave any additional thoughts or questions in the comments – or if you have more to say, consider writing up a guest blog and we’ll post it!


Til next time kids!


Emily (@DrEmilySKlein)
Doctor PMS (@Doctor_PMS)