Peer Review & Poor Refereeing

[CraigS]

Ok, so we've brought this topic up in many threads recently, but we've never tackled it on its own (hence the new thread).

Here we go .. an article published yesterday (Sept 9, 2010) in Physicsworld.com:
"Peer review highly sensitive to poor refereeing, claim researchers"

Just a small number of bad referees can significantly undermine the ability of the peer-review system to select the best scientific papers. That is according to a pair of complex systems researchers in Austria who have modelled an academic publishing system and showed that human foibles can have a dramatic effect on the quality of published science.
...
The researchers created a model of a generic specialist field where referees, selected at random, can fall into one of five categories. There are the "correct" who accept the good papers and reject the bad. There are the "altruists" and the "misanthropists", who accept or reject all papers respectively. Then there are the "rational", who reject papers that might draw attention away from their own work. And finally, there are the "random" who are not qualified to judge the quality of a paper because of incompetence or lack of time.

So, what type are you ???

And just to balance things up and offer some constructive suggestions ..

When asked by physicsworld.com to offer an alternative to the current peer-review system, Thurner argues that science would benefit from the creation of a "market for scientific work". He envisages a situation where journal editors and their "scouts" search preprint servers for the most innovative papers before approaching authors with an offer of publication. The best papers, he believes, would naturally be picked up by a number of editors leaving it up to authors to choose their journal. "Papers that no-one wants to publish remain on the server and are open to everyone – but without the 'prestigious' quality stamp of a journal," Thurner explains.

So there you go .. there are scientists working on improving the system and thereby its quality goals. (As expected .. unless you're a pseudoscientist, I suppose).



Cheers
PS: Not that there's anything wrong with being a pseudoscientist.

[snas (Stuart)]

The researchers created a model of a generic specialist field where referees, selected at random, can fall into one of five categories. There are the "correct" who accept the good papers and reject the bad. There are the "altruists" and the "misanthropists", who accept or reject all papers respectively. Then there are the "rational", who reject papers that might draw attention away from their own work. And finally, there are the "random" who are not qualified to judge the quality of a paper because of incompetence or lack of time.

Human nature being, well, human nature, you must wonder if anyone is truly capable of being a "correct" referee on every occasion. But if all referees could be "correct" every time, that would create an ideal peer review system.

Thurner argues that science would benefit from the creation of a "market for scientific work".

Would a market driven system not lead to an increased rate of publication of "drivel", given that there would be no peer review system to filter out the rot?
How would a particular journal be able to claim itself as being more "prestigious" than the next if there is no peer review system to ensure the quality of published papers? It would seem to me (maybe I am wrong!) that a system like this would have the potential to bring all journals more or less into line with each other. I guess you may have one journal only bidding for papers of very high quality while a second journal (let's call it the "Thunderbolts Journal") might bid for more spurious papers and that would create a quality difference.
In my own profession we suffer from a number of fools who perform veterinary homeopathy. They will happily quote, as proof of their ludicrous claims, publications in "letters to the editor" sections of veterinary journals, or in the "Dog Breeders' newsletter" etc. These quotes mean nothing to me because I am a rational veterinarian, but when the average pet owner sees the same thing, they can be misled into thinking that a few paragraphs on quackery in a breed society newsletter can actually be valid information.

Stuart

[renormalised (Carl)]

A pseudoscientist rejects any sort of review, simply because it means their nonsense won't get the airing it so richly doesn't deserve. Pseudoscientist don't practice science, they practice quackery and as such should be ignored. That some actual happen across valuable scientific insights at times is just a random chance event, more a case of luck than good science.

The fact the guys who did this study use complex systems mathematics would immediately make them suspect in the eyes of the EU crowd. It'd be a conspiracy, according to them. The maths isn't important etc etc. This is precisely why peer review is there...to sort the wheat from the chaff, or in their case the fecal matter from the rest of the crop.

Now...peer review...The idea of a market place for work has some merit but it also has a lot of bad points as well. It could give scientists a means to get papers seen that normally aren't because of the variability of the peer review process between journals etc. But it could also make it almost impossible for young researchers to get a voice because they could get drowned out by their more experienced colleagues. Plus, as Stuart reiterated, you may also get a lot of garbage getting through that otherwise wouldn't.

[snas (Stuart)]

Oops
Craig, I forgot to add, what's wrong with being a pseudoscientist? Probably nothing. After all, it must be a very happy little place being right all the time, simply because you know that you must be right.

Stuart

[CraigS]

Stuart;
I think Mr. Pressure had something to say (at some stage in his quality contributions to us here), which started with something like:

'Do you want to be happier or wiser ?'

Pity we didn't get his next instalment on that one !

I gather he's right about whatever that one was about .... which must imply that he is pretty happy sort of guy !!


Cheers
PS: We're all pseudoscientists, (at times), if we're human ! (I was being semi serious about my original "PS").

[renormalised (Carl)]

Being happier, in his case (and in many other cases) means you haven't a clue about what's going on and when something ends up destroying your nice little self delusion, which it invariably does, the consequences can be rather disturbing. Nope, I'd much rather be wiser, knowing that whatever happiness I had was solely my own responsibility and not something I abrogated to some nebulous external influence or agency.

[CraigS]

Oops I was using the wrong terminology .. I meant "Pathological Scientist" not "Pseudoscientist".
Mr Pressure is a pseudo .. mere mortals can be pathological scientists, from time to time.

Apologies for that .. a slip-up.

Cheers

[renormalised (Carl)]

Yeah, but quite often the pathological leads to the pseudo, especially when the "mere mortals" try to run with it, thinking they have the answers (or believing they do).

No one truly has the answers, it's just that some know enough to be able to figure out what some might be, or how to go about figuring them out.

[higginsdj]

Originally Posted by CraigS

So, what type are you ???

In my daily work I'm in the 'Correct' category - thats my job - to peer review software specifications.

In Astronomy I don't peer review - I'm just not qualified (although I do comment where I feel appropriate)

Cheers

[CraigS]

So getting back to the paper ...

After running the model with 1000 scientists over 500 time-steps, Thurner and Hanel find that even a small presence of rational or random referees can significantly reduce the quality of published papers. When just 10% of referees do not behave "correctly" the quality of accepted papers drops by one standard deviation. If the fractions of rational, random and correct referees are about 1/3 each, the quality selection aspect of peer review practically vanished altogether.

"Our message is clear: if it can not be guaranteed that the fraction of rational and random referees is confined to a very small number, the peer-review system will not perform much better than by accepting papers by throwing (an unbiased!) coin," explain the researchers.

From the first post;
"rational": those who reject papers that might draw attention away from their own work;
"random" who are not qualified to judge the quality of a paper because of incompetence or lack of time.
and the types who do the best job (in the mix) are:
"correct" who accept the good papers and reject the bad and;
"altruists" who accept all papers and;
"misanthropists", who reject all papers.

So, maintaining the quality of the peer review process is critically dependent on keeping career-oriented, busy and incompetent types out of the process.

Seems logical ... (almost self-evident, really) but difficult to achieve in practice (?).
We need empirical evidence to take this one further !!

Cheers
PS: This could be good ammo for you David. Ie: That journal who published the work that's flawed ?

[renormalised (Carl)]

I wouldn't even call the "altruists" or "misanthropists" good for peer review either. If you have any amount of them in the process, you might as well let it be a "free for all anything goes" system, or, "damn them all to hell no one will publish anything because I said so" system. Where's the good in that??!!!!.

[CraigS]

Originally Posted by higginsdj

In my daily work I'm in the 'Correct' category - thats my job - to peer review software specifications.

In Astronomy I don't peer review - I'm just not qualified (although I do comment where I feel appropriate)

Cheers

Hang on a sec ...some time ago (30th Aug) you said you were an "Orthodox"!

Orthodox. I have my favourite 'kings'.

"Orthodox: dominated by the groupthink, following a leader’s opinion as in the “Naked king” tale, good workers performing monotonous tasks without ideas in large groups, specialists in a small field which they know very well, conformist, domestic. His/her dream is getting a permanent position at an university or research center, to be leader of a project, to do astropolitics. Most of them are like sheep (or geese), some of them with vocation of shepherds too."

.... Geez I'm mean ...!!

I don't know what I am !! ...er... I'm a Hamster !!
Depends on the situation !!

I'm only kidding .. don't take this, or me, too seriously !! ..I reckon you're doing a good job !

Cheers

[CraigS]

Originally Posted by renormalised

I wouldn't even call the "altruists" or "misanthropists" good for peer review either. If you have any amount of them in the process, you might as well let it be a "free for all anything goes" system, or, "damn them all to hell no one will publish anything because I said so" system. Where's the good in that??!!!!.

Daniel Kennefick, a cosmologist at the University of Arkansas with a special interest in sociology, believes that the study exposes the vulnerability of peer review when referees are not accountable for their decisions. "The system provides an opportunity for referees to try to avoid embarrassment for themselves, which is not the goal at all," he says.
Kennefick feels that the current system also encourages scientists to publish findings that may not offer much of an advance.
"Many authors are nowadays determined to achieve publication for publication's sake, in an effort to secure an academic position and are not particularly swayed by the argument that it is in their own interests not to publish an incorrect article."

Ahh .. those damn cosmologists getting in on the act again !!
They all think alike !!

woof !!

Cheers

[renormalised (Carl)]

Geez, I wonder how the EU crowd would categorise me, then

[CraigS]

Originally Posted by renormalised

Geez, I wonder how the EU crowd would categorise me, then

We already have that answer !!

woof !! woof !!

[renormalised (Carl)]

He is right, though. If you either reject or accept someone's paper for publication, then you should be able to justify why you made those decisions which brought you to that point. If you can't, then you have no business being in the position of being a reviewer.

Not all science or research is necessarily going to offer an advance on current thinking. It can't, simply because that's not what a lot of research is about and not everyone is capable of bringing about an advance in current thinking. For younger researchers there is a pressure to "publish or perish", in that the more you can get published, the more you'll get noticed. Once you get to the stage of being well known, then you can back off a bit. But you still have that pressure to keep up the work. It's usually the younger ones who are expected to come up with all the new ideas.

[CraigS]

Michèle Lamont a sociologist at Harvard University who analyses peer review in her 2009 book, How Professors Think: Inside the Curious World of Academic Judgment, feels that we expect too much from peer review. Lamont believes that we should never hope for "uncorrupted" evaluation of new science as all researchers are embedded in social and psychological networks. She feels that one way to improve the system, however, is to make assessment criteria more relevant to specific disciplines.

and the final gem is found right at the end of the article ....

This research is described in a paper submitted to the arXiv preprint server.

Classic stuff !!


Cheers

[renormalised (Carl)]

Originally Posted by CraigS

We already have that answer !!

woof !! woof !!

Indomitable Ogre of IIS....Harrying Harpy of the High End Thinking.... Damnable Dogged Destroyer of Dangerous Drivel

[renormalised (Carl)]

Can you imagine her trying to get that past a peer review board...."Heresy!!!!!. Peer review is sacred!!!!", and then it was consigned to the waste paper basket

[higginsdj]

Originally Posted by CraigS

Hang on a sec ...some time ago (30th Aug) you said you were an "Orthodox"!



"Orthodox: dominated by the groupthink, following a leader’s opinion as in the “Naked king” tale, good workers performing monotonous tasks without ideas in large groups, specialists in a small field which they know very well, conformist, domestic. His/her dream is getting a permanent position at an university or research center, to be leader of a project, to do astropolitics. Most of them are like sheep (or geese), some of them with vocation of shepherds too."

.... Geez I'm mean ...!!

I don't know what I am !! ...er... I'm a Hamster !!
Depends on the situation !!

I'm only kidding .. don't take this, or me, too seriously !! ..I reckon you're doing a good job !

Cheers

When it comes to astronomy I do have my 'Kings'. I run everything past them

Cheers