Peer Review & Poor Refereeing

[CraigS]

Ok, so we've brought this topic up in many threads recently, but we've never tackled it on its own (hence the new thread).

Here we go .. an article published yesterday (Sept 9, 2010) in Physicsworld.com:
"Peer review highly sensitive to poor refereeing, claim researchers (http://physicsworld.com/cws/article/news/43691)"



So, what type are you ???

And just to balance things up and offer some constructive suggestions ..



So there you go .. there are scientists working on improving the system and thereby its quality goals. (As expected .. unless you're a pseudoscientist, I suppose).

:)

Cheers
PS: Not that there's anything wrong with being a pseudoscientist. :)

[snas]

Human nature being, well, human nature, you must wonder if anyone is truly capable of being a "correct" referee on every occasion. But if all referees could be "correct" every time, that would create an ideal peer review system.


Would a market driven system not lead to an increased rate of publication of "drivel", given that there would be no peer review system to filter out the rot?
How would a particular journal be able to claim itself as being more "prestigious" than the next if there is no peer review system to ensure the quality of published papers? It would seem to me (maybe I am wrong!:shrug:) that a system like this would have the potential to bring all journals more or less into line with each other. I guess you may have one journal only bidding for papers of very high quality while a second journal (let's call it the "Thunderbolts Journal") might bid for more spurious papers and that would create a quality difference.
In my own profession we suffer from a number of fools who perform veterinary homeopathy. They will happily quote, as proof of their ludicrous claims, publications in "letters to the editor" sections of veterinary journals, or in the "Dog Breeders' newsletter" etc. These quotes mean nothing to me because I am a rational veterinarian, but when the average pet owner sees the same thing, they can be misled into thinking that a few paragraphs on quackery in a breed society newsletter can actually be valid information.

Stuart

[renormalised]

A pseudoscientist rejects any sort of review, simply because it means their nonsense won't get the airing it so richly doesn't deserve. Pseudoscientist don't practice science, they practice quackery and as such should be ignored. That some actual happen across valuable scientific insights at times is just a random chance event, more a case of luck than good science.

The fact the guys who did this study use complex systems mathematics would immediately make them suspect in the eyes of the EU crowd. It'd be a conspiracy, according to them. The maths isn't important etc etc. This is precisely why peer review is there...to sort the wheat from the chaff, or in their case the fecal matter from the rest of the crop.

Now...peer review...The idea of a market place for work has some merit but it also has a lot of bad points as well. It could give scientists a means to get papers seen that normally aren't because of the variability of the peer review process between journals etc. But it could also make it almost impossible for young researchers to get a voice because they could get drowned out by their more experienced colleagues. Plus, as Stuart reiterated, you may also get a lot of garbage getting through that otherwise wouldn't.

[snas]

Oops
Craig, I forgot to add, what's wrong with being a pseudoscientist? Probably nothing. After all, it must be a very happy little place being right all the time, simply because you know that you must be right.:lol:

Stuart

[CraigS]

Stuart;
I think Mr. Pressure had something to say (at some stage in his quality contributions to us here), which started with something like:

'Do you want to be happier or wiser ?'

Pity we didn't get his next instalment on that one !

I gather he's right about whatever that one was about .... which must imply that he is pretty happy sort of guy !!
:)

Cheers
PS: We're all pseudoscientists, (at times), if we're human ! (I was being semi serious about my original "PS").

[renormalised]

Being happier, in his case (and in many other cases) means you haven't a clue about what's going on and when something ends up destroying your nice little self delusion, which it invariably does, the consequences can be rather disturbing. Nope, I'd much rather be wiser, knowing that whatever happiness I had was solely my own responsibility and not something I abrogated to some nebulous external influence or agency.

[CraigS]

Oops I was using the wrong terminology .. I meant "Pathological Scientist" not "Pseudoscientist".
Mr Pressure is a pseudo .. mere mortals can be pathological scientists, from time to time.

Apologies for that .. a slip-up.

Cheers

[renormalised]

Yeah, but quite often the pathological leads to the pseudo, especially when the "mere mortals" try to run with it, thinking they have the answers (or believing they do).

No one truly has the answers, it's just that some know enough to be able to figure out what some might be, or how to go about figuring them out.

[higginsdj]

In my daily work I'm in the 'Correct' category - thats my job - to peer review software specifications.

In Astronomy I don't peer review - I'm just not qualified (although I do comment where I feel appropriate)

Cheers

[CraigS]

So getting back to the paper ...



From the first post;
"rational": those who reject papers that might draw attention away from their own work;
"random" who are not qualified to judge the quality of a paper because of incompetence or lack of time.
and the types who do the best job (in the mix) are:
"correct" who accept the good papers and reject the bad and;
"altruists" who accept all papers and;
"misanthropists", who reject all papers.

So, maintaining the quality of the peer review process is critically dependent on keeping career-oriented, busy and incompetent types out of the process.

Seems logical ... (almost self-evident, really) but difficult to achieve in practice (?).
We need empirical evidence to take this one further !!

Cheers
PS: This could be good ammo for you David. Ie: That journal who published the work that's flawed ?

[renormalised]

I wouldn't even call the "altruists" or "misanthropists" good for peer review either. If you have any amount of them in the process, you might as well let it be a "free for all anything goes" system, or, "damn them all to hell no one will publish anything because I said so" system. Where's the good in that??!!!!.

[CraigS]

Hang on a sec ...some time ago (30th Aug) you said you were an "Orthodox"! :P:)



"Orthodox: dominated by the groupthink, following a leader’s opinion as in the “Naked king” tale, good workers performing monotonous tasks without ideas in large groups, specialists in a small field which they know very well, conformist, domestic. His/her dream is getting a permanent position at an university or research center, to be leader of a project, to do astropolitics. Most of them are like sheep (or geese), some of them with vocation of shepherds too."

.... Geez I'm mean ...!!
:(
I don't know what I am !! ...er... I'm a Hamster !! :confused2:
Depends on the situation !!

I'm only kidding .. don't take this, or me, too seriously !! ..I reckon you're doing a good job !

Cheers

[CraigS]

Ahh .. those damn cosmologists getting in on the act again !!
They all think alike !!
:):P
woof !!

Cheers

[renormalised]

Geez, I wonder how the EU crowd would categorise me, then:):P:P:P:P

[CraigS]

We already have that answer !!
:):P:P:P:P:P:P:P
woof !! woof !!

[renormalised]

He is right, though. If you either reject or accept someone's paper for publication, then you should be able to justify why you made those decisions which brought you to that point. If you can't, then you have no business being in the position of being a reviewer.

Not all science or research is necessarily going to offer an advance on current thinking. It can't, simply because that's not what a lot of research is about and not everyone is capable of bringing about an advance in current thinking. For younger researchers there is a pressure to "publish or perish", in that the more you can get published, the more you'll get noticed. Once you get to the stage of being well known, then you can back off a bit. But you still have that pressure to keep up the work. It's usually the younger ones who are expected to come up with all the new ideas.

[CraigS]

and the final gem is found right at the end of the article ....



Classic stuff !!
:)

Cheers

[renormalised]

Indomitable Ogre of IIS....Harrying Harpy of the High End Thinking.... Damnable Dogged Destroyer of Dangerous Drivel:):P:P:P:P

[renormalised]

Can you imagine her trying to get that past a peer review board...."Heresy!!!!!. Peer review is sacred!!!!", and then it was consigned to the waste paper basket:):P

[higginsdj]

When it comes to astronomy I do have my 'Kings'. I run everything past them :)

Cheers

[avandonk]

I have seen horrific examples of referees who also were responsible for grants, blocking both grant applications and or papers to then use the brilliant idea of the grant application in their own labs to beat the originators of the idea. This again comes out in the wash as there is a paper trail.

When applying for grants always leave out a very important bit so thieving mongrels cannot begin to even start to appropriate your ideas.

Bert