Monday, 8 July 2013

The PNAS hotel

The Rat, 2013



Hi rats,

This post is about high impact factor journals, i.e., those publications where you have to convince an entity called “editor” that your article is “cool” and will “appeal to a wide audience” before having a chance to get it reviewed by someone who's not an ignorant. Most of you won’t pass this first filter. That is, unless an individual whom I've never seen in any nonlocality workshop claims in Nature News that your result is “the most important general theorem relating to the foundations of quantum mechanics since Bell’s theorem”

Hence I will focus my rant on this aspect, the editor stage, because I find astounding that editors of professional scientific journals regularly reject submissions on completely unscientific grounds. Whenever I get a good original result and am able to summarize it in four pages, I know that I will probably get it published in PRL. I wished I could say that whenever I get an exceptional result it will most likely be published in Science, Nature or Nature Physics. If that were the case, I wouldn’t be writing this post.

What’s wrong with high impact factor journals? Is it true that under their human masks Nature editors are a reptilian species sent to destroy Earthen civilization by featuring articles like this? Desperate for answers, I went to the Bristol University Mensa and asked researchers on this matter. I found that their opinions fall into three categories:

a) The Conformist: “Yes, sure, the system is far from perfect, but that’s all there is. So stop whining and send those families to the reeducation cam- sorry, wrong forum.”

b) The Outraged Conformist: “Those journals are a shame and we should boycott them all. But first let me submit my new paper “If quantum mechanics were more nonlocal, an avian compass would violate the second law of thermodynamics in the Canary Islands””.

c) The Understanding: “You have to put yourself in their shoes… these editors receive a lot of papers every week, they have to come up with a system to release that load. And the simplest way is to reject all papers written by junior scientists. Now I have to leave you, that man with the scars over there says that he wants my “fucking” shirt. Poor soul, he must have suffered so much. I will offer him my money, my house and my first-born baby”.

Between the lines, one can read some skepticism about the transparency of editorial decisions in high impact factor journals. Let us dig into this:

In his book “Reason strangled”, chemist and journalist Carlos Elías argues that Nature’s top-one priority is to maintain its impact factor; that allows the journal to set the prize of the publication. Problem is, Nature’s impact factor is already so high that a lot of effort must go just into not letting it drop. Hence Nature editors have to make sure that each article will be highly cited. Articles signed by famous scientists are read more, and, consequently, have more citations. Likewise with prestigious affiliations, fancy titles, articles already mentioned in the press. Elías writes about Nature, but I guess that his conclusions apply to other high impact factor journals as well.

It follows that the paper “Full algorithmic characterization of LOCC quantum operations”, by John Unknown, from Mac&Cheese Community College, will have a cold reception. On the other hand, the article “Quantum mechanics is extra-spicy”, signed by Stephen Hawking, Edward Witten and David Bowie, and featured on TV by Beakman and the Myth Busters, will make the same Nature editors twist and shout. Don’t forget that we’re speaking of a journal that in 1996 published an article about the analgesic effects of myrrh. Because it was a relevant result in the field? No! Because it was Christmas time!

The need to increase the journal’s impact factor explains many things, but does not answer all my questions. How come that exactly the same document is called “Supplementary Material” in Science, “Supplementary Information” in Nature, “Supplemental Information” in PNAS and “Supplemental Material” in PRL*? Do they want to drive us crazy? Why can’t PNAS editors read a reference where the author’s initials are - sacrilege! - before the family name? Are they aware of the amount of time that it takes to submit a paper to this journal, only to see it rejected the next week? If I conduct the research, write a referee list and prepare the paper in their damned unique format, what do the so-called editors do? I mean, besides checking my affiliation and h-index and replying “we receive more papers than we can publish, so we have to select those that will be of the greatest interest to a wide audience”. Do they understand how that sentence feels like when the next day they publish whatever crap with the words “spooky” and “quantum” in the abstract? Come to think of it, why do high impact factor journals have an impact factor at all? Shouldn’t they be in the same lot as other popular science magazines, like Scientific American, New Scientist and Physical Review A**?

*I know, PRL is not a high impact factor journal. But it’s where most good Physics papers end up after being rejected by the first three, they should make the transition easier!

**OK, here I went too far.

Epilogue

When I was a young postdoc, within two months, two different groups proposed two physical principles to limit quantum nonlocality. One group included important figures in QI who had previously published in Nature. Even unpublished, their work was soon echoed in the press (what? You never heard of information causality? How long did you say you stayed in that coma?). The other group was composed by a relatively unknown postdoc (me) and a second-year PhD student.

My collaborator and I knew that nothing short of building a time machine would have allowed us to pass the editors of a high impact factor journal. However, when we heard that the first group had managed to get their paper accepted in Nature, we thought: “now we have them by the balls”.

On one hand, Nature editors could not claim that our work wasn’t of general interest, because the same considerations would apply to information causality. On the other hand, they couldn’t argue that our work didn’t represent a significant advance, because it advanced the field as much as information causality did. In sum, the two main arguments for rejection in Nature didn’t apply. Cowabunga!

Of course, we were assuming that Nature editors can feel human emotions, like shame. Our article never went to referees. Guess why? Because “it was not of general interest and did not represent a significant advance”. A prior submission to Nature Physics had had exactly the same response.

Outraged, I wrote to Nature’s Editor-in-Chief, explaining my case*. My letter ended with some recommendations for the journal:

[…]

I therefore suggest you to change the contents of the Nature webpage concerning how to get published. It will not be so glamorous, but at least it will be honest. It could start by:

1. It is completely admissible to exaggerate one’s work to the point that no future research can compete with your so-claimed results.

2. If you are not a key figure in your field, stop reading. We are currently working on a webpage that can only be accessed by scientific celebrities, but, meanwhile, we would appreciate your cooperation if… wait a moment! Why am I wasting time talking to a Nobody? Leave! Now!

3. Oh, it’s you again! Didn’t I tell you to come back when you are famous? Go away, your anonymity smell is making me dizzy… what? How can you become famous? Err… I don’t know… By publishing in Nature?

4. Even if your work is not good enough for Nature, you can still try with Nature Physics. There, an editor with a PhD in experimental ultraviolet LEDs will review your theoretical paper on Foundational Physics and copy-paste his opinion about it. In order to guarantee a polite response, we have removed the exclamation sign from his keyboard.

[…]

And so on. That way, people will not get the wrong impression that Nature is a scientific journal, but the nerdy version of “Hello!”.

I think I have already written enough, and I do not want to waste more of your precious time; you must be very busy eating young researchers. But do not worry, this is not a “see you soon”, it is more a “see you never”.

A rat had been born.

Yours truly,

The Rat





16 comments:

  1. Everybody knows Science & Nature are scientific pornography, more the nerdy "Playboy" than the nerdy "Hallo!"... But you put it in a funny way, congrats. :)

    ReplyDelete
    Replies
    1. This comment has been removed by the author.

      Delete
  2. Miguel,

    I enjoy your rants, I really do, but this was a low blow:

    "That is, unless two individuals who haven’t read a paper on Foundations since the advent of pilot-wave theory claim in Nature News that your theorem is “the most important result in the foundations of quantum mechanics”."

    First of all, I can confirm that Valentini has read papers on Foundations since the advent of pilot-wave theory. He just doesn't like any of them. You and I may not share his opinions, but his views are not obviously bunk. Secondly, he only claimed that it was the most important result since Bell's theorem, not in foundations as a whole. There is an argument to be made that this is indeed the case, and I would be willing to have that argument with you at some point. It is not as obviously a moronic statement as some of the other stuff you criticize on this blog.

    The second individual you are referring to is I guess David Wallace, although if you read his papers it is fairly obvious that he has read a lot of modern stuff about quantum foundations, whether or not you agree with his opinions about it. He only claimed that it was the most important result he had seen in his 15 year career, which is even weaker than Valentini's claim.

    I was interviewed for this article and I said it was the most important result in foundations in the last five years. I was worried that I might get ridiculed for being overly hyperbolic, but I guess there is some truth in the idea that you cannot be too hyperbolic if you want to get quoted. Still, I'd much rather be accurate than be quoted.

    Secondly, and more importantly, your reference to this result is mistaken because the news article actually hurt this article's chances of appearing in Nature rather than improving them. You can read the full sordid story at http://blogs.discovermagazine.com/cosmicvariance/2012/08/27/guest-post-terry-rudolph-on-nature-versus-nurture/ In fact, if anything, this is yet another tale of the sort of editorial arbitrariness that you are criticising in your blog post.

    Finally, I just wanted to point out that there is a strict editorial wall between the "news" section of Nature and the "scientific articles" section, so when you criticize things that appear in the news section you are really criticizing an entirely different magazine from the one that you submit articles to. There is an argument to be had about whether this editorial wall is really a good idea, especially when it seems that whether the news section reports on a preprint can affect the chances of the scientific article itself appearing in Nature. Nevertheless, you can't criticize the scientific editors at Nature on the basis of things that appear in the news section. Both sections have their problems, but you would do better to focus on the editors that are the actual target of your rant.

    ReplyDelete
    Replies
    1. Hi Matt,

      I agree with some of your points, and have modified the offending paragraph accordingly.

      I am still pissed off by Valentini's statement, but not because I do not share his view. When you make claims like: "this is the best work in foundations since Bell's theorem", you should be absolutely sure that you have read all foundational work published since 1964. Otherwise, you may better invoke Wittgenstein's last proposition ("whereof one cannot speak, thereof one must be silent") and shut up.

      That said, like you, I enjoyed a lot the PBR paper, and I find the reasons argued by the Nature editor to justify its rejection inadmissible. From Terry's account of the events, though, it is clear that the rejection had nothing to do with Nature News, but with the provocative title of their second article. If anything, the publicity generated by Nature News must have helped the PBR paper to get published in Nat. Phys.

      Delete
    2. Hi Rat,

      could you please more closely explain "I enjoyed a lot the PBR paper..." - what did you find so fascinating about the paper?

      Delete
  3. As usual, bitingly sarcastic and enjoyable (I especially liked "Michal Navackowski" ;)). However, what do you *realistically* propose, assuming that a total boycott of such journals by respectable authors is not feasible?

    ReplyDelete
    Replies
    1. You can start by signing the San Francisco Declaration on Research Assessment (DORA). The purpose of DORA is that research output is evaluated by its content, not by the journal where it gets published.

      The moment funding agencies start adopting this policy, that will be the end of high impact factor journals.

      Delete
  4. I think that you are being too kind to PRL with "... it’s where most good Physics papers end up
    after being rejected by the first three". Accepted PRLs are often junk, and rejected ones can often enough be good.
    (e.g. elders of the field sometimes talk about the difficulty of getting foundational work into the prestige journals
    (including PRL) in the early years). So the problem of unfair and disgraceful editorial decisions is neither new nor
    limited to Nature or Science. It is sad that we still put up with the system however, and even help perpetuate it.

    It would be great to have a mechanism to significantly penalize journals for rejecting good papers (can we give them a "negative
    impact factor" for great papers they rejected 20 years ago, and have turned out to be gems?), but something like that
    would be hard to implement.

    ReplyDelete
    Replies
    1. I agree: many papers published in PRL are of very bad quality, and PRL sometimes rejects very good stuff. However, this rarely happens at the editor stage.

      In my experience, if you truly believe that your paper is good and original, the chances that it gets rejected in PRL are low. Of course, you may bump into a referee that does not understand your point, but that's what replies and appeals are for.

      >It would be great to have a mechanism to significantly
      >penalize journals for rejecting good papers

      I think it would be enough to make the correspondence with the editor and referees public. If a rejected paper turned out to be a milestone in the field, it would be hilarious to read the reasons for rejection.

      Delete
    2. Ah, I remember the "No Cloning Theorem"....

      Delete
  5. I think you could do something like the negative impact factor on a voluntary basis. I'm shooting from the hip here but... Set up a website and let people "register" their published papers on it in a data base. As part of the registration process they say what journals the paper was rejected from and provide the reviews/editorial letters if they want. If the paper has been published in a peer-reviewed journal they should be required to give the paper's coordinates. With appropriate data mining you could query the data base and ask it all sorts of interesting questions I think. I think it's a great idea. Someone ought to do it!

    ReplyDelete
  6. Cool post! this time I agree on almost anything!

    But I must admit I'd still like to have a paper in nature or science or whatever. I guess it would still make me feel kind of cool despite the above discussion, and most importantly I have the impression it can be useful for my future academic career (if I'll have one).

    I think at the moment I am an outraged conformist, but I have the impression most of us are at the moment no?

    ReplyDelete
  7. Not citing the science !hola! magazines articles gives me an equally good feeling as not buying from Nestle. And I will continue as long as I can.....

    ReplyDelete
  8. Thanks so much Miguel for this blog. I always laugh like crazy with each new entry. The not funny part is that, even with my very limited experience in physics, I have personally experienced most of the problems that you humorously denounced. I think this is a tragedy. The meager resources that are put into science are not used efficiently. As you and some of the authors of the comments seem to think, the whole "high impact factor journal" measure of how good an article is may be at the core of it all. And with a bad measure of article quality comes then a bad measure of researchers quality, and the good positions in physics do not go to the best candidates... and so on.

    Let us say that we were able to design a different evaluation system. How would you do it ?. Here is my two cents.

    In something like the arxiv (for instance, in the arxiv), registered users are able to comment on your article publicly and the comments are signed with real names. You can then respond to the comment. Maybe at the end of the interchange, the initiator of the comment could somehow indicate that he/she is satisfied with the exchange (or changes in the article). All is public. Everyone can see all comments, replies, pending comments, comments on comments ... or whatever. Some metric can then be computed, taking also the number of citations into account.

    Physicists do all the important work: Research and Peer review. I am not sure that anything else is needed.

    ReplyDelete
    Replies
    1. Good point! Also I was thinking, since as you say all the work is done by us (writing the paper and refereeing the paper), why are the editors so goddamn busy that they have to resort to automated computer programs to reply to our mails (assuming what the rat says is true)? Am I being naive? What goes on all day in the offices of scientific journals?
      I admit my lack of knowledge of the real world (and of physics for that matter), so if any publishing person wants to explain more please do!

      Delete
  9. Could someone explain what is bad about only publishing in more specialized journals? When I think of physics, Elliott Lieb comes to mind and as far as I can see from his publication lists he does not have any publications in nature. So does that mean that Lieb is a bad physicist, ... that does not sound right...

    ReplyDelete