NASA - James Webb Space Telescope - Discussion Thread 2

[LouScheffer]

Before I reply, I'd like to express my appreciation for your thoughtful and rational responses based on real experience.  I think we largely agree on the state of the current system, just not on the path going forward.

I think science was actually advanced in this case.

We're talking about the impact on scientists, not just "science". It makes little difference in the grand scheme whether a paper comes out this month or in 3 months. It does however make a big difference to someone's career if they get a paper in a big journal with global media attention. The original team missed out on that. Note that without the original team this result would have been delayed much further, because they are the ones who got the data taken in the first place.

In just one paragraph, you've mentioned three big and interlocking problems:
(a) Speed of dissemination (I'll talk about that later)
(b) The need to publish in major journals.  This is a huge problem.  Since these publications are so coveted, the journals get an enourmous number of submissions.  A considerable majority are tossed by the editor without even sending out for peer review, so your entire career depends on the reactions of a single person to a quick reading of your article.  Which ones even make it to peer review is (in my experience) a total crapshoot.  This is a horrible way to run a gate-keeping system.
(c) The problem of getting credit for data generation.  This too is broken, related to (b).  Despite the fact that collecting the data is often by far the largest task of solving the problem, top journals show no such respect and require a "science story".   This leads precisely to data sets (which could be used for many purposes) being held back until the one particular science question that concerns the main researcher is answerable.

All these people were scooped, someone published their project before them. That's what scooped means, this is a no-true scotsman fallacy.

This statement is only true if you restrict "published" to mean "published in a journal".  If you include pre-prints, they were not scooped.  This is not a hypothetical distinction - in some fields, such as physics, an arxiv publication is perfectly good for claiming scientific priority.

If the manuscript was less developed then there would be no response and we simply wouldn't have heard about it. It is a survivorship bias.

This is exactly why I am asking for real examples.  There are two possibilities:  (a) It happens all the time, we just don't hear about it, and (b) it doesn't really happen often, if at all.  I think it's largely (b), and the way to disprove my hypothesis is to show real examples.

And when things are rushed mistakes happen. If you want examples of rushed analyses which turned out false then we have BICEP2 and some of the high redshift pre-flight calibration JWST papers.

These are good examples of haste makes garbage.  But they were invalidated quite quickly, and the authors certainly did not gain reputation in the longer run - quite the opposite.

It's not at all obvious that peer review contributes more to science than it hurts by delaying distribution of knowledge. 

Peer review absolutely contributes to science by screening obvious errors. You're putting so much emphasis on speed of publication but none on quality control, which is absolutely the wrong view in my opinion.  [...] If you want to see what science looks like without peer review take a look on viXra. Peer review isn't perfect but it's the only filter against absolute nonsense.

I certainly agree there is a tradeoff between speed and accuracy.  But requiring peer review before publication, in my mind, is way past the point of diminishing returns - other points on the speed-accuracy curve are better overall.  And peer review is certainly not the only filter against nonsense.  Arxiv, for example, does not have any peer review, but the content-to-garbage ratio is reasonable and the speed much faster than peer review.  xiVra, on the other hand, is indeed a dumpster fire, but that's not since it lacks peer review, it's because it does no gatekeeping whatsoever.  Arxiv instead (at least the last time I checked) requires a one-time certification by someone in the field  that you are a serious researcher.  That, plus researcher's own incentive not to publish trash, seems to be enough.

[ttle2]

(b) The need to publish in major journals.  This is a huge problem.  Since these publications are so coveted, the journals get an enourmous number of submissions.  A considerable majority are tossed by the editor without even sending out for peer review, so your entire career depends on the reactions of a single person to a quick reading of your article.  Which ones even make it to peer review is (in my experience) a total crapshoot.  This is a horrible way to run a gate-keeping system.

This isn't really true in astrophysics, unless one tries to go for a publication in Nature or Science (and if one does, one can really only blame oneself... ) Major field journals are not nearly as selective and desk rejections (or even outright rejections by referees) are not common.

(c) The problem of getting credit for data generation.  This too is broken, related to (b).  Despite the fact that collecting the data is often by far the largest task of solving the problem, top journals show no such respect and require a "science story".   This leads precisely to data sets (which could be used for many purposes) being held back until the one particular science question that concerns the main researcher is answerable.

I agree that this is a problem. This could be helped at least partly if people using date collected by some program cited the proposal and the citations were counted by ADS (for example), so they would help the proposer(s) at least somehow. I'm not sure if this already happens with some journals.

This is exactly why I am asking for real examples.  There are two possibilities:  (a) It happens all the time, we just don't hear about it, and (b) it doesn't really happen often, if at all.  I think it's largely (b), and the way to disprove my hypothesis is to show real examples.

As said before, most people don't want to accuse someone of unethical scooping in public. A public argument is almost always going to hurt your career even more than getting scooped. If you choose to believe that it doesn't happen, there's not much I can do to convince you.

These are good examples of haste makes garbage.  But they were invalidated quite quickly, and the authors certainly did not gain reputation in the longer run - quite the opposite.

Because they were very high-profile cases. They are not representative of the way things happen normally.

[Dizzy_RHESSI]

This statement is only true if you restrict "published" to mean "published in a journal".  If you include pre-prints, they were not scooped.  This is not a hypothetical distinction - in some fields, such as physics, an arxiv publication is perfectly good for claiming scientific priority.

I do not. The sunburst arc was scooped by a pre-print. In the case of the K2-18b the original team did ultimately go public sooner, but only because they were tipped off that there was going to be a press release the next day. But putting your paper on the arXiv is not really on par with a Nature Astronomy paper and international press coverage.

These are good examples of haste makes garbage.  But they were invalidated quite quickly, and the authors certainly did not gain reputation in the longer run - quite the opposite.

You previously brought up the effects on science as a whole. The fact that there are documentaries and books about the BICEP2 fiasco shows what a toxic legacy it has had. The public did not forget and it is a stain on the reputation of science. This never would have occurred if they simply took their time and waited for the Planck data to be published, or made more careful claims. But they were worried about getting scooped. And you talk about speed, but you don't consider the months researchers wasted testing the BICEP2 results, correlating them with other data and theorizing on the implications for inflation.

I certainly agree there is a tradeoff between speed and accuracy.  But requiring peer review before publication, in my mind, is way past the point of diminishing returns - other points on the speed-accuracy curve are better overall.  And peer review is certainly not the only filter against nonsense.  Arxiv, for example, does not have any peer review, but the content-to-garbage ratio is reasonable and the speed much faster than peer review.  xiVra, on the other hand, is indeed a dumpster fire, but that's not since it lacks peer review, it's because it does no gatekeeping whatsoever.  Arxiv instead (at least the last time I checked) requires a one-time certification by someone in the field  that you are a serious researcher.  That, plus researcher's own incentive not to publish trash, seems to be enough.

The arXiv basically relies on the fact that most of the papers are submitted to be peer reviewed. They have in the past removed research notes published in RNAAS because they are not peer reviewed, their current policy is vague. The arXiv does manual moderation, with subjective criteria and it's not without the occasional controversy. It's not a replacement for peer review, if you watch it long enough you will see nonsense which leaks through. Peer review is much more detailed and transparent than the arXiv moderation. A researchers incentive to publish good work is because their output is largely measured in peer reviewed publications. But I think this is getting rather off topic, NASA's decisions about proprietary time isn't going to reform publishing. Astronomers are at the mercy of academia as it is today, not how we might want it to be.

[edzieba]

IIRC, wasn't the BICEP2 'scooping' based on data scraped from slides presented by the original team prior to publication? i.e. the data the original team gathered WAS embargoed, but an embargo does no good if you go and disseminate it anyway.

[LouScheffer]

But I think this is getting rather off topic,

Agreed this discussion on scientific publishing is off topic here - it's not only not Webb specific, it's not even space or astronomy specific.

NASA's decisions about proprietary time isn't going to reform publishing. Astronomers are at the mercy of academia as it is today, not how we might want it to be.

Agree this one decision alone won't reform publishing.  But NASA is not completely helpless, either.   As a big player, can help to push in the right direction by the example it sets. 

[leovinus]

At least some high redshift galaxies are getting confirmed now.

Discovery and properties of the earliest galaxies with confirmed distances

https://arxiv.org/abs/2212.04480

[Star One]

At least some high redshift galaxies are getting confirmed now.

Discovery and properties of the earliest galaxies with confirmed distances

https://arxiv.org/abs/2212.04480

NASA press release:

https://blogs.nasa.gov/webb/2022/12/09/nasas-webb-reaches-new-milestone-in-quest-for-distant-galaxies/

[ttle2]

The arXiv basically relies on the fact that most of the papers are submitted to be peer reviewed. They have in the past removed research notes published in RNAAS because they are not peer reviewed, their current policy is vague.

Extremely off topic: the ban didn't have anything to do with RNAAS not being peer reviewed; there are a lot of papers on arxiv that are not (and are not intended to be) peer reviewed. The reason was that they were afraid of being flooded by lots of very short papers so they enforced some minimum length limit (though I think shorter papers could still be accepted in special cases). I'm not sure if the policy changed.

[Dizzy_RHESSI]

IIRC, wasn't the BICEP2 'scooping' based on data scraped from slides presented by the original team prior to publication? i.e. the data the original team gathered WAS embargoed, but an embargo does no good if you go and disseminate it anyway.

That is called scraping. And no, that was a different aspect. One of the reasons they pushed to publish quickly was that they were worried that Planck could publish the detection of primordial B modes before them. Planck wasn't all that sensitive to such features, but if it was as strong as the BICEP2 team believed they could have detected it.

Extremely off topic: the ban didn't have anything to do with RNAAS not being peer reviewed; there are a lot of papers on arxiv that are not (and are not intended to be) peer reviewed. The reason was that they were afraid of being flooded by lots of very short papers so they enforced some minimum length limit (though I think shorter papers could still be accepted in special cases). I'm not sure if the policy changed.

The policy is rather vague. The blog where the policy was changed specifically refereed to "short, unrefereed submissions".

[deadman1204]

can we have a different thread for people who want to change publishing policy? Its not even specific to jwst

[Star One]

The Science Hour - JWST the first six months

NASA's James Webb Space Telescope has produced amazing images in its first 5 months, but amazing science as well. Roland hears from one of the leading astronomers on the JWST programme, Dr Heidi Hammel, as well as other experts on what they are already learning about the first galaxies in the Universe, the birth places of stars, the strange behaviour of some other stars, and the first view of Neptune's rings in over 30 years.

https://www.bbc.co.uk/sounds/play/w3ct3b09

[leovinus]

They are also going to try to image the zodiacal light. Question: Are trace CFC gases in the zodiacal light something they could register with the spectrometer at this distance? There are obvious world-shattering implications if they see any...

How would CFC's in the zodiacal light be "world shattering". The zodiacal light is reflected light from sol inside the solar system.

Observing CFCs in the atmospheres of exoplanets is considered by astronomers as compelling evidence of a technosignature as they are only produced in significant quantities on Earth by industrial processes. So, reasoning by analogy ...

Zodiacal light is not from the atmosphere of a planet.  It is light reflected from the dust floating in space around a star.  In our solar system the figure I get for the dust that reflects zodiacal light is the particles have a total volume equivalent to a 30 km sphere broken down into dust, that is a reflective surface many orders of magnitude greater than Earth's surface.  CFCs are an incredibly small percent of our atmosphere.  I highly doubt you would be able to find the signature of CFCs in the atmosphere of a planet within that dust cloud within all that light noise from zodiacal light.

CFCs in our atmosphere can be monitored by satellites.  The Aura satellite uses a microwave limb sounder to build a vertical profile of the gasses in our atmosphere.  The microwaves are emitted by the gasses and not part of reflected sunlight.  But to do it at stellar distances, is more than likely going to be beyond our technical capabilities for a long time if ever.  So looking for CFCs from a planet buried inside the cloud of dust reflecting sunlight isn't likely to find CFCs anytime soon.

I obviously put it obscurely, so trying to be clearer: The (probably tongue-in-cheek) original suggestion seemed to be that there would be world-shattering implications if they see traces of CFCs in the zodiacal light generally - i.e. from the circum-stellar dust; not specifically from any exoplanets - as it would be taken as evidence of widespread in-space industrial activity in the stellar system.

Moving the discussion to the Discussion thread. While we might only know "industrial processes" for CFC, that does not mean there cannot be a natural process as well. Remember the discussion on Phosphene in the Venus atmosphere, and even Methane on Mars. Keep an open mind.

[Star One]

Cool Worlds - Can JWST Detect Biosignatures on Exoplanets?