NASA CRS-7 report released

[envy887]

Summary of NASA report on CRS-7 was released today.

https://www.nasa.gov/sites/default/files/atoms/files/public_summary_nasa_irt_spacex_crs-7_final.pdf

--

Article on the above, by Michael Baylor:
https://www.nasaspaceflight.com/2018/03/nasas-irt-publishes-crs-7-failure/

Additional:

CRS-7 Failure thread:
https://forum.nasaspaceflight.com/index.php?topic=37739.0

CRS-7 News Articles:
https://www.nasaspaceflight.com/?s=CRS-7

[Craig_VG]

NASA concluded that it was not the manufacturer's fault for faulty strut, but instead SpaceX's fault for not following manufacturer guidelines.

Key Parts:

The use of an industrial grade 17-4 PH SS (precipitation-hardening stainless
steel) casting in a critical load path under cryogenic conditions and flight environments,
without additional part screening, and without regard to manufacturer recommendations for a
4:1 factor of safety, represents a design error – directly related to the F9-020 CRS-7 launch
failure as a “credible” cause

Lastly, the key technical finding by the
IRT with regard to this failure was that it was due to a design error: SpaceX chose to use an
industrial grade (as opposed to aerospace grade) 17-4 PH SS (precipitation-hardening stainless
steel) cast part (the “Rod End”) in a critical load path under cryogenic conditions and strenuous
flight environments.

[Coastal Ron]

NASA concluded that it was not the manufacturer's fault for faulty strut, but instead SpaceX's fault for not following manufacturer guidelines.

Key Parts:

The use of an industrial grade 17-4 PH SS (precipitation-hardening stainless
steel) casting in a critical load path under cryogenic conditions and flight environments,
without additional part screening, and without regard to manufacturer recommendations for a
4:1 factor of safety, represents a design error – directly related to the F9-020 CRS-7 launch
failure as a “credible” cause

Lastly, the key technical finding by the
IRT with regard to this failure was that it was due to a design error: SpaceX chose to use an
industrial grade (as opposed to aerospace grade) 17-4 PH SS (precipitation-hardening stainless
steel) cast part (the “Rod End”) in a critical load path under cryogenic conditions and strenuous
flight environments.

I think everyone should read the report, since selective quoting can sometimes be misleading. For instance, my own selective quoting:

SpaceX in their AIT report identifies “material defect” as the “most probable” cause for the rod end breaking. However, the IRT’s view is that while “rod end breakage due to material defect” is credible, the IRT does not denote it a “most probable” since the IRT also views “rod end manufacturing damage”, “rod end strut mis-installation”, “rod end collateral damage” or some other part of the axial strut breaking as equally credible causes to have liberated the COPV.

I'll also note that I didn't see any mention of SpaceX requiring certification from the supplier, and that the supplier had not been consistently doing the certification process (or doing it at all). And while it can be argued that "as designed" the parts were not of sufficient strength for the application they were used in, SpaceX knew this, which was why they required additional testing to weed out any non-compliant parts - and they paid for that service.

Maybe my perspective is colored by my manufacturing background, and that the processes SpaceX paid for would have produced acceptable parts.

Regardless, history has shown that SpaceX has addressed the problem, and I don't really find anything too damning in this report - though that may not stop SpaceX detractors from damning them anyways...

[Craig_VG]

Sorry if my quoting came off as pushing a particular viewpoint. That was not my intention.

I thought it best described the 'new info' released in this document.

[Nomadd]

I thought they had a 4 to 1 margin. And NASA did not conclude it wasn't the manufacturer's fault. They concluded that SpaceX should have used higher qualification standards that are there to guard against manufacturing faults. I don't really see anything in that report that contradicts what SpaceX came up with.

[S.Paulissen]

I thought they had a 4 to 1 margin. And NASA did not conclude it wasn't the manufacturer's fault. They concluded that SpaceX should have used higher qualification standards that are there to guard against manufacturing faults. I don't really see anything in that report that contradicts what SpaceX came up with.

YMMV with source.

"He said that the strut was designed to handle 10,000Ibs of force but failed at 2,000Ibs of force."

https://www.siliconrepublic.com/discovery/spacex-rocket-exploded-because-of-broken-strut-explains-ceo-elon-musk

[Coastal Ron]

Sorry if my quoting came off as pushing a particular viewpoint. That was not my intention.

I thought it best described the 'new info' released in this document.

No, I did not want to imply that. And of course any time anyone uses a partial amount of an original text it's "selective quoting" - as I noted my selective quote was.

My apologies for the confusion... 

[FutureSpaceTourist]

Couple of key quotes:

It is important to note that the IRT’s conclusions regarding the direct, and immediate causes are consistent with the determination made by the SpaceX AIT investigation findings. Where the IRT differs with SpaceX is in regards to the initiating cause. SpaceX in their AIT report identifies “material defect” as the “most probable” cause for the rod end breaking. However, the IRT’s view is that while “rod end breakage due to material defect” is credible, the IRT does not denote it a “most probable” since the IRT also views “rod end manufacturing damage”, “rod end strut mis-installation”, “rod end collateral damage” or some other part of the axial strut breaking as equally credible causes to have liberated the COPV. Lastly, the key technical finding by the IRT with regard to this failure was that it was due to a design error: SpaceX chose to use an industrial grade (as opposed to aerospace grade) 17-4 PH SS (precipitation-hardening stainless steel) cast part (the “Rod End”) in a critical load path under cryogenic conditions and strenuous flight environments. The implementation was done without adequate screening or testing of the industrial grade part, without regard to the manufacturer’s recommendations for a 4:1 factor of safety when using their industrial grade part in an application, and without proper modeling or adequate load testing of the part under predicted flight conditions. This design error is directly related to the Falcon 9 CRS-7 launch failure as a “credible” cause.

In summary, the IRT determined that subject to the normal technical review of SpaceX’s corrective actions implementation, including correction of their design error, the F9-020 CRS-7 flight anomaly is resolved with “credible”, direct, intermediate, and initiating causes. That the initiating causes include more than just rod end “material defect”, and the initiating causes are rated by the IRT as “probable”.

P.S. I’ve put this in SpaceX General as I think discussion of the summary report’s contents naturally goes rather beyond the one mission.

[woods170]

Just about the only new thing from this report is that the stage 2 LOX tank failed due to the liberated COPV shooting upwards and colliding with - and thus damaging - the upper dome of the LOX tank.

Everything else has already been reported, and discussed, ad nauseam.

[CorvusCorax]

I think the main lesson from this report to be taken is - 4 eyes see more than 2.

It was generally in the best interest of both NASA AND SpaceX, that an independent review was undertaken in close cooperation with SpaceX to verify the internal investigation, procedures and to investigate alternate causes.

Most importantly the document concludes with:

*The IRT notes that all credible causes and technical findings identified by the IRT were corrected and/or mitigated by SpaceX and LSP
for the Falcon 9 Jason-3 mission.  That flight, known as “F9-19”, was the last flight of the Falcon 9 version 1.1 launch vehicle, and flew successfully on 17 January 2016.

The review also looked into some issues other than the causes of the mishap, such as the mentioned low nitrogen purge rates or telemetry protocols - all of which led to vehicle improvements.

Competitors, and political speakers opposed to SpaceX might want to use this report as propaganda against SpaceX and their methods.
I think likely everyone at SpaceX knows better. They will be thankful to NASA for both vindicating and improving their own procedures as well as the vehicle.

The public sometimes sees investigations like this as "looking who is to blame" -- but this is a technical investigation. The do not search for a scapegoat, but for things that can be improved for the benefit of everyone.  Same as with NTSB investigations after aircrashes and the like. They are not to find some poor fool to be liable, but to improve the design and procedures, learn, and prevent bad things from happening again or ever in the future.

As such, the more issues such an investigation can identify for improvement - whether related to the mishap or not - is a good thing (tm)

[Chris Bergin]

Article on the above, by Michael Baylor:
https://www.nasaspaceflight.com/2018/03/nasas-irt-publishes-crs-7-failure/

[sanman]

Another article:

https://qz.com/1227488/elon-musk-spacex-nasa-says-a-design-error-is-behind-the-failure-of-the-2015-crs-7-mission/

[Michael Baylor]

I thought they had a 4 to 1 margin. And NASA did not conclude it wasn't the manufacturer's fault. They concluded that SpaceX should have used higher qualification standards that are there to guard against manufacturing faults. I don't really see anything in that report that contradicts what SpaceX came up with.

They did not abide by the 4 to 1 margin.

Quote from the IRT report.

The implementation was done without adequate screening or testing of the
industrial grade part, without regard to the manufacturer’s recommendations for a 4:1 factor of
safety when using their industrial grade part in an application, and without proper modeling or
adequate load testing of the part under predicted flight conditions.

[jg]

TF-4 (technical finding 4)  is interesting: it shows SpaceX's telemetry system is suffering from bufferbloat. Bufferbloat made their analysis much harder (since they lost a lot of their telemetry data as a result).  As gamers say: "Lag kills".

I'd love to be able to help Spacex out with this problem.  Note that I coined the word "bufferbloat" 6 or so years ago; we
didn't have a succinct term for the problem of excess buffering before that, and have been helping the Internet community face up to the problem.  If you know a good person I could contact who would pay attention, PM me.

BTW: most people suffer from bufferbloat routinely in their home networks.  See my blog for more information:

https://gettys.wordpress.com

There are beginning to be commercial products that dramatically help the problem for your home network.

But solutions depend on the circumstances; there is no magic wand here for SpaceX.

[envy887]

TF-4 (technical finding 4)  is interesting: it shows SpaceX's telemetry system is suffering from bufferbloat. Bufferbloat made their analysis much harder (since they lost a lot of their telemetry data as a result).  As gamers say: "Lag kills".

I'd love to be able to help Spacex out with this problem.  Note that I coined the word "bufferbloat" 6 or so years ago; we
didn't have a succinct term for the problem of excess buffering before that, and have been helping the Internet community face up to the problem.  If you know a good person I could contact who would pay attention, PM me.

BTW: most people suffer from bufferbloat routinely in their home networks.  See my blog for more information:

https://gettys.wordpress.com

There are beginning to be commercial products that dramatically help the problem for your home network.

But solutions depend on the circumstances; there is no magic wand here for SpaceX.

I'd be wary of assuming anything in the report is still current. 2015 was a LONG time ago in the SpaceX dev cadence. SpaceX has flow a new major version and 4 new sub-versions of Falcon 9 since then, with a 5th sub-version to fly within a month or so.

The last page notes that all the technical findings were already resolved or mitigated before Jason-3 flew in 2015.

[jg]

TF-4 (technical finding 4)  is interesting: it shows SpaceX's telemetry system is suffering from bufferbloat. Bufferbloat made their analysis much harder (since they lost a lot of their telemetry data as a result).  As gamers say: "Lag kills".

I'd love to be able to help Spacex out with this problem.  Note that I coined the word "bufferbloat" 6 or so years ago; we
didn't have a succinct term for the problem of excess buffering before that, and have been helping the Internet community face up to the problem.  If you know a good person I could contact who would pay attention, PM me.

BTW: most people suffer from bufferbloat routinely in their home networks.  See my blog for more information:

https://gettys.wordpress.com

There are beginning to be commercial products that dramatically help the problem for your home network.

But solutions depend on the circumstances; there is no magic wand here for SpaceX.

I'd be wary of assuming anything in the report is still current. 2015 was a LONG time ago in the SpaceX dev cadence. SpaceX has flow a new major version and 4 new sub-versions of Falcon 9 since then, with a 5th sub-version to fly within a month or so.

The last page notes that all the technical findings were already resolved or mitigated before Jason-3 flew in 2015.

I hope so.  But my experience is that bufferbloat is not well understood, and it rears its ugly head again and again. It's nature is you fix one place, and find it rears its ugly head elsewhere all too soon.

It's a mistake we've made all over the industry.  We didn't even have a decent word for bufferbloat when I started. Most engineers don't think in the right terms.  And as to general solutions, while we had fixes for ethernet pretty quickly, other technologies are far lagging.  We didn't have *any* running code for WiFi until 18 months ago, and there is at best
one commercial product on the market at this date that has that.

Jim

[Andy Bandy]

Less technical but more details on why it took this long a summary to be made public. NASA said it would, then it said it wouldn't, and it seems to have taken an effort by Congress to get the full story out there. There are some links in the post to more detailed stories on the whole thing.

http://www.parabolicarc.com/2018/03/13/nasa-falcon-9-failure-2015-caused-design-error/

[Lar]

I thought they had a 4 to 1 margin. And NASA did not conclude it wasn't the manufacturer's fault. They concluded that SpaceX should have used higher qualification standards that are there to guard against manufacturing faults. I don't really see anything in that report that contradicts what SpaceX came up with.

They did not abide by the 4 to 1 margin.

Quote from the IRT report.

The implementation was done without adequate screening or testing of the
industrial grade part, without regard to the manufacturer’s recommendations for a 4:1 factor of
safety when using their industrial grade part in an application, and without proper modeling or
adequate load testing of the part under predicted flight conditions.

I thought a key SpaceX finding from their own internal investigation contradicted the first part of this... that SpaceX had contracted for, and was relying on, screening/testing by the part vendor, had reason to beleive that was "adequate" but determined that they were misled and added an additional layer of testing after part receipt.

[kaiser]

Interesting to see the note about the telemetry latency.  I'm finding a lot of newer systems aggregating and streaming in lots of ways that result in lots of extra latency, and how that latency can ripple through so many other aspects of the system. 

Good to see NASA point out that low latency transmission is key to having as much data as possible for FRB, and that SpaceX implemented it. Kudos to them.

[LouScheffer]

Quote from the IRT report.

The implementation was done without adequate screening or testing of the industrial grade part, without regard to the manufacturer’s recommendations for a 4:1 factor of safety when using their industrial grade part in an application, and without proper modeling or adequate load testing of the part under predicted flight conditions.

This, in a nutshell, is a large part of what makes designing and operating rockets and airplanes so expensive.  For commercial stuff like building and bridges, you over-design by a factor or 4 or 6.   This margin helps in two ways - you can use very simple (often visual) inspection, and you are protected against errors in calculating loads.   For most civil construction, you could have a hidden flaw that makes each part only half as strong as it should be, AND a calculation error that makes the loads twice what you calculated, and your building will still stand.  Of course the building is 6 times as heavy as it really needs to be, but that's OK.

But if mass concerns drive you to reduce your margins to 50%, now you need both fancy and detailed inspections, to insure no hidden flaws, and much more accurate load calculations, to make sure you really found the worst case and calculated it correctly.  From the quote it appears SpaceX made errors in both areas.

This line of reasoning leads some to believe that space stuff could be much cheaper if we had cheaper transport, since then you could use the higher mass margins to allow for cheaper design and construction.   So far this has not happened, perhaps since the needed margins are so big, or because other considerations (such as the difficulty of repair once in operation) also require extreme design care and inspections in any event.