FAILURE: New Shepard - NS-23 - 12 Sep 2022 - 14:27 UTC

[matthewkantar]

It melted because it got too hot.

[GWH]

Great to see that they released full details after all!

[davispw]

I wouldn’t call that “full”. Many details left unsaid.

What design changes were made to the boundary layer cooling system that led to this? When? Why?

Had this “flight configuration” been flown before? With humans? When?

How did the changes pass testing?  What changes to design and testing processes have been made?

Are the remediative changes reverting to a previous configuration or are they new?  What will requalification entail?  Timeline? 

What are the design changes to the combustion chamber vs. changes to “operating parameters”?  Will performance be impacted?

What G load was experienced by the payload?

[kevinof]

So they made design changes, fired up the new design but didn’t catch the problem in testing this change?

Sounds like a testing process failure to me.

[woods170]

It melted because it got too hot.

Nope. It didn't melt. Contrary to what some of you might think, not all things melt when they become too hot. Some stuff just breaks or cracks. That's what happened in this case. Localized spots in the nozzle got too hot, which weakened them by thermal cracking. Those localized weakened spots continued to build cracks to the point where they could no longer contain the physical forces working on the nozzle (aeroloads on the outside, exhaust loads on the inside). The result is (part of) the nozzle breaking off.

[woods170]

So they made design changes, fired up the new design but didn’t catch the problem in testing this change?

Sounds like a testing process failure to me.

That would be a fair assumption.

BUT, please remember that even with substantial hotfire testing, not every nasty surprise will be caught. The history of rocketry is riddled with examples where engines, under flight conditions, acted differently from what witnessed during hotfire testing. External factors such as aeroloading and acceleration are rather hard to simulate during hotfire testing.

[ulm_atms]

So they made design changes, fired up the new design but didn’t catch the problem in testing this change?

Sounds like a testing process failure to me.

That would be a fair assumption.

BUT, please remember that even with substantial hotfire testing, not every nasty surprise will be caught. The history of rocketry is riddled with examples where engines, under flight conditions, acted differently from what witnessed during hotfire testing. External factors such as aeroloading and acceleration are rather hard to simulate during hotfire testing.

It actually sounds like they were already using the thermal margin they had and in "production" it finally fell over.  Metal will usually give you signs its running too hot before everything lets loose.

Does anyone know how many flights this specific engine had at time of failure?

[trimeta]

So they made design changes, fired up the new design but didn’t catch the problem in testing this change?

Sounds like a testing process failure to me.

That would be a fair assumption.

BUT, please remember that even with substantial hotfire testing, not every nasty surprise will be caught. The history of rocketry is riddled with examples where engines, under flight conditions, acted differently from what witnessed during hotfire testing. External factors such as aeroloading and acceleration are rather hard to simulate during hotfire testing.

It actually sounds like they were already using the thermal margin they had and in "production" it finally fell over.  Metal will usually give you signs its running too hot before everything lets loose.

Does anyone know how many flights this specific engine had at time of failure?

Do they change the engine on a given "tail" (booster)? Because this had been NS3's ninth flight.

[matthewkantar]

It melted because it got too hot.

Nope. It didn't melt. Contrary to what some of you might think, not all things melt when they become too hot. Some stuff just breaks or cracks. That's what happened in this case. Localized spots in the nozzle got too hot, which weakened them by thermal cracking. Those localized weakened spots continued to build cracks to the point where they could no longer contain the physical forces working on the nozzle (aeroloads on the outside, exhaust loads on the inside). The result is (part of) the nozzle breaking off.

You are one hundred percent sure nothing melted?

[Robert_the_Doll]

So they made design changes, fired up the new design but didn’t catch the problem in testing this change?

Sounds like a testing process failure to me.

That would be a fair assumption.

BUT, please remember that even with substantial hotfire testing, not every nasty surprise will be caught. The history of rocketry is riddled with examples where engines, under flight conditions, acted differently from what witnessed during hotfire testing. External factors such as aeroloading and acceleration are rather hard to simulate during hotfire testing.

It actually sounds like they were already using the thermal margin they had and in "production" it finally fell over.  Metal will usually give you signs its running too hot before everything lets loose.

Does anyone know how many flights this specific engine had at time of failure?

Do they change the engine on a given "tail" (booster)? Because this had been NS3's ninth flight.

Good question as the NS2 engine flew all five missions from NS-2 to NS-6 (the famous escape test the booster survived), and was retired with it.

NS4 must have something different about it as it was flown every two months on average and flew eight missions within a year's time compared to the ill-fated NS3 booster which flew nine in five years.

[Vettedrmr]

So, I wonder if NS4 is cleared for flight?

[Rakietwawka2021]

Does anyone know how many flights this specific engine had at time of failure?

Bezos said that they've never had to remove engine from NS, so I think that was ninth flight for this engine

[FutureSpaceTourist]

https://twitter.com/thesheetztweetz/status/1639350756280242176

Update: The FAA says in a statement that its “mishap” investigation into Blue Origin's NS-23 mission failure “remains open.”

https://www.cnbc.com/2023/03/24/blue-origin-ns-23-failure-cause.html

The FAA-required investigation into the Sept. 12, 2022, Blue Origin NS-23 launch mishap remains open. The agency is currently reviewing the company's submission of its mishap report. FAA approval is required to close the investigation and for the New Shepard System to return to flight.
Learn more about the FAA Commercial Space Mishap Response Program here.

[HVM]

So it's was design failure at film cooling, and structural fatigue (or fatigue over all) means multiple thermal (or stress) cycles? I hate that this to be is first example of problems with reuse. Ok, SSMEs had their own problems and one Merlin liked too much the rubbing alcohol.

[whitelancer64]

So it's was design failure at film cooling, and structural fatigue (or fatigue over all) mean multiple thermal (or stress) cycles? I hate that this to be is first example of problems with reuse. Ok, SSMEs had their own problems and one Merlin liked too much the rubbing alcohol.

The isopropyl alcohol incident was in March 2020, booster 1048.5 (the first time a booster had flown 5 times) had an engine failure just before MECO. The engine failure prevented the booster from landing.

Also a hole developed in a flexible heat shield boot around one engine that caused the loss of booster 1059.6 on descent, in February 2021.

[Robert_the_Doll]

So it's was design failure at film cooling, and structural fatigue (or fatigue over all) means multiple thermal (or stress) cycles? I hate that this to be is first example of problems with reuse. Ok, SSMEs had their own problems and one Merlin liked too much the rubbing alcohol.

You forgot the corroded fuel line nut on Falcon-1 FalconSAT2 which caused an engine fire and the loss of vehicle. And keep in mind that the Falcon-1 and Merlin 1A was designed around parachute-assisted sea landing and recovery.

[LouScheffer]

...the flight configuration of the nozzle operated at hotter temperatures than previous design configurations.

So they made design changes, fired up the new design but didn’t catch the problem in testing this change?

Sounds like a testing process failure to me.

Surely when you change the design of the nozzle you'd worry about the effect on cooling, but apparently this was not tested.  Assuming BO engineers are competent, I'd guess that the change, whatever it was, was not expected to change cooling materially, perhaps concentrating on manufacturability, weight, outer jacket material, attachments,  etc.  But it DID affect cooling and hence the problem.

I'd be very interested in seeing the part of the failure report that talks about HOW this problem slipped through the cracks, not just what the final problem was.  Also, of course, what steps are being taken to avoid similar problems.  I hope this portion also sees the light of day.

[HVM]

So it's was design failure at film cooling, and structural fatigue (or fatigue over all) means multiple thermal (or stress) cycles? I hate that this to be is first example of problems with reuse. Ok, SSMEs had their own problems and one Merlin liked too much the rubbing alcohol.

You forgot the corroded fuel line nut on Falcon-1 FalconSAT2 which caused an engine fire and the loss of vehicle. And keep in mind that the Falcon-1 and Merlin 1A was designed around parachute-assisted sea landing and recovery.

I worded that badly, I mean problems rising from the reuse; wear and tear (fatigue). Those rocket were not reused.

Thermal damage and hot streaks were visible at nozzle. It is possible those, (or beginning of them) could be detected between flights? Example Space Shuttle solid rocket boosters showed marks of leaking O-rings before Challenger accident, but those were not correctly identified or understood*.

Airliners use “the big data” and deep learning to find signs of problems in engines or other system’s data, and correct them with maintenance, before those become actual problems. We also know that Tesla uses similar system in their cars and factories, and so we can assume that SpaceX uses them too. Both high fly-rate and multi-engine design (and lot of testing) means lot more of data, and that is what big data analytics needs. This can be one reason why they have now the most reliable orbital launch vehicle currently in operation

Blue Origin lacks experience in flight data both traditional engineering sense and with “big data”.
I hope they are fast learners.

* Or there were no consensus of them.

[ThatOldJanxSpirit]

...the flight configuration of the nozzle operated at hotter temperatures than previous design configurations.

So they made design changes, fired up the new design but didn’t catch the problem in testing this change?

Sounds like a testing process failure to me.

Surely when you change the design of the nozzle you'd worry about the effect on cooling, but apparently this was not tested.  Assuming BO engineers are competent, I'd guess that the change, whatever it was, was not expected to change cooling materially, perhaps concentrating on manufacturability, weight, outer jacket material, attachments,  etc.  But it DID affect cooling and hence the problem.

I'd be very interested in seeing the part of the failure report that talks about HOW this problem slipped through the cracks, not just what the final problem was.  Also, of course, what steps are being taken to avoid similar problems.  I hope this portion also sees the light of day.

Totally agree. With all investigations into failures the really important part is understanding the underlying procedural and cultural issues. Here we seem to have questions on the robustness of the change control, testing and inspection processes.

The problem for me is that I’d expect those processes to be common across all Blues engine programs.

[JayWee]

...the flight configuration of the nozzle operated at hotter temperatures than previous design configurations.

So they made design changes, fired up the new design but didn’t catch the problem in testing this change?

Sounds like a testing process failure to me.

Surely when you change the design of the nozzle you'd worry about the effect on cooling, but apparently this was not tested.  Assuming BO engineers are competent, I'd guess that the change, whatever it was, was not expected to change cooling materially, perhaps concentrating on manufacturability, weight, outer jacket material, attachments,  etc.  But it DID affect cooling and hence the problem.

I'd be very interested in seeing the part of the failure report that talks about HOW this problem slipped through the cracks, not just what the final problem was.  Also, of course, what steps are being taken to avoid similar problems.  I hope this portion also sees the light of day.

I wonder - how many people who originally designed the BE3 are still at BO ?