SpaceX F9 : Starlink 6 (v1.0 L5) : Mar. 18, 2020 - Discussion

[JMS]

It seems a mistake to me to punish SpaceX for doing more testing and pushing their engines harder, which was in part to find problems, than anyone else has ever done.

Any delay would not be meant to "punish" SpaceX.

[ugordan]

It seems a mistake to me to punish SpaceX for doing more testing and pushing their engines harder, which was in part to find problems, than anyone else has ever done.

I think you're looking at this from the wrong standpoint.

If this engine failure turns out to be a consequence of material wear on the hardware (old turbine blades cracking, coking build up, name your poison) then the following question/reasoning obviously comes to mind. SpaceX's own refurbishment, testing and modelling deemed the vehicle was fit to fly - otherwise they would not have launched in the first place, I don't think you'll find any disagreement there. Since we did have an in-flight engine anomaly that, then, suggests that that analysis/lifetime assessment of the hardware or just plain margins in the hardware to begin with were modeled wrongly by SpaceX. So it makes sense to take a deeper look at this discrepancy, if for no other reason, but to understand what effects could have slipped through the cracks during refurbishment that SpaceX should know about or just plain simply understand the actual hardware better.

Any anomaly should be analyzed with due diligence by the launch provider. Since SpaceX is the only one actually reflying booster stages right now, situations like this are just something that goes along with the territory.

As for "punishing" SpaceX for pushing the hardware hard, yes, they can do that on their own dime, but SpaceX themselves
a) want to keep their reputation as a reliable launch provider with no-fuss launches and, for NASA, more importantly
b) this launch went off from LC-39A, the Crew Dragon pad. Imagine if there was an Amos-6 like anomaly on the pad simply because SpaceX pushed reused hardware 1 launch too far and had a really bad day. NASA does have a vested interest in understanding all the rationale and reasoning that goes into every SpaceX launch decision.

If it turns out (which I hope) that this is a multiple reuse-effect, then NASA will not be "punishing" SpaceX about it, but they might have comments on launching life-leader boosters from LC-39A.

But, at the end of the day, the actual root cause has to be determined first. It's the only reasonable course of action by SpaceX and also, by way of invested interest, NASA.  I certainly don't expect a separate NASA anomaly board to investigate this. DM-2 is still 2 months away so there is the possibility that this won't make any appreciable impact on that schedule. Probably less so than certain human malware floating around the world right now.

[TheRadicalModerate]

Since we did have an in-flight engine anomaly that, then, suggests that that analysis/lifetime assessment of the hardware or just plain margins in the hardware to begin with were modeled wrongly by SpaceX. So it makes sense to take a deeper look at this discrepancy, if for no other reason, but to understand what effects could have slipped through the cracks during refurbishment that SpaceX should know about or just plain simply understand the actual hardware better.

Any anomaly should be analyzed with due diligence by the launch provider. Since SpaceX is the only one actually reflying booster stages right now, situations like this are just something that goes along with the territory.

Sounds right to me.

I'd add that if this was a not-very-well-contained engine failure that did aerodynamic damage (see my comment a bit up-thread for why I believe this might be the case), you don't get a whole lot of chances like this to verify your structural models.  If you wind up doing the Bayesian rumba on the structural and aero failure trees as a result, NASA will at least want to run their Monte Carlo failure sims through with the new probabilities.

[mandrewa]

A failure like this is an opportunity and a risk.  It's an opportunity if they get enough data on what happened to identify a problem, and in particular a problem that they can do something about.

It's a risk if it's not clear what caused the problem and or even worse they falsely attribute it to something.

Attempting to get to 10 reuses of a first stage with minimal refurbishment becomes risky if every failure like this means a lengthy and prolonged investigation.

Look if some basic issue turns up that they were not aware of, well then we should be grateful for that, and take as long as necessary to fix it.  But I think that the way these boosters are being used, it is more or less inevitable that there will be problems.  And it is going to be valuable to see how they fail! 

But only if we do not treat each failure as a crisis, or as a sign that things have gone significantly wrong.

[Vettedrmr]

Any anomaly should be analyzed with due diligence by the launch provider. Since SpaceX is the only one actually reflying booster stages right now, situations like this are just something that goes along with the territory.

Which is EM's position very early after the flight:
https://twitter.com/jeff_foust/status/1240263546732240897

[marsbase]

If this engine failure turns out to be a consequence of material wear on the hardware (old turbine blades cracking, coking build up, name your poison) then the following question/reasoning obviously comes to mind. SpaceX's own refurbishment, testing and modelling deemed the vehicle was fit to fly - otherwise they would not have launched in the first place, I don't think you'll find any disagreement there.

In general I agree with you.  But I wouldn't be so black and white about it.  Elon has said that "life leaders" , i.e. the most used rockets, will only be used for SpaceX internal missions and that customer satellites would not be risked in the same way.  SpaceX does static test fires without payload except in the case of SpaceX payload. So there is a gray area in "fit to fly".   If SpaceX modeling says a rocket will blow on the launch pad, they aren't going to push the button.  But beyond that, there are acceptable risks.  There are chances you take that are thought to be worth it.  This mission was one of those and the bet paid off.  Mission success.  I'm sure NASA wants to know the details (may have known the details before the launch) but that does not mean that SpaceX modeling was wrong.  The model may have correctly indicated an acceptable risk.

[lrk]

There was a fireball in the plume visible from ground, but that could be nothing more than a bit of vented unburnt propellant flaring up in the exhaust

My theory on that flare-up: M1Ds vent a bit of GOX for a brief moment post-shutdown - see the white cloud briefly visible at MECO on any F9 mission.  This would lead to additional combustion of unburned exhaust - like how the plume on CRS-7 got brighter due to the added venting O2:

[VIY]

It is no big tragedy that an engine failed at the fifth reflight. It is a wonder that the buster launched satelites to orbit 5 times! I never bought EM's talk about 10-100 reflights. Though 10 might be achievable after enough experience.

[TheRadicalModerate]

My theory on that flare-up: M1Ds vent a bit of GOX for a brief moment post-shutdown - see the white cloud briefly visible at MECO on any F9 mission.  This would lead to additional combustion of unburned exhaust - like how the plume on CRS-7 got brighter due to the added venting O2:

https://www.youtube.com/watch?v=OAX7UFd70M8?t=164

Go two frames before the flash.  There's a hunk of something coming off the vehicle.  This was not an engine that shut down due to some parameter being a bit out of bounds.  Some piece of hardware failed.

FWIW, there's also a four-frame dropout just before the frame with the debris.  At 30 fps 1080, that's 133 ms unaccounted for.  Giving the antenna a big jolt would account for that.

[su27k]

I haven't seen any talk about how the engine anomaly during the last Starlink launch might affect the scheduling of DM2. The announcement by NASA of media accreditation for DM2 seems to suggest there won't be any holdup, which I find a bit surprising.

Eric Berger doesn't think it'll have a big impact on DM-2:

https://twitter.com/SciGuySpace/status/1240655195555667972

Media invite for May Demo-2 launch is interesting given high uncertainties w/ Covid-19. Crews still training for now, but situation changing from day-to-day.

Engine-out issue with Falcon 9 probably will not significantly effect things as DM-2 launches on new first stage.

[Steven Pietrobon]

Here are screen shots from this video. I've cropped and enlarged the frames from the 1080p video. The frame grabber I used could only extract every other frame. There's no way to tell which engine went, but the engine failure was pretty dramatic. It looks like there was a plume of burning fuel that moves to the right of vehicle.

[JamesH65]

All the hand wringing, but to me, all this is is an opportunity for SpaceX to improve their modelling, to improve their refurb process (if necessary), to improve the design.

It means that the next F9 they make will be even better.

This one paid for itself and it's sad demise will make future first stages and Merlins even better.

[TheRadicalModerate]

Is there any news on whether they recovered the core?

[AndrewRG10]

Is there any news on whether they recovered the core?

They said on the live stream they didn't recover it, the droneship left without doing any securing like it usually does, the booster looked like it was breaking up on entry. No, they didn't recover it

[TheRadicalModerate]

Is there any news on whether they recovered the core?

They said on the live stream they didn't recover it, the droneship left without doing any securing like it usually does, the booster looked like it was breaking up on entry. No, they didn't recover it

That was my supposition when I saw how squirrelly the reentry burn went, but my suppositions are often wrong.  I was hoping that this would be one of those (many) cases.  ISTM that this radically increases the probability that an aerosurface was damaged in the failure, and that's something that NASA is gonna want to understand before DM-2.

Bummer.

So I assume that that leads to one of two cases:

1) There's clear evidence of the nature of the failure in the telemetry, and that's enough to close out the incident in short order.

2) There's not clear evidence, and SpaceX may be forced to go hunting for the thrust structure to get answers.  That would almost certainly delay DM-2.

Of course, DM-2 is gonna get delayed anyway because of COVID.

[Lars-J]

So I assume that that leads to one of two cases:
...
2) There's not clear evidence, and SpaceX may be forced to go hunting for the thrust structure to get answers.  That would almost certainly delay DM-2.

They would never be able to recover the thrust structure if it sank. It just isn't happening, and they would have to analyze it the old-fashioned way, with informed guesswork, fault modelling, and so on. They way it has always been done for rockets pre-re-usability.

[TheRadicalModerate]

So I assume that that leads to one of two cases:
...
2) There's not clear evidence, and SpaceX may be forced to go hunting for the thrust structure to get answers.  That would almost certainly delay DM-2.

They would never be able to recover the thrust structure if it sank. It just isn't happening, and they would have to analyze it the old-fashioned way, with informed guesswork, fault modelling, and so on. They way it has always been done for rockets pre-re-usability.

You're right: that's case #3.  But I doubt it's any quicker to resolve than case #2.

Update:  I looked at the OCSILY position, which was very near the Blake Escarpment, where the continental shelf falls into the Atlantic Basin.  But, assuming that a breakup would have fallen a bit south and west of the OCSILY's position, that would be still well up on the Blake Plateau.  I'm just eyeballing the charts, but debris might be in 500m water, which isn't incredibly hard to survey.  And I'd think that a thrust structure would stick out pretty well on sonar.  Not impossible to recover, but certainly a lot of work.  Kinda depends on how bad they need answers.

[Vettedrmr]

Kinda depends on how bad they need answers.

I think SpaceX will be talking to their customer's about what happened and how much concern their customers have.  That will determine how bad they need to know exactly what happened.  Most comments have been that the anticipated impact will be either minor or absorbed into the normal launch schedule.  SpaceX may be the biggest driver into analyzing what happened, just because they're going to be using the most re-used rockets for Starlink launches.

I do think they may have found the upper end of how fast their launch cadence can run.  They've had (correct me if I'm wrong):

1. A successful, but hard landing from Starlink L3
2. An unsuccessful landing next to the drone ship from L4, due to incorrect wind data being loaded into the flight control system
3. A totally successful landing from CRS-20
4. An inflight engine failure from Starlink L5, with the most re-used booster

Don't get me wrong, I think SpaceX is doing amazing things.  But, based on EM's tweet right after this latest recovery failure, this kind of engine failure wasn't even close to being expected.

I anticipate maintaining launch schedules for their customers.  But Starlink launches may just start taking a bit more internal scrutiny.

Have  good one,
Mike

[CorvusCorax]

It seems a mistake to me to punish SpaceX for doing more testing and pushing their engines harder, which was in part to find problems, than anyone else has ever done.

I think you're looking at this from the wrong standpoint.

If this engine failure turns out to be a consequence of material wear on the hardware (old turbine blades cracking, coking build up, name your poison) then the following question/reasoning obviously comes to mind. SpaceX's own refurbishment, testing and modelling deemed the vehicle was fit to fly - otherwise they would not have launched in the first place, I don't think you'll find any disagreement there. Since we did have an in-flight engine anomaly that, then, suggests that that analysis/lifetime assessment of the hardware or just plain margins in the hardware to begin with were modeled wrongly by SpaceX. So it makes sense to take a deeper look at this discrepancy, if for no other reason, but to understand what effects could have slipped through the cracks during refurbishment that SpaceX should know about or just plain simply understand the actual hardware better.

Any anomaly should be analyzed with due diligence by the launch provider. Since SpaceX is the only one actually reflying booster stages right now, situations like this are just something that goes along with the territory.

As for "punishing" SpaceX for pushing the hardware hard, yes, they can do that on their own dime, but SpaceX themselves
a) want to keep their reputation as a reliable launch provider with no-fuss launches and, for NASA, more importantly
b) this launch went off from LC-39A, the Crew Dragon pad. Imagine if there was an Amos-6 like anomaly on the pad simply because SpaceX pushed reused hardware 1 launch too far and had a really bad day. NASA does have a vested interest in understanding all the rationale and reasoning that goes into every SpaceX launch decision.

If it turns out (which I hope) that this is a multiple reuse-effect, then NASA will not be "punishing" SpaceX about it, but they might have comments on launching life-leader boosters from LC-39A.

But, at the end of the day, the actual root cause has to be determined first. It's the only reasonable course of action by SpaceX and also, by way of invested interest, NASA.  I certainly don't expect a separate NASA anomaly board to investigate this. DM-2 is still 2 months away so there is the possibility that this won't make any appreciable impact on that schedule. Probably less so than certain human malware floating around the world right now.

Emphasis mine.

While this is overall a thoughtful analysis, the highlighted statements suggest it's based on a misconception on how SpaceX operates. Specifically when it comes to unknowns about margins and edge of the envelope cases, most operators would not fly if there is any doubt about the outcome.

But SpaceX is not most operators. If their models suggest uncertainty about the outcome, they try it out, fully knowing that there is a high chance of losing hardware in the process. Another operator might have decided to put that flown booster on a test stand and do a full duration test fire before launching it (or not launch it at all, take it apart and only use that information to decide whether to launch the next booster a 5th time) - but that is much more work intensive, much slower, and you learn more by simply flying it. There are failure cases that won't happen on the test stand.

The same with the previous flights. The landing winds were beyond the supported envelope for F9 landing. The safe option would have been to either postpone launch OR to forego the landing or land at sea to measure the accuracy in a "safe test". But by trying to land it at the pad, although there is a higher risk of unforseen and unexpected outcomes.

It's exactly that higher risk for unforseen and unexpected outcomes that SpaceX is seeking to gain the most valuable data.

As such I would assume the engine failure on this launch was a calculated risk. It was not expected to happen, but it was not a case of wrong modelling leading to thinking they were safe. They knew they were potentially unsafe, which made this flight extra valuable as a data point.

Modeling and ground tests will only teach you more details about known unknowns. If you fire it up and fly you learn about the unknown unknowns.

Doing that from an operational pad with an operational payload (although their own. They obviously wouldn't do that with a customers sats) is a calculated risk, but this part of the margins, SpaceX has under control. They would not launch a booster if their model would tell them the risk of it blowing up on the pad is unacceptably high. They would not launch a payload on it if the risk of payload not getting to orbit is unacceptably high. But 100% flawless mission is not needed. SpaceX HAS enough margin to lose an engine or two on the way up and still make the intended orbit (which they did by the way).

Now of course this needs to be analyzed. Even if an engine fails beyond its so far qualified operational life, the failure mode and details might still expose hidden flaws that can impact operational missions. That is the case regardless if this happens on an operational launch or if a flaw is detected on a test bench or on a test stand. It would just be less public and open. We might not have heard about it, if a crack had been found during x-ray inspection of every single turbine blade in a shed in Mac Gregor. But the potential impact on DM2 and other operational launch is just the same.

Or not. This booster was old. So old, that AFAIK the engines didn't have the latest modifications SpaceX did to mitigate crack forming for human rated missions. As such it's completely possible whatever caused this won't even affect DM2 hardware.

But nevertheless the analysis needs to be completed either way - even if just to rule out any effect on DM2, you still need to find out what exactly happened. Elon Musk tweeted nothing less:

https://twitter.com/elonmusk/status/1240262636547100672

Yeah. There was also an early engine shutdown on ascent, but it didn’t affect orbit insertion. Shows value of having 9 engines! Thorough investigation needed before next mission.

(emphasis mine)

[wannamoonbase]

All the hand wringing, but to me, all this is is an opportunity for SpaceX to improve their modelling, to improve their refurb process (if necessary), to improve the design.

It means that the next F9 they make will be even better.

This one paid for itself and it's sad demise will make future first stages and Merlins even better.

I agree.  This is an opportunity and to be expected sometime in the development of reuse.  There will be lessons in this for Merlin and likely Raptor.

The stated design goal of 10 flights per booster was going to test the limits of the hardware at some point. 

It will be interesting to see what they learn and what improvements are made.

I’m surprised it wasn’t the center engine since it takes so much of the work load.

Edit: I don’t think SpaceX would have taken a known risk if they suspected the engine was troubled before flight.  To much to risk wit the booster, payload, commercial clients and upcoming crewed flights.