Why did FH center core run out of TEA-TEB?

[IainMcClatchie]

http://www.latimes.com/business/la-fi-spacex-falcon-heavy-core-20180212-story.html

Musk said Monday that there wasn't enough ignition fluid to light the outer two engines of the booster "after several three engine relights."

It seems the center core performed a three-engine boostback, three-engine reentry, and (attempted) three-engine landing burn, and the last relight failed.

They've done lots of relights in the past, including triple three-engine burns, and they've gotten the rockets back afterwards and been in a position to measure the amount of TEA-TEB actually used.

If we set aside the possibility of a dumb mistake, what might have happened on this flight that would cause the actual flight consumption of TEA-TEB to be different from what they expected?

What was different this time?

Is the TEA-TEB ignition system even feedback controlled?  The naive way to implement an ignitor is to simply open a valve for a fixed amount of time each time a relight is commanded.  Maybe the flow of TEA-TEB varies with the backpressure or stage acceleration.  It's hard for me to imagine the launch or boostback burns having any significant variation in how that might proceed.  I also expect that the reeentry burn timing is driven by heating rather than dynamic pressure, so that the reentry reignition happens at vanishingly small dynamic pressure and so would be very repeatable.

The only reignition that has interesting dynamics would be the landing burn.  Did the center core relight while travelling faster than prior three-engine landings have?  This is possible if the center core arrived on a more vertical trajectory than prior landings.

[Jim]

Look on the mission discussion threads and you will see why

https://forum.nasaspaceflight.com/index.php?topic=44778.msg1787712;topicseen#msg1787712

[IainMcClatchie]

Thank you.

[Lar]

in https://twitter.com/elonmusk/status/963107229523038211 Musk said:

Not enough ignition fluid to light the outer two engines after several three engine relights. Fix is pretty obvious.

[atsf90east]

in https://twitter.com/elonmusk/status/963107229523038211 Musk said:

Not enough ignition fluid to light the outer two engines after several three engine relights. Fix is pretty obvious.

Of all of the possible anomalies they could have encountered on this mission this one was probably the one that would do the least harm {other than losing a core that was not going to be re-flown anyway), and also one of the easiest to fix.

[smh]

How much TEA-TEB would it normally take to light an engine?

[speedevil]

How much TEA-TEB would it normally take to light an engine?

And what's lit?
As I understand it, you have the partial flow of fuel and oxidiser running into the turbopump - and then the engine bell.
Do both of these require lighting, or is something clever done?

[TheRadicalModerate]

http://www.latimes.com/business/la-fi-spacex-falcon-heavy-core-20180212-story.html

Musk said Monday that there wasn't enough ignition fluid to light the outer two engines of the booster "after several three engine relights."

It seems the center core performed a three-engine boostback, three-engine reentry, and (attempted) three-engine landing burn, and the last relight failed.

They've done lots of relights in the past, including triple three-engine burns, and they've gotten the rockets back afterwards and been in a position to measure the amount of TEA-TEB actually used.

If we set aside the possibility of a dumb mistake, what might have happened on this flight that would cause the actual flight consumption of TEA-TEB to be different from what they expected?

What was different this time?

Is the TEA-TEB ignition system even feedback controlled?  The naive way to implement an ignitor is to simply open a valve for a fixed amount of time each time a relight is commanded.  Maybe the flow of TEA-TEB varies with the backpressure or stage acceleration.  It's hard for me to imagine the launch or boostback burns having any significant variation in how that might proceed.  I also expect that the reeentry burn timing is driven by heating rather than dynamic pressure, so that the reentry reignition happens at vanishingly small dynamic pressure and so would be very repeatable.

The only reignition that has interesting dynamics would be the landing burn.  Did the center core relight while travelling faster than prior three-engine landings have?  This is possible if the center core arrived on a more vertical trajectory than prior landings.

I wonder if, as part of the throttle-down of the center core immediately after launch, they actually did a shutdown of some of the engines, and then re-lit them once the boosters were jettisoned.  That would prevent the engines in the center core from running in a prolonged throttled state, which may be hard on the engine and almost certainly results in some loss of specific impulse.

This is pure speculation, but it would explain why they didn't get the TEA-TEB levels right: because they'd never done a re-light in this kind of flight regime.

[Jim]

And what's lit?
As I understand it, you have the partial flow of fuel and oxidiser running into the turbopump - and then the engine bell.
Do both of these require lighting, or is something clever done?

both, gas generator for turbopump and combustion chamber have to be lit.  The gas generator first

[Jim]

I wonder if, as part of the throttle-down of the center core immediately after launch, they actually did a shutdown of some of the engines,

no shutdowns

[IainMcClatchie]

Well, shoot, there goes the best theory I've heard so far for what was different than simulation.

My next best theory is a sticky valve that let too much TEA-TEB go during one of the early ignitions.  This is almost a trite conclusion, because there's not much else in the fault tree.

* faulty planning
* faulty loading
* in-flight variation from plan
   * TEA-TEB leaked through new hole (burn through?)
   * more TEA-TEB than planned went through a valve
      * valve was commanded because the computer saw something unplanned in flight (assumes ignition shot control uses feedback)
      * valve was not commanded (sticky valve)

I guess I'm struggling to imagine what was different on the center core for this flight than for booster cores recovered on drone ships in previous flights -- different that would have affected ignition.

[Norm38]

The official explanation still doesn't make much sense.  They've done 7 engine relights (3-3-1) in the past, landed the stage and knew how much fluid they'd used.  Then they decide to do 9 relights (3-3-3).
Take the amount of fluid used for 7, multiply by 9/7 (29% extra), and use that.

So if they didn't have 30% more fluid loaded, they simply messed up.
If they did have 30% more, then it's not so simple.

[Lar]

If the control computer goes on "did it start" maybe it took longer to start so it used more. If the control computer goes on time, maybe the flow rate is higher in this environment for whatever reason. A misfill is possible but ... really?

[LouScheffer]

Well, shoot, there goes the best theory I've heard so far for what was different than simulation.

I guess I'm struggling to imagine what was different on the center core for this flight than for booster cores recovered on drone ships in previous flights -- different that would have affected ignition.

How about the obvious?
(a) The core did its boostback burn at a higher velocity than any core recovery to date (2750 m/s as opposed to about 2350 m/s)
(b) The core is heavier, which will affect its aerodynamics, angle of attack, approach to terminal velocity, center of gravity, and so on.

Likely one of these effects made it (unexpectedly) hard to relight at least one of the engines.  It would also be quite difficult to accurately simulate or predict these conditions without trying it.

[envy887]

The official explanation still doesn't make much sense.  They've done 7 engine relights (3-3-1) in the past, landed the stage and knew how much fluid they'd used.  Then they decide to do 9 relights (3-3-3).
Take the amount of fluid used for 7, multiply by 9/7 (29% extra), and use that.

So if they didn't have 30% more fluid loaded, they simply messed up.
If they did have 30% more, then it's not so simple.

"Why" may not be that simple. "How" to fix it could easily just be more fluid, regardless of the reasons it took more than expected.

[Jim]

Well, shoot, there goes the best theory I've heard so far for what was different than simulation.

I guess I'm struggling to imagine what was different on the center core for this flight than for booster cores recovered on drone ships in previous flights -- different that would have affected ignition.

How about the obvious?
(a) The core did its boostback burn at a higher velocity than any core recovery to date (2750 m/s as opposed to about 2350 m/s)
(b) The core is heavier, which will affect its aerodynamics, angle of attack, approach to terminal velocity, center of gravity, and so on.

Likely one of these effects made it (unexpectedly) hard to relight at least one of the engines.  It would also be quite difficult to accurately simulate or predict these conditions without trying it.

Or it affected the fluid management of the TEA-TEB in its tank

[IainMcClatchie]

Or it affected the fluid management of the TEA-TEB in its tank

Hey there we go.  Something like the rotation of the booster on it's first attempt at a soft ocean touchdown causing the propellant to climb the walls of the tank and starve the engine.

I am suspicious of the idea that backpressure in the engines causes relighting difficulty.  The stagnation pressure during the landing relight is 17 kPa (boosters) and 13 kPa (core), according to Zach's simulation here.  That's less than 0.2 bar, which is going to be less than the pressure drop across whatever nozzle injects the TEA-TEB for relight.  My guess is the TEA-TEB tank is probably at least ten times that pressure, and if the nozzle is choked flow, no minor pressure fluctuation downstream like that is going to affect the mass flow.

The entry burn relight is at even lower stagnation pressure: under 300 Pa, so that's not a problem either.

I think it's interesting that the peak stagnation pressure during the entry burn is so low, about 10 kPa.  It suggests that a very slow sooty burn might be able to keep the entry plasma and its radiative transfer away from the vehicle during an upper stage reentry.  That's a big deal, because while the Shuttle was able to use a ceramic face that was solid at the reentry stagnation temperature, reentry from the Moon or Mars will be fast enough that radiative heat transfer will dominate, and that can't be fixed with a higher temperature ceramic.  Fast reentry will have to use some expendable mass to dump the heat into.  Might be nicer to have that be propellant, which is easier to replace than a heat shield when you're millions of miles from the rocket parts store.

[cuddihy]

I wonder about lighting a Raptor with torch ignition under these conditions— will SpaceX lose a $500M test BFS the first time they try reentry landing? May be worth a F9 launched test reentry vehicle.

[rory]

How about the obvious?
(a) The core did its boostback burn at a higher velocity than any core recovery to date (2750 m/s as opposed to about 2350 m/s)

Why would this matter? Boostback burn happens at about the Karman line — the atmosphere (and thus speed relative to it/the surface) should be inconsequential.

[LouScheffer]

I am suspicious of the idea that backpressure in the engines causes relighting difficulty.  The stagnation pressure during the landing relight is 17 kPa (boosters) and 13 kPa (core), according to Zach's simulation here.  That's less than 0.2 bar, which is going to be less than the pressure drop across whatever nozzle injects the TEA-TEB for relight.  My guess is the TEA-TEB tank is probably at least ten times that pressure, and if the nozzle is choked flow, no minor pressure fluctuation downstream like that is going to affect the mass flow.

The entry burn relight is at even lower stagnation pressure: under 300 Pa, so that's not a problem either.

I think it's interesting that the peak stagnation pressure during the entry burn is so low, about 10 kPa. 

These values seemed way too low to me, based strictly on experience of sticking my hand out the window of a car and feeling the forces.

But rockets are light for their size.   If a landing rocket masses 30 tonnes, and is falling at terminal velocity, then the force must be 300 kN.  The cross sectional area is about 10 m^2, so that's about 30 kPa as a very crude approximation.   So well under 1 bar (100 kPa), as above, and the engine uses much higher pressures than this, so starting into the wind should not be a big deal.

That's why you do the calculations....

[hkultala]

I am suspicious of the idea that backpressure in the engines causes relighting difficulty.  The stagnation pressure during the landing relight is 17 kPa (boosters) and 13 kPa (core), according to Zach's simulation here.  That's less than 0.2 bar, which is going to be less than the pressure drop across whatever nozzle injects the TEA-TEB for relight.  My guess is the TEA-TEB tank is probably at least ten times that pressure, and if the nozzle is choked flow, no minor pressure fluctuation downstream like that is going to affect the mass flow.

The entry burn relight is at even lower stagnation pressure: under 300 Pa, so that's not a problem either.

I think it's interesting that the peak stagnation pressure during the entry burn is so low, about 10 kPa. 

These values seemed way too low to me, based strictly on experience of sticking my hand out the window of a car and feeling the forces.

But rockets are light for their size.   If a landing rocket masses 30 tonnes, and is falling at terminal velocity, then the force must be 300 kN.

Terminal velocity changes with altitude. As the rocket starts with high velocity, and atmosphere is getting thicker and terminal velocity is getting lower, it's all the time coming at considerably higher velocity than the "theoretical" terminal velocity for that altitude, and is constantly decelerating.

This means that the aerodynamical forces affecting it are higher than 9.81 N/kg.

[IainMcClatchie]

The aerodynamic forces on the rocket when it lights up for reentry are far less than 9.81 N/kg.  It's basically in freefall at that point, and dynamic pressure is miniscule.

For the booster landing, Zach has the booster relighting at 174 m/s at an altitude of 900 m.  Standard air density at that altitude is 1.13 kg/m^3.  Dynamic pressure is 17106 Pa.

Now what is interesting is that Zach has the deceleration, just before booster landing relight, at 1.194 G, and the mass at 24605 kg.  So that says there is 288 kN aerodynamic force on the booster.  If Cd = 1, area is 16.8 m^2.  The vehicle base is 10.7 m^2, and I think the rest is the body (which is still at a bit of an angle), the grid fins, and perhaps a Cd a little higher than 1.

I don't know that the booster's velocity profile has much to do with terminal velocity at any given point.  At the start of reentry, it is well below its terminal velocity and accelerating toward the ground.  After the reentry burn, it's still in freefall (well below terminal velocity). As it drops into the thicker air, the terminal velocity comes way down and the rocket experiences up to 7 G deceleration (so its way above terminal velocity).  During the landing burn, it briefly matches terminal velocity and then drops below, all the way to zero velocity at the ground obviously.

[Roy_H]

I wonder if, as part of the throttle-down of the center core immediately after launch, they actually did a shutdown of some of the engines,

no shutdowns

I know, Jim, that you are one of the best authorities on this web site, and if I have my info correct, you are a NASA employee, not a SpaceX employee. I certainly agree that the likelihood of FH center core engines being shut down during ascent is slim, but the argument makes sense. So how do you know for sure it didn't happen? If SpaceX did want to try this, would they be required to inform NASA even though it was not a NASA payload?

[docmordrid]

I wonder if, as part of the throttle-down of the center core immediately after launch, they actually did a shutdown of some of the engines,

no shutdowns

>
So how do you know for sure it didn't happen?

Only 3 engines can restart because they're the only ones tied to the rockets internal TEA-TEB system. As the FH center core proved you don't want to exhaust the TEA-TEB. Better to throttle.

[speedevil]

Only 3 engines can restart because they're the only ones tied to the rockets internal TEA-TEB system. As the FH center core proved you don't want to exhaust the TEA-TEB. Better to throttle.

Merlin 1D can probably throttle to 40% or so of maximum.
Shutting down 3/9 engines gets you to 26%.

So, for it to be worth it, it would have to be valulable to operate in that 14% range. This seems not hugely likely on the face of it.

[smndk]

I wonder if, as part of the throttle-down of the center core immediately after launch, they actually did a shutdown of some of the engines,

no shutdowns

>
So how do you know for sure it didn't happen?

Only 3 engines can restart because they're the only ones tied to the rockets internal TEA-TEB system. As the FH center core proved you don't want to exhaust the TEA-TEB. Better to throttle.

How are the engines that are not tied to the rockets internal TEA-TEB system initially startet?

/Svend

[docmordrid]

I wonder if, as part of the throttle-down of the center core immediately after launch, they actually did a shutdown of some of the engines,

no shutdowns

>
So how do you know for sure it didn't happen?

Only 3 engines can restart because they're the only ones tied to the rockets internal TEA-TEB system. As the FH center core proved you don't want to exhaust the TEA-TEB. Better to throttle.

How are the engines that are not tied to the rockets internal TEA-TEB system initially startet?

/Svend

On the ground TEA-TEB is supplied externally.

https://forum.nasaspaceflight.com/index.php?topic=42705.msg1745087#msg1745087

[Roy_H]

Only 3 engines can restart because they're the only ones tied to the rockets internal TEA-TEB system. As the FH center core proved you don't want to exhaust the TEA-TEB. Better to throttle.

Merlin 1D can probably throttle to 40% or so of maximum.
Shutting down 3/9 engines gets you to 26%.

So, for it to be worth it, it would have to be valulable to operate in that 14% range. This seems not hugely likely on the face of it.

You guys are missing the point here. Rocket engines are less efficient when throttled. Now I don't know how much but at 40% throttle maybe it uses 60% fuel rate. Anybody know actual figures?

[hkultala]

Only 3 engines can restart because they're the only ones tied to the rockets internal TEA-TEB system. As the FH center core proved you don't want to exhaust the TEA-TEB. Better to throttle.

Merlin 1D can probably throttle to 40% or so of maximum.
Shutting down 3/9 engines gets you to 26%.

So, for it to be worth it, it would have to be valulable to operate in that 14% range. This seems not hugely likely on the face of it.

You guys are missing the point here. Rocket engines are less efficient when throttled.

yes, but...

Now I don't know how much but at 40% throttle maybe it uses 60% fuel rate. Anybody know actual figures?

No, isp does not drop from 300 seconds to 200 seconds because of throttling to 40%.

it drops maybe something like from 300 seconds to 290-294 seconds, like 2-3% instead of 33%.

[envy887]

Only 3 engines can restart because they're the only ones tied to the rockets internal TEA-TEB system. As the FH center core proved you don't want to exhaust the TEA-TEB. Better to throttle.

Merlin 1D can probably throttle to 40% or so of maximum.
Shutting down 3/9 engines gets you to 26%.

So, for it to be worth it, it would have to be valulable to operate in that 14% range. This seems not hugely likely on the face of it.

You guys are missing the point here. Rocket engines are less efficient when throttled.

yes, but...

Now I don't know how much but at 40% throttle maybe it uses 60% fuel rate. Anybody know actual figures?

No, isp does not drop from 300 seconds to 200 seconds because of throttling to 40%.

it drops maybe something like from 300 seconds to 290-294 seconds, like 2-3% instead of 33%.

Depends on altitude. I'll plot isp curves for Merlin as a function of throttle and altitude in RPA tomorrow. If I remember

OK, here's the plot. Some assumptions here but should be pretty close. At SL throttling to 40% costs about 15% in isp. But this very rapidly drops with altitude and at optimal expansion (only 2.4 km up!) it only costs about 1%.

[IainMcClatchie]

How are the engines that are not tied to the rockets internal TEA-TEB system initially startet?

On the ground TEA-TEB is supplied externally.

I am amazed that the post-ignition quick disconnect for a hypergolic fluid is lighter weight than just having a larger tank on board.

[Lar]

How are the engines that are not tied to the rockets internal TEA-TEB system initially startet?

On the ground TEA-TEB is supplied externally.

I am amazed that the post-ignition quick disconnect for a hypergolic fluid is lighter weight than just having a larger tank on board.

Why? A larger tank (or tanks) would have to hold 9 more starts than the tank (or tanks, because we don't know if there is a central tank pair or one pair per engine) that only  has to support restarts.

[Nomadd]

How are the engines that are not tied to the rockets internal TEA-TEB system initially startet?

On the ground TEA-TEB is supplied externally.

I am amazed that the post-ignition quick disconnect for a hypergolic fluid is lighter weight than just having a larger tank on board.

Why? A larger tank (or tanks) would have to hold 9 more starts than the tank (or tanks, because we don't know if there is a central tank pair or one pair per engine) that only  has to support restarts.

Somebody must have mentioned it somewhere, but I can't find it.
 How much fluid are we talking about to start an engine? And, is the fluid pressure fed or use pumps?

[IainMcClatchie]

I am amazed that the post-ignition quick disconnect for a hypergolic fluid is lighter weight than just having a larger tank on board.

Why? A larger tank (or tanks) would have to hold 9 more starts than the tank (or tanks, because we don't know if there is a central tank pair or one pair per engine) that only  has to support restarts.

Because scaling up a tank is very lightweight.  For small quantity things like this, the fittings and plumbing can chew up most of the mass.  I wouldn't be surprised to hear that there is enough TEA-TEB in the plumbing for a full restart.

Also note that the ground support equipment, and the rocket plumbing must have sufficient pipe diameter to deliver the shot of TEA-TEB in much less than 1 second.  That plumbing goes from two sources (ground and flight tanks) to 9 destinations.

[Herb Schaltegger]

I am amazed that the post-ignition quick disconnect for a hypergolic fluid is lighter weight than just having a larger tank on board.

Why? A larger tank (or tanks) would have to hold 9 more starts than the tank (or tanks, because we don't know if there is a central tank pair or one pair per engine) that only  has to support restarts.

Because scaling up a tank is very lightweight.  For small quantity things like this, the fittings and plumbing can chew up most of the mass.  I wouldn't be surprised to hear that there is enough TEA-TEB in the plumbing for a full restart.

Also note that the ground support equipment, and the rocket plumbing must have sufficient pipe diameter to deliver the shot of TEA-TEB in much less than 1 second.  That plumbing goes from two sources (ground and flight tanks) to 9 destinations.

I really love it when people are smarter than professionals like Tom Mueller.

[Roy_H]

Depends on altitude. I'll plot isp curves for Merlin as a function of throttle and altitude in RPA tomorrow. If I remember

OK, here's the plot. Some assumptions here but should be pretty close. At SL throttling to 40% costs about 15% in isp. But this very rapidly drops with altitude and at optimal expansion (only 2.4 km up!) it only costs about 1%.

Thank you for clearing this up. Obviously I was wrong as full thrust from all 3 cores is required at lift-off and throttling only occurs when g-forces exceed desired limit. I don't know where that is, but I would expect it to be about the 2.4km altitude so at only 1% loss it becomes a non-issue. I think there would still be some gain by shutting down center core engines to save fuel until after side boosters have done their job, but now I think all 9 engines would have to be shut down and re-started to make any significant difference.

[groundbound]

Depends on altitude. I'll plot isp curves for Merlin as a function of throttle and altitude in RPA tomorrow. If I remember

OK, here's the plot. Some assumptions here but should be pretty close. At SL throttling to 40% costs about 15% in isp. But this very rapidly drops with altitude and at optimal expansion (only 2.4 km up!) it only costs about 1%.

Thank you for clearing this up. Obviously I was wrong as full thrust from all 3 cores is required at lift-off and throttling only occurs when g-forces exceed desired limit. I don't know where that is, but I would expect it to be about the 2.4km altitude so at only 1% loss it becomes a non-issue. I think there would still be some gain by shutting down center core engines to save fuel until after side boosters have done their job, but now I think all 9 engines would have to be shut down and re-started to make any significant difference.

I'l chime in by repeating something I was schooled on in another NSF thread a couple of years ago that impinges on this as well. Merlin is a GG engine, not SC. The GG losses are not necessarily a constant percentage as you go to higher chamber pressures. So if GG losses go up at higher throttle, that comes out of the higher ISP you thought you would get from higher chamber pressure.

[IainMcClatchie]

I am amazed that the post-ignition quick disconnect for a hypergolic fluid is lighter weight than just having a larger tank on board.

I really love it when people are smarter than professionals like Tom Mueller.

Let me say it differently then.

Can someone point out the non obvious thing that makes having the post-ignition quick disconnect preferrable to a bigger tank?

For instance, deleting the quick disconnect means that, should they start the engines and then shut down on the pad, they probably can't try an engine restart without reconnecting, which requires draining the fuel tanks and so on.  Maybe that's the reason.

Your statement contained no information about rockets.

[Herb Schaltegger]

I am amazed that the post-ignition quick disconnect for a hypergolic fluid is lighter weight than just having a larger tank on board.

I really love it when people are smarter than professionals like Tom Mueller.

Let me say it differently then.

Can someone point out the non obvious thing that makes having the post-ignition quick disconnect preferrable to a bigger tank?

For instance, deleting the quick disconnect means that, should they start the engines and then shut down on the pad, they probably can't try an engine restart without reconnecting, which requires draining the fuel tanks and so on.  Maybe that's the reason.

Your statement contained no information about rockets.

The tail service masts don't release until the hold-downs do. And fluid QD's are very light, very easy to service/replace and when they leak, it's very obvious, neither of which are true for internal fluid tanks and lines.

(DISCLAIMER: I have spec'd/designed fluid QD for crewed spaceflight applications in life-critical operations; I'm not talking out of my ass).

[Perchlorate]

I am amazed that the post-ignition quick disconnect for a hypergolic fluid is lighter weight than just having a larger tank on board.

I really love it when people are smarter than professionals like Tom Mueller.

Let me say it differently then.

Can someone point out the non obvious thing that makes having the post-ignition quick disconnect preferrable to a bigger tank?

For instance, deleting the quick disconnect means that, should they start the engines and then shut down on the pad, they probably can't try an engine restart without reconnecting, which requires draining the fuel tanks and so on.  Maybe that's the reason.

Your statement contained no information about rockets.

The tail service masts don't release until the hold-downs do. And fluid QD's are very light, very easy to service/replace and when they leak, it's very obvious, neither of which are true for internal fluid tanks and lines.

(DISCLAIMER: I have spec'd/designed fluid QD for crewed spaceflight applications in life-critical operations; I'm not talking out of my ass).

Actually, Herb, I believe that qualifies more as a CLAIMER than a DISCLAIMER.   

[Jim]

I think there would still be some gain by shutting down center core engines to save fuel until after side boosters have done their job, but now I think all 9 engines would have to be shut down and re-started to make any significant difference.

There isn't