FAILURE: SpaceX Falcon I Launch III - August 2

[antonioe]

We're guilty of the same optimism: here's a 15-year old picture of Pegasus flight 4 (the USAF Alexis mission, launched April 25, 1993).  You can clearly see the "F-004" marking on the tail.  And here we are today, ready to launch Pegasus number... 40!  At this rate, around the year 2035 we will be glad we used three digits.

By the way, this is a good picture of the Pegasus operations at Dryden:  the spacecraft is being attached to the rocket under a class-10,000 clean tent (notice the two techs in bunny suits).  As you can see, it is a pretty spartan operation (the guy sitting in the chair is the obligatory safety inspector).

[just-nick]

...the guy sitting in the chair is the obligatory safety inspector...

What's he get paid, hourly?

[Antares]

With the expertise available I bet they are "kicking themselves" for missing this issue considering they know staging is a main problem for failure.  From the reading here it seems to be a "rookie" mistake to miss it or am I over simplifying the issue?

IMO, each of their failures has come from system interactions
1) material with the environment
2) software with propulsion with vehicle dynamics with sep system
3) flight control with propulsion with sep system

This is the result of an organization with a seriously lacking or non-existent systems engineering and integration organization.  IMEO, SpaceX has hired experts in all of the major single systems.  But, until they figure out some way to manage the interactions, they will have systems that each function perfectly alone without any guarantee of functioning well together.

A proper SE&I organization doesn't cause success.  It prevents failure.  (Though I suppose it can optimize too.)

[A_M_Swallow]

At some point SpaceX changed the fuel, that is a big hardware/requirements change.  Several years later the software was still trying to burn the original fuel.

Where the heck did you read that they changed the fuel?  It's been LOX/RP-1 for as long as they've been around.  Given that the quotes refer to mixture ratios, it simply sounds like an incorrect profile was loaded (in terms of when to run rich/lean).

I thought I remembered a change in fuel - it was the upper stage.

"... This Falcon 5 would be fitted with a liquid hydrogen second stage powered by one or more RL10 engines. "
http://www.geocities.com/launchreport/falcon9.html

[Comga]

I thought I remembered a change in fuel - it was the upper stage.

"... This Falcon 5 would be fitted with a liquid hydrogen second stage powered by one or more RL10 engines. "
http://www.geocities.com/launchreport/falcon9.html

This is incorrect, at least for the foreseeable future.
The Falcon 9 will use a single large expansion version of the same LOX/RP-1 engine as the nine engines in the first stage.

[joncz]

This is the result of an organization with a seriously lacking or non-existent systems engineering and integration organization.  IMEO, SpaceX has hired experts in all of the major single systems.  But, until they figure out some way to manage the interactions, they will have systems that each function perfectly alone without any guarantee of functioning well together.

A proper SE&I organization doesn't cause success.  It prevents failure.  (Though I suppose it can optimize too.)

This echoes for me the approach that a lot of .com companies evolved - no rigorous design, review, build methodology.  Instead they keep iterating on builds until they get something that sells.

That's easy to do when all you're expending are recycled electrons and time.  It's a lot harder to do when you have to bend metal.

I think Musk brings with him a lot of that seat of the pants baggage and bias from his .com background.

[Danderman]

This is the result of an organization with a seriously lacking or non-existent systems engineering and integration organization.  IMEO, SpaceX has hired experts in all of the major single systems.  But, until they figure out some way to manage the interactions, they will have systems that each function perfectly alone without any guarantee of functioning well together.

A proper SE&I organization doesn't cause success.  It prevents failure. 

My understanding is that the chief systems engineer for SpaceX is Elon. Now, this may simply be a misunderstanding, and he may have 50  systems engineers at SpaceX. However, the model that SpaceX seems to be using is not that different from that used to develop many successful launchers.

[marsavian]

This is the result of an organization with a seriously lacking or non-existent systems engineering and integration organization.  IMEO, SpaceX has hired experts in all of the major single systems.  But, until they figure out some way to manage the interactions, they will have systems that each function perfectly alone without any guarantee of functioning well together.

A proper SE&I organization doesn't cause success.  It prevents failure. 

My understanding is that the chief systems engineer for SpaceX is Elon. Now, this may simply be a misunderstanding, and he may have 50  systems engineers at SpaceX. However, the model that SpaceX seems to be using is not that different from that used to develop many successful launchers.

What is Elon's experience for this role ? There's the problem. If he doesn't mind learning on the job burning up rockets and tens of million of dollars that's fine but a few experienced professionals may have saved him the trouble.

[mikegi]

This echoes for me the approach that a lot of .com companies evolved - no rigorous design, review, build methodology.  Instead they keep iterating on builds until they get something that sells.

That's easy to do when all you're expending are recycled electrons and time.  It's a lot harder to do when you have to bend metal.

I think Musk brings with him a lot of that seat of the pants baggage and bias from his .com background.

There's a huge difference in producing software for general distribution versus one-shot usage. Rockets are one-shot usage and that requires getting it right the first time. That imposes *significant* constraints on the design and requires orders of magnitude more testing. You need many different sets of software designer eyeballs looking at the design so that any logic errors are discovered.

I remember when I went to Vancouver, BC to have laser vision correction (PRK) back in the mid 90s. As I was laying in the chair, the opthamologist was configuring the "burn" using Windows 3.1! I said nervously, "don't tell me you use Windows 3.1 to control the laser". He replied that it was simply functioning as a terminal to setup the laser's internal computer. I still considered bolting out of the office!

[Jim]

This is the result of an organization with a seriously lacking or non-existent systems engineering and integration organization.  IMEO, SpaceX has hired experts in all of the major single systems.  But, until they figure out some way to manage the interactions, they will have systems that each function perfectly alone without any guarantee of functioning well together.

A proper SE&I organization doesn't cause success.  It prevents failure. 

My understanding is that the chief systems engineer for SpaceX is Elon. Now, this may simply be a misunderstanding, and he may have 50  systems engineers at SpaceX. However, the model that SpaceX seems to be using is not that different from that used to develop many successful launchers.

He isn't the chief systems engineer.  He is the chief technology officer, whatever that means

[iamlucky13]

At stage separation would there be any atmospheric drag?  Could the 1st stage of slowed down as well as the 2nd stage holding speed or accelerating?   Hope this isn't a stupid question from watching to much NASCAR.

Minimal, and there should be as much or more drag on the larger, yet nearly empty (low density) first stage as there is on the upper stage, especially with the fairing still attached.

Actually, during flight 2 there was appreciable drag that caused the rocket to rotate, but the first stage still fell-clear of the 2nd prior to ignition.

[jongoff]

There's a huge difference in producing software for general distribution versus one-shot usage. Rockets are one-shot usage and that requires getting it right the first time. That imposes *significant* constraints on the design and requires orders of magnitude more testing. You need many different sets of software designer eyeballs looking at the design so that any logic errors are discovered.

Well, rocket vehicles don't *have* to be one-shot use.  For expendable launch vehicles, sure they do have to essentially function perfectly right out of the box.  But that isn't necessarily the only way of doing things--it may be possible to develop reusable rocket vehicles in an incremental fashion.  By doing that you can start with a subset of the overall problem, and work out issues in an incremental fashion, where at least a decent chunk of failures can be found in a non-catastrophic manner.  We're not positive it will work, but at least a few groups (Masten, Armadillo, XCOR, and Blue Origin) are trying that approach.  While we're mostly focusing on suborbital vehicles for now, there's nothing that is keeping a better-funded group from using this approach to attack orbital problems. 

But that isn't the approach they took, and the one they did take is a lot harder to make work.  I still think they'll get it right in the next attempt or two, but perfection isn't going to be easy.

~Jon

[Dalon]

But that isn't necessarily the only way of doing things--it may be possible to develop reusable rocket vehicles in an incremental fashion.  By doing that you can start with a subset of the overall problem, and work out issues in an incremental fashion, where at least a decent chunk of failures can be found in a non-catastrophic manner.  We're not positive it will work, but at least a few groups (Masten, Armadillo, XCOR, and Blue Origin) are trying that approach.  While we're mostly focusing on suborbital vehicles for now, there's nothing that is keeping a better-funded group from using this approach to attack orbital problems. 

But that isn't the approach they took, and the one they did take is a lot harder to make work.  I still think they'll get it right in the next attempt or two, but perfection isn't going to be easy.

~Jon

I agree that an incremental approach can solve a great many developmental issues.  The hard part about rockets is that some things cannot be easily tested and accurately tested.

SpaceX's failures on launch 2 and 3 were both only evidenced at the edge of the atmosphere with engines firing in near vacuum conditions.  While there are some ways of simulating this on the ground, my understanding is the available test beds are only able to accommodate 2nd stage engines, not larger first stage engines.

I don't see how Masten, Armadillo, XCOR, or Blue Origin's incremental approach will make them better able to test for high-speed, near vacuum operations.   Of course, this isn't currently an issue for them as none of their initial products will be orbital. 

If and when any of those companies make the leap to orbital, I expect they'll have to jump in the fire and hope for the best, just like everyone else.

[guru]

I don't see how Masten, Armadillo, XCOR, or Blue Origin's incremental approach will make them better able to test for high-speed, near vacuum operations.   Of course, this isn't currently an issue for them as none of their initial products will be orbital. 

If and when any of those companies make the leap to orbital, I expect they'll have to jump in the fire and hope for the best, just like everyone else.

Near vacuum operation testing isn't so hard.  The engines on most of those companies' vehicles are small enough that they could be tested on altitude pressure test stands like the E-6 stand at PWRs West Palm Beach facility, or the vacuum test chamber at NASA's Plum Brook facility in Ohio.  I'm sure there are other facilities and additional options as well (like building their own diffuser and steam generators).

High speed is pretty hard to test, though.

[Dalon]

Near vacuum operation testing isn't so hard.  The engines on most of those companies' vehicles are small enough that they could be tested on altitude pressure test stands like the E-6 stand at PWRs West Palm Beach facility, or the vacuum test chamber at NASA's Plum Brook facility in Ohio.  I'm sure there are other facilities and additional options as well (like building their own diffuser and steam generators).

High speed is pretty hard to test, though.

I completely agree, their current products could certainly be tested in near vacuum conditions.  The problem arises when and if they scale up for an orbital vehicle. 

If any of those companies ever deploy an orbital vehicle, it will certainly have more or larger engines, perhaps Much larger engines.  This could certainly put them in the same boat as SpaceX and the rest.  Like the rest, they would have a massive single point of failure for which they would be unable to definitively test prior to launch.

Do you know the limits of those facilities?  One would think that SpaceX's Merlin is too large to be tested at such a facility for the simple reason that SpaceX never made use of those facilities. 

[William Barton]

Is there a universally (or at least widely) accepted compendium of "everything that can possibly go wrong with a rocket engine?" This is not a new technology. Regenerative liquid fuel rocket engines have been being designed, developed, and flown since the says of the Hindenberg.

[jongoff]

SpaceX's failures on launch 2 and 3 were both only evidenced at the edge of the atmosphere with engines firing in near vacuum conditions.  While there are some ways of simulating this on the ground, my understanding is the available test beds are only able to accommodate 2nd stage engines, not larger first stage engines.

First off, this depends a lot on the scale of engine you're talking about.  For instance, there's a company whose test site is about 800yds from our test site that has a high-altitude testing rig good to over 24klbf.  That's not too shabby.  Armadillo, XCOR, and Masten aren't planning on doing stuff bigger than that anytime soon.

More importantly, you're incorrect about the testing failures.  The first failure was due to an incorrect set of engine code leading to staging at an unexpectedly low altitude.  In an RLV, if the vehicle was acting that far off-nominally, you'd abort the flight.  Early.  On the third flight failure mode, if shutdown transients were a concern, you'd fly a flight or two where you intentionally don't separate the upper stage, but do an attached-stage abort after measuring in-situ the shutdown transient.

I don't see how Masten, Armadillo, XCOR, or Blue Origin's incremental approach will make them better able to test for high-speed, near vacuum operations.   Of course, this isn't currently an issue for them as none of their initial products will be orbital.

While it's true that most of us are targetting suborbital operations first (Blue Origin has been a bit more vague on what its actual plan is), most of us are intending to offer nanosat launch services, where our suborbital vehicle launches a 1-2 stage nanosat launcher sometime after MECO.  We'll need to know those things for that situation, but as I said, there are ways of getting that information that doesn't put much risk on your vehicle.

If and when any of those companies make the leap to orbital, I expect they'll have to jump in the fire and hope for the best, just like everyone else.

If someone's designing a fully-reusable orbital LV and makes it so it can only be tested using full-up testing, with no option for incremental testing or envelope expansion....well, they're being retarded. 

Full-up testing for ELVs may make sense, but for RLVs?  Why?

~Jon

[jongoff]

Near vacuum operation testing isn't so hard.  The engines on most of those companies' vehicles are small enough that they could be tested on altitude pressure test stands like the E-6 stand at PWRs West Palm Beach facility, or the vacuum test chamber at NASA's Plum Brook facility in Ohio.  I'm sure there are other facilities and additional options as well (like building their own diffuser and steam generators).

High speed is pretty hard to test, though.

I completely agree, their current products could certainly be tested in near vacuum conditions.  The problem arises when and if they scale up for an orbital vehicle. 

If any of those companies ever deploy an orbital vehicle, it will certainly have more or larger engines, perhaps Much larger engines.  This could certainly put them in the same boat as SpaceX and the rest.  Like the rest, they would have a massive single point of failure for which they would be unable to definitively test prior to launch.

Do you know the limits of those facilities?  One would think that SpaceX's Merlin is too large to be tested at such a facility for the simple reason that SpaceX never made use of those facilities. 

Dalon,

Not to be too offensive, but you're illustrating that you either don't know what you're talking about, or haven't thought this through very well.  There are plenty of ways of getting the data you're talking about incrementally without having to have gargantuan vacuum test stands or rolling-the-dice.  That dichotomy exists only in your mind.

~Jon

[edkyle99]

I thought I remembered a change in fuel - it was the upper stage.

"... This Falcon 5 would be fitted with a liquid hydrogen second stage powered by one or more RL10 engines. "
http://www.geocities.com/launchreport/falcon9.html

This is incorrect, at least for the foreseeable future.
The Falcon 9 will use a single large expansion version of the same LOX/RP-1 engine as the nine engines in the first stage.

This is *not* incorrect.  Read the thread.  The discussion was about the initial Falcon 5 (not Falcon 9) designs.  Aviation Week reported at the time (2003-4) that SpaceX had contemplated a future RL10-powered Falcon 5 upgrade.  Here's a link.   

http://www.aviationweek.com/aw/generic/story_generic.jsp?channel=awst&id=news/03294top.xml
 - Ed Kyle

[dunderwood]

The assertion that a 'fuel change' had something to do with the software glitch on flight 2 is still incorrect.  Changes in the upper stage of F9 don't have anything to do with upper stage of F1-002.