FAILURE: SpaceX Falcon I Launch III - August 2

[kkattula]

...  For most commercial players or government payloads the payload costs override the launch costs.  Those comm sats and probes cost a lot of money to design and build and operate, often much more money over their lifetime than the launchers with launch services combined.  It's a huge, often one-of-a-kind investment for the customer.  There is a premium placed on reliability, not as much on the lower launch costs.  (Hence insurance costs are on the payload)  They'd rather pay more for a launcher with some kind of track record.  ...

Have you considered the possibility that comm sats and probes cost a lot of money BECAUSE launches cost so much, and have long lead times?

If I have to pay $100m+ for a launch, 2 or 3 years in advance, I want to be damn sure my comm sat will work perfectly for 10 years. Even if it costs $1b.

On the other hand, if launches were cheap & regular, I'd fly lots of cheap sats, adding capacity or replacing defective units as required. Not to mention constatntly upgrqading technology. I'd probably even maintain spare capacity on orbit.

Making lots of sats & launchers would probaby drop the prices even further.

[pippin]

As it turns out, Elon Musk has just commented on the recent issues.  His explanations clearly squares with my take on events.  He states that the latest failure was not a QA issue, it was a design issue.  A hurdle he says can be quickly resolved, allowing them to move forward.

Fortunately, Elon Musk is a guy who acts smarter than he talks.
He made very similar statements after the first two failures and it took them over a year to launch again after those "quickly to resolve" issues. Expect the same to happen here.

The big argument here is not that SpaceX is institutionally incapable or that 3 failures in a row vs. 2 in a row doom you but that the impression between flight two and three is that they haven't made much progress from a quality POV while claiming that all those costly QA procedures that make spaceflight expensive are not necessary due to their reliable design. No that exact design shows to be unreliable. That makes the original assumption look pretty bad since so far there is absolutely NO evidence I can see that they will be able to be reliable without strict QA.
I haven't worked in spaceflight myself, only aeronautics, defense and automotive, but there QA on complex systems absolutely kills you cost wise. You can get a long way by establishing simple designs but everybody tries to do that and for some problems there are no simple solutions.

Maybe the biggest issue is their talk: Hey, everything's easy, no problems, we lost a flight, not a big issue, dumb error, won't happen again. Oh, you lost a payload? Sorry for that...

Their claim was: 10x more reliable, 10x cheaper. I simply don't believe it.

[Analyst]

1a) You and others here continue to suggest that "Three Failures" is some sort of magic number that damns SpaceX as a company with huge institutional problems.  While it's not good to have three failures, neither is it good to have one, or two.
1b) Yet each of the major players in the industry have suffered far more than three failures during their time in the business. 

2) As I said above, I believe SpaceX is working through these show-stoppers one by one.  Once they have cleared each of these hurdles, they will achieve success and reliability.

3) We must also consider the fact that SpaceX is competing with companies that have been in the business for 20, or 30, or 40 years.  The Big Rocket companies have certainly accrued institutional knowledge completely irrespective of their large and costly Big Company Cost Structure. 

SpaceX is a company doing everything for the first time.  With or without employing the Big Company Cost Structure, they were bound to stumble over problems that their competition had solved years or decades ago. 

4) While I will concede that my opinion of SpaceX merely working through design issues is not backed by a tremendous abundance of evidence, your allegation that SpaceX's failures have been caused by a lack of the Big Company Cost Structure and/or institutional problems is an opinion without Any evidence at all.

5) As it turns out, Elon Musk has just commented on the recent issues.  His explanations clearly squares with my take on events.  He states that the latest failure was not a QA issue, it was a design issue.  A hurdle he says can be quickly resolved, allowing them to move forward.

Three launches, no QA issues, no technical defects, ALL design issues.

Looks like I was right, go figure.

"And this doesn't appear to be a quality issue or a manufacturing issue. It's a design issue related to new hardware that has only flown on this flight."

1a) I said three failures in a row. No successful flight out of three is very uncommon these days.
1b) Having three failures out of 50 flights wouldn't be.

2) Sure they will. But how long and how many failures will it take? And what does it cost and what will be the impact on the price? Other did and do way better.

3) Its the other way arround: The big companies have the cost structure they have because institutional procedures turned out to be needed to achieve constant mission success. SpaceX does repeat failures others did 50 years ago. They didn't learn from these lessons. They were and still are kinda arrogant about the procedures coming from these lessons.

4) Three failures in a row say different. With the right and known (dismissed) procedures in place, some (probably not all) problems would not have shown up.

5) Musk has been wrong before, many times. This is no evidence, its company talk.

"Quickly resolved": With this attitude, the next failures looms behind. This business is never easy nor quick.

Your different reasons for the failures don't matter: Design comes first. If you have design issues in the first place, technical or QA issues are not even needed for a failure. Design issues are the worst, they stem from the (wrong) thinking, not the (wrong) implementation.

Analyst

[Mighty-T]

Three launches, no QA issues, no technical defects, ALL design issues.

I think the approach chosen by SpaceX is very much the one quoted by Elon in his latest interview: Falcon 1 is a technology testbed; reaching orbit and services to the customers are lower ranking motivations.
Of course it is interesting to see, how the new Merlin 1C and Kestrel 2 perform on a launcher, however, if they were determined to go to orbit asap, they should have stayed with the initial configuration and get that system to work first.
What we see is that they fix one design issue and in the mean time create another. How can anyone expect to quickly succeed with that strategy.

[hop]

While it's not good to have three failures, neither is it good to have one, or two.  Yet each of the major players in the industry have suffered far more than three failures during their time in the business. 

So ? It's still a 100% failure rate vs. a 2%-4% failure rate. Sure, it could just be a run of bad luck, but it's suggestive.

Three failures in a row doesn't prove that spacex is a fundamentally broken organization, but it definitely suggests they have a problem.

There's more to it than simple statistics:
Historically, two in a row is enough to trigger a major step back to the drawing board and a concerted drive to fix not just the specific problem, but the organizational problems that led to the problem going undetected. Many struggling programs have been turned around this way (often with heavy outside oversight).

If you are correct that it was all "design issues" that suggests that their design validation is among the worst in the industry.

You seem to think this is better than losing rockets to QA problems, but I'm not sure why. If their validation is that bad, design flaws are going to keep popping up for a long time.

Just because something gets to orbit once doesn't mean the design is OK. See STS, or the recent Briz-M failure, or any of dozens of other failures caused by something marginal in an apparently mature design finally going bust.

[DmitryP]

Now - the email was sent to all his employees. All employees have access to the telemetry, most in read-only but they can all see it. He specifically stated that only HR and Finance are behind the firewall. He deliberately created a completely open work environment. It would be super easy for any one of them to look at it to see if Elon was telling them the truth.

Excuse me, is it possible to find a source of statement that all employees at SpaceX have access to the telemetry. Could not find it through search...

[William Barton]

From Gary, above:
"Changing things on a test launch vehicle that's already had two failures also strikes me as a bad idea. Surely you test the fix then change only what needs to be changed to fix the problem and only after a successful flight do you add enhancements."

I tend to agree with the sentiment. but the fact is, they made massive changes between Falcon 1 #2 and #3, up to and including switching from a successful ablative engine to an unflown regenerative engine. On the one hand, I am not inspired to confidence by what I'm seeing (the very hesitant launch campaign this time around was unnerving, plus Musk's tour of his factory made him look goofy--if someone gave me a few hundred million dollars, and I did the same thing, I'd probably come off the same way, right down to the rumpled polo shirt). On the other hand, _someone_ at SpaceX sure knows what they're doing. The development of the Merlin engine is no mean feat. When was the last time an organization did that without it being a government-funded project. Has it been done since the Verein fur Raumschiffarht did it in the 1930s? This makes me thin it's a top-down management issue.

From the Wired interview:
"Musk: Patience is a virtue, and I'm learning patience. It's a tough lesson."

I know exactly what he means, and maybe even what lesson he needs to learn. I'm more than 20 years older than he is, and I had to learn patience the hard way too. When I'm engaged in a high-pressure project and things start to go wrong, I have learned not to bull ahead. I have learned to stop work, take a deep breath, and step back for a project overview, because I know the mistake isn't with my project team (who are doing tings my way, if they're on my team). The problem is always with _me_, and I need to get outside myself to see what the hell I'm doing wrong.

This might be a good time for him to hunt down the world's best systems engineer and see if he or she will take a seven-figure salary. It's important, because if SpaceX fails (even though it may only be due to top-level mismanagement), it will be taken as evidence that what they're trying to do is "impossible," proof that only the government-based, taxpayer funded large-budget development paradign will work. That would be a bad thing.

[marsavian]

The design area where SpaceX appear to be weak at is rocket flight dynamics as opposed to rocket engine design where they seem very strong. Considering Musk was the de facto Chief Designer from basically a pure Physics background I don't believe he's fully aware of what level of personnel they are lacking in until things actually go wrong. Combine extreme confidence with little experience and 0 for 3 is not too hard to envisage.

In his defense though Falcon 1 flight 3 was really Falcon 1c flight 1 as flight 2 was really Falcon1b flight 1 so in effect a new rocket has been tested each time. Also he has tremendous drive, commitment (both mental and financial) and is a quick learner which he will all need. The next Falcon 1 flight will probably work but I expect the first and maybe second flight of Falcon 9 to fail due to some interaction they have not designed for. Fun and exciting to watch though compared to more reliable launchers .

[mr.columbus]

1. Considering Musk was the de facto Chief Designer from basically a pure Physics background I don't believe he's fully aware of what level of personnel they are lacking in until things actually go wrong.

2. In his defense though Falcon 1 flight 3 was really Falcon 1c flight 1 as flight 2 was really Falcon1b flight 1 so in effect a new rocket has been tested each time.

3. The next Falcon 1 flight will probably work but I expect the first and maybe second flight of Falcon 9 to fail due to some interaction they have not designed for.

ad 1. Musk is not the "Chief Designer" at SpaceX. He is the company's CEO. He has experienced people who know what they are doing. The problem seems to be that he is told were shortcomings in personnel are, and he sometimes just dismisses these deficits.

ad 2. If we apply that logic to other rockets, there bascially wouldn't be any rocket at all with more than one failure. If a failure of a launch vehicle occurs, the problems are analysed and in most instances there are remedies taken that the next launch will be ok. The reality is, small design changes, fixes or new engines don't change Falcon 1's overall design. The rocket failed 3 times. That is a fact.

ad 3. From all we know currently, Falcon 1 is the most unreliable operational vehicle there currently is. That the next flight works is by no means certain. Regarding Falcon 9, we don't have any flight history to base our assumptions on. It could work flawlessly in a dozen launches or could fail 3 times in a row just as Falcon 1 did.

[marsavian]

You are wrong.

http://www.pbs.org/thinktank/transcript1292.html

WATTENBERG: So, now we’re on this most remarkable factory floor of SpaceX. I felt I’ve been around. I’ve never quite seen anything like this. What are you doing here?
MUSK: SpaceX is developing markets for taking satellites and people to orbit and beyond. So, we’ve finished development of and done a few test launches in our small rocket, which is the Falcon One, which you see part of over here.
And we have in development a big rocket which is the Falcon 9. And that’s intended to service the space station, as well as deliver very large satellites to orbit.
WATTENBERG: Did you design these or are they your concept?
MUSK: Yes, I’m the chief designer in the company.
WATTENBERG: -- designer and the businessman.
MUSK: Yes.
WATTENBERG: A good combination.
MUSK: Yeah, I think it is a good combination.

Also the regen engine was not a design failure fix.

On a related note ...

SPACEX RECEIVES $20 MILLION INVESTMENT FROM FOUNDER’S FUND

http://spacex.com/press.php?page=47

[rkoenn]

The design area where SpaceX appear to be weak at is rocket flight dynamics as opposed to rocket engine design where they seem very strong. Considering Musk was the de facto Chief Designer from basically a pure Physics background I don't believe he's fully aware of what level of personnel they are lacking in until things actually go wrong. Combine extreme confidence with little experience and 0 for 3 is not too hard to envisage.
.

The obvious to note here is that with the engines he can do sufficient testing on the stand to validate them and correcting any problems is much easier on the ground.  You can simulate and analyze all you want but ultimately you have to fly the rocket and that is where unknowns will manifest themselves.  Unfortunately that is costly and the community watches much more closely than they do for an engine test.

Also as far a chief designer goes, Elon may have conceptualized the design but I truly doubt he did much more than that as far as in depth engineering.  He may have been the person making certain design decisions when various options were available but if he did not follow the recommendations of the expert/s he has working for him on those decisions then he is not too good of a project manager either.  Elon is somewhat arrogant from what I know of him and likes the glory that he gets, but he needs to be realistic and possibly a bit humble at this point in time.

[mr.columbus]

You are wrong.

http://www.pbs.org/thinktank/transcript1292.html

WATTENBERG: So, now we’re on this most remarkable factory floor of SpaceX. I felt I’ve been around. I’ve never quite seen anything like this. What are you doing here?
MUSK: SpaceX is developing markets for taking satellites and people to orbit and beyond. So, we’ve finished development of and done a few test launches in our small rocket, which is the Falcon One, which you see part of over here.
And we have in development a big rocket which is the Falcon 9. And that’s intended to service the space station, as well as deliver very large satellites to orbit.
WATTENBERG: Did you design these or are they your concept?
MUSK: Yes, I’m the chief designer in the company.
WATTENBERG: -- designer and the businessman.
MUSK: Yes.
WATTENBERG: A good combination.
MUSK: Yeah, I think it is a good combination.

Also the regen engine was not a design failure fix.

On a related note ...

SPACEX RECEIVES $20 MILLION INVESTMENT FROM FOUNDER’S FUND

http://spacex.com/press.php?page=47

What Musk says and the reality are two different things. The Falcon 1 and 9 rockets are designed by a large number of people. He may be involved in the process, but he is not the one in charge of the overall design or specific technical issues.

[JesseD]

You are wrong.

http://www.pbs.org/thinktank/transcript1292.html

WATTENBERG: So, now we’re on this most remarkable factory floor of SpaceX. I felt I’ve been around. I’ve never quite seen anything like this. What are you doing here?
MUSK: SpaceX is developing markets for taking satellites and people to orbit and beyond. So, we’ve finished development of and done a few test launches in our small rocket, which is the Falcon One, which you see part of over here.
And we have in development a big rocket which is the Falcon 9. And that’s intended to service the space station, as well as deliver very large satellites to orbit.
WATTENBERG: Did you design these or are they your concept?
MUSK: Yes, I’m the chief designer in the company.
WATTENBERG: -- designer and the businessman.
MUSK: Yes.
WATTENBERG: A good combination.
MUSK: Yeah, I think it is a good combination.

Also the regen engine was not a design failure fix.

On a related note ...

SPACEX RECEIVES $20 MILLION INVESTMENT FROM FOUNDER’S FUND

http://spacex.com/press.php?page=47

What Musk says and the reality are two different things. The Falcon 1 and 9 rockets are designed by a large number of people. He may be involved in the process, but he is not the one in charge of the overall design or specific technical issues.

more than likely, 'chief engineer' means 'head of the engineering dept'.  which may mean that the team builds it, and then says, 'here's the rocket, okay?' and he just has to sign off on it.

One interesting thought that just occurred to me - having one guy as the CEO, head engineer, chief investor, etc., is taking a LOT of pressure off his people and putting it all on him.  He is personally setting himself up as the target/whipping boy, and letting his people do the nitty-gritty nuts-and-bolts work.

[marsavian]

No guys, he really IS the Chief Engineer


http://beyond438.wordpress.com/2007/08/29/elon-musk-of-spacex/

Musk serves as chief engineer at SpaceX and designed much of the Falcon himself.


http://www.inc.com/magazine/20071201/entrepreneur-of-the-year-elon-musk_Printer_Friendly.html

and its CEO, a man with no prior experience with rockets, doubles as chief engineer.


http://www.npr.org/templates/story/story.php?storyId=12484430

In 2002, Musk started Space Exploration Technologies (SpaceX). He hired engineers from Boeing, TRW, Lockheed, and the U.S. Air Force. He couldn't lure a chief engineer, so the person doing that job is him.


http://www.globalsecurity.org/org/news/2003/030422-space01.htm

Musk, who describes himself as "essentially chief engineer of the rocket," said he spends his typical day delving into production details with his team.

[synchrotron]

If I have to pay $100m+ for a launch, 2 or 3 years in advance, I want to be damn sure my comm sat will work perfectly for 10 years. Even if it costs $1b.

On the other hand, if launches were cheap & regular, I'd fly lots of cheap sats, adding capacity or replacing defective units as required. Not to mention constatntly upgrqading technology. I'd probably even maintain spare capacity on orbit.

No sir.  Leaving dead comm sats in valuable GEO slots is a very bad thing.  You want to leave enough margin at EOL to graveyard up a couple hundred kilometers.  Leaving your cheap junk lying around will kill your own business model and that of others who want to use those slots.

[jongoff]

Well what do I know.  Strangely enough the customer in question SpaceDev is thankful to SpaceX for "delivering" their sat which was deemed a success:  http://spacedev.com/press_more_info.php?id=278  without actualy being delivered and doing its thing.  If I had customers like that...

Well, you have to remember what the purpose of that satellite was (as opposed to the secondary payloads).   It was an ORS launch.  Their main purpose was to show that you could on short notice integrate a satellite and launch it quickly.  So, in a way, in this case, just by getting off the pad on such short notice, they fulfilled at least one of the key goals of that payload.  That's not a normal situation though, and I'm sure the secondaries didn't appreciate a fishing orbit.

Interestingly enough, there were 3 ORS satellites from which the SpaceDev one was selected.  So, there's a non-zero chance that one of those other satellites (which likely wouldn't have had a flight opportunity otherwise) might be on F1 Take-4 or F1 Take-5. 

~Jon

[jongoff]

Three launches, no QA issues, no technical defects, ALL design issues.

Dalon, I'm not saying this as a detractor (I'm on fairly good terms with several members of the SpaceX team), but launch number 2 was definitely a QA issue.  The wrong code was loaded into the engine controller (probably an outdated lookup table like a throttle map).  The incorrect code led to lower thrust, lower staging, more aerodynamic forces at staging, and the 2nd stage nozzle hit that initiated the slosh.  If the payload on that flight had been a full, max capacity payload, then even if the slosh hadn't happened, the payload wouldn't have made it to orbit.  That was a QA issue.

The first was definitely a design issue.  As for the third?  We'll see. 

Elon's got a pretty solid team there, and fortunately has enough money to see things through.  For me the question isn't if he'll be able to get Falcon 1 to fly.  For me the question is, once they have that first successful flight, will they be able to consistently duplicate it.  I hope so.

~Jon

[aero313]

As it turns out, Elon Musk has just commented on the recent issues.  His explanations clearly squares with my take on events.  He states that the latest failure was not a QA issue, it was a design issue.  A hurdle he says can be quickly resolved, allowing them to move forward.

Three launches, no QA issues, no technical defects, ALL design issues.

Looks like I was right, go figure.

Elon also said the design was fully qualified after the second, "successful" mission.

Go figure.

[Chris Bergin]

Ewwww, I know Wired like to be all "blog trendy" with their interview style, but that line of questioning seemed a bit patronizing. Just a personal take.

[synchrotron]

Well what do I know.  Strangely enough the customer in question SpaceDev is thankful to SpaceX for "delivering" their sat which was deemed a success:  http://spacedev.com/press_more_info.php?id=278  without actualy being delivered and doing its thing.  If I had customers like that...

Well, you have to remember what the purpose of that satellite was (as opposed to the secondary payloads).   It was an ORS launch.  Their main purpose was to show that you could on short notice integrate a satellite and launch it quickly.  So, in a way, in this case, just by getting off the pad on such short notice, they fulfilled at least one of the key goals of that payload.

Poor demonstration of short notice integration.  Yeah, the test plugs say everything is o.k. etc.; that could all be done just as well in the integration facility.  Integrating ballast on short notice would have been just as useful.