SpaceX Systems Engineering

[Kabloona]

I just stumbled across this SpaceX Systems Engineering presentation and found it interesting as a former systems engineer myself.

https://www.aiaa.org/uploadedFiles/Events/Conferences/2012_Conferences/2012-Complex-Aerospace-Systems-Exchange-Event/Detailed_Program/CASE2012_2-4_Muratore_presentation.pdf

One interesting bit is the chart that shows the flow of qual and acceptance testing. There's a box labeled "structures component acceptance testing." And a bullet on page 16 says "All hardware acceptance tested." But we know from the CRS-7 failure investigation that the helium tank struts were not acceptance (load) tested.

And not acceptance testing the struts ran counter to their philosophy of extensive testing. So it's interesting that not adhering to their own philosophy in the case of the struts came back to bite them.

[Dante80]

Nice presentation, thanks for sharing it.

A question. How accurate are the below cost numbers?

[Kabloona]

I have no insight into those numbers, but a post on parabolicarc says NASA has verified the Falcon 1 and Falcon 9 development cost numbers:

SpaceX has publicly indicated that the development cost for Falcon 9 launch vehicle was approximately $300 million. Additionally, approximately $90 million was spent developing the Falcon 1 launch vehicle which did contribute to some extent to the Falcon 9, for a total of $390 million. NASA has verified these costs.

http://www.parabolicarc.com/2011/05/31/nasa-analysis-falcon-9-cheaper-traditional-approach/

And according to that same article, NASA's own analysis said it would have cost them between $1.7B and $4.0B to develop F9. So SpaceX beat the NASA development cost estimate by at least a factor of 5, and possibly a factor of 10+.

Extremely impressive.

[mfck]

Very interesting presentation. Thanks for posting it.


Looking still at the price numbers in that screenshot, would it be safe to assume an F9 pad is an order of magnitude more, i.e. in the low hundreds of $M and that subsequent F9 integration hangars are at least half the price?

Also, what would be the educated guess about how much that $390M has grown since the presentation - F9 1.1, F9 FT, etc. and what would be the cost of the CRS-7 investigation and direct changes as consequence (not the costs associated with not flying)?

[guckyfan]

Very interesting presentation. Thanks for posting it.


Looking still at the price numbers in that screenshot, would it be safe to assume an F9 pad is an order of magnitude more, i.e. in the low hundreds of $M and that subsequent F9 integration hangars are at least half the price?

The quoted cost for the Brownsville site are just short of 100 million $. That's for a FH launch pad, TE, HIF, control center and satellite integration facility with handling of hypergolic fuels.

[CyndyC]

I just stumbled across this SpaceX Systems Engineering presentation and found it interesting as a former systems engineer myself.

https://www.aiaa.org/uploadedFiles/Events/Conferences/2012_Conferences/2012-Complex-Aerospace-Systems-Exchange-Event/Detailed_Program/CASE2012_2-4_Muratore_presentation.pdf

One interesting bit is the chart that shows the flow of qual and acceptance testing. There's a box labeled "structures component acceptance testing." And a bullet on page 16 says "All hardware acceptance tested." But we know from the CRS-7 failure investigation that the helium tank struts were not acceptance (load) tested.

And not acceptance testing the struts ran counter to their philosophy of extensive testing. So it's interesting that not adhering to their own philosophy in the case of the struts came back to bite them.

Your experience with component testing must have been under an alternate definition. According to definitions for "acceptance testing" online, the general and most common usage, and probably the way SpaceX meant it, refers to testing by the manufacturer prior to delivery to the customer. Labautopaedia.org goes on to point out that a substantial portion of a customer's payment goes toward the manufacturer's testing, and that portion of the payment is not due & paid unless and until the testing is successful.

The discussion on the subject of product testing was one of the long ones in post CRS-7 threads, so I don't want restart too much of it by responding, but wherever the previous discussion left off, I don't think the supplier was ever identified. I don't think much more can be said w/o knowing that identity, and how reliable the company could have or should have been considered. As far as manufacturing techniques alone were to blame, I didn't have to go any further than my 100th frozen Tuscan Chicken casserole, which had every ingredient but the chicken. SOMEbody kept the production line going just a touch too long.

[Kabloona]

I just stumbled across this SpaceX Systems Engineering presentation and found it interesting as a former systems engineer myself.

https://www.aiaa.org/uploadedFiles/Events/Conferences/2012_Conferences/2012-Complex-Aerospace-Systems-Exchange-Event/Detailed_Program/CASE2012_2-4_Muratore_presentation.pdf

One interesting bit is the chart that shows the flow of qual and acceptance testing. There's a box labeled "structures component acceptance testing." And a bullet on page 16 says "All hardware acceptance tested." But we know from the CRS-7 failure investigation that the helium tank struts were not acceptance (load) tested.

And not acceptance testing the struts ran counter to their philosophy of extensive testing. So it's interesting that not adhering to their own philosophy in the case of the struts came back to bite them.

Your experience with component testing must have been under an alternate definition. According to definitions for "acceptance testing" online, the general and most common usage, and probably the way SpaceX meant it, refers to testing by the manufacturer prior to delivery to the customer. Labautopaedia.org goes on to point out that a substantial portion of a customer's payment goes toward the manufacturer's testing, and that portion of the payment is not due & paid unless and until the testing is successful.

I'm quite familiar with common aerospace practice for component subcontracts and acceptance testing, and I'm trying but failing to understand why you think my post somehow contradicts what you wrote above.

When I said "not adhering to their own philosophy," what I meant was that apparently they did not require load testing by the vendor as part of the acceptance criteria. It's up to to buyer to specify what their acceptance criteria are, not the vendor. So by not writing a load testing acceptance requirement into the procurement contract, SpaceX did not adhere to their "test what you fly" philosophy.

So I think we are in agreement. 

[CyndyC]

One interesting bit is the chart that shows the flow of qual and acceptance testing. There's a box labeled "structures component acceptance testing." And a bullet on page 16 says "All hardware acceptance tested." But we know from the CRS-7 failure investigation that the helium tank struts were not acceptance (load) tested.

And not acceptance testing the struts ran counter to their philosophy of extensive testing. So it's interesting that not adhering to their own philosophy in the case of the struts came back to bite them.

Your experience with component testing must have been under an alternate definition. According to definitions for "acceptance testing" online, the general and most common usage, and probably the way SpaceX meant it, refers to testing by the manufacturer prior to delivery to the customer. Labautopaedia.org goes on to point out that a substantial portion of a customer's payment goes toward the manufacturer's testing, and that portion of the payment is not due & paid unless and until the testing is successful.

I'm quite familiar with common aerospace practice for component subcontracts and acceptance testing, and I'm trying but failing to understand why you think my post somehow contradicts what you wrote above.

When I said "not adhering to their own philosophy," what I meant was that apparently they did not require load testing by the vendor as part of the acceptance criteria. It's up to to buyer to specify what their acceptance criteria are, not the vendor. So by not writing a load testing acceptance requirement into the procurement contract, SpaceX did not adhere to their "test what you fly" philosophy.

So I think we are in agreement. 

You are in more of a rank & position to call SpaceX on this subject than I am, so I don't want to try saying I completely agree, at least not without more information. That's enlightening that the buyer has to require and set the testing criteria, but by saying "apparently", it sounds like you are only assuming SpaceX didn't require any load testing at all. Someone in the prior discussions brought up "batch testing" in manufacturing, so although that person was only assuming too, at least that much testing might have been required by SX & then done by the manufacturer. Maybe not though. BTW, isn't turning an argument around to convince the opposite gender has agreed with you supposed to be my line?

[Coastal Ron]

That's enlightening that the buyer has to require and set the testing criteria, but by saying "apparently", it sounds like you are only assuming SpaceX didn't require any load testing at all. Someone in the prior discussions brought up "batch testing" in manufacturing, so although that person was only assuming too, at least that much testing might have been required by SX & then done by the manufacturer. Maybe not though.

SpaceX, and Musk, have stated that the strut was supposed to be "certified" by the supplier.  Keeping in mind that the minimum strength of the strut was supposed to be a breaking point of 10,000 lbs of force, it normally would be up to the supplier to figure out how to ensure that 100% of the product that they supply would not fail below 10,000 lbs of force.  The parts may actually be designed for a strength far higher than 10,000 lbs of force, and the supplier may have calculated that it was not possible to manufacture finished parts that failed below 10,000 lbs of force.  If so then obviously they were wrong.

And that there was a certification process tells me that SpaceX was paying to ensure that 100% of the parts they received would not fail below 10,000 lbs of force.  SpaceX supplier QA would know that "batch testing" does not ensure 100% good parts, so we can rule that out.  And since there were multiple parts that SpaceX found that failed internal SpaceX testing post-failure, that tells me that the supplier was not being forthcoming with SpaceX about their testing program.  Read into that whatever you want.

I've experience enough production stoppages because of supplier issues to understand how difficult it is to ensure 100% reliability from suppliers.  And interestingly reusability is really the best way for SpaceX to find out how well their quality is in-house, since you don't know how close to failure you really are until you can examine something after it's been used as intended.  And I say that because we'll never really know if it was a part failure, or an assembly failure that led to the CRS-7 accident - all we know are that sub-standard parts were found in SpaceX inventory that could have caused the accident.  So reusability will strengthen their internal quality.

[Kabloona]

That's enlightening that the buyer has to require and set the testing criteria, but by saying "apparently", it sounds like you are only assuming SpaceX didn't require any load testing at all.

We're having to read between the lines a bit. What Elon said was that (1) they did not load test *every* strut produced, and (2) they relied instead on materials certifications (ie from the raw materials supplier, not the strut manufacturer). He also said that they had designed the strut with a large safety factor (I don't remember the number, but it was something like a factor of 5, which on a rocket is a huge factor (for man-rated systems, it's usually 1.4)

In my experience, what that means is:

-SpaceX designed the strut and had the vendor qualify the design by testing a bunch of struts to the breaking point, thereby verifying they had a lot of margin.

-As a result, SpaceX probably felt comfortable not requiring 100% acceptance load testing of every strut, but instead relied on a materials certification from the raw material supplier that showed the metallurgical content of each batch of steel along with ultimate tensile strength test samples from each batch.

Whether or not SpaceX required the vendor to do some random load testing from each batch is unknown and not really relevant anyway, because as they learned, one flaw in one casting can ruin your whole rocket. That's why you do 100% load testing of all structure. I'm quite surprised by this lapse in their QA criteria, but no doubt they won't make that mistake again.

BTW, isn't turning an argument around to convince the opposite gender has agreed with you supposed to be my line?

Turnabout is fair play. 

SpaceX, and Musk, have stated that the strut was supposed to be "certified" by the supplier.

No, you misunderstand what a "materials certification" is. It's a piece of paper from the raw materials vendor that the strut maker bought the steel bar stock (or whatever) from. It gives traceability that shows where the raw steel came from, its metallurgical properties, ultimate strength, etc. its basically a "birth certificate" that shows where the raw materials came from and proves they have "good genes."

The vendor then machines the raw steel into a finished product and hands the materials certification over to the buyer (SpaceX) to prove that the raw materials that the part is made of were good.

A materials certification is *not* a certification by the vendor that their strut passed a load test, or any other test. It's simply a "confidence builder" piece of paper that they get from the raw materials supplier and pass on to SpaceX.

Normally a materials cert is not sufficient proof that the part meets spec. It's one element of a QA traceability chain, but it certainly doesn't prove that any manufactured part will meet spec, which is why it's so surprising that SpaceX didn't require 100% load testing of the struts.

[CyndyC]

Whether or not SpaceX required the vendor to do some random load testing from each batch is unknown and not really relevant anyway, because as they learned, one flaw in one casting can ruin your whole rocket. That's why you do 100% load testing of all structure. I'm quite surprised by this lapse in their QA criteria, but no doubt they won't make that mistake again.

That's why I brought up the identity of the supplier earlier, which I think could explain some things one way or the other. At times it has appeared rocketry is a brotherhood behind the scenes, not pure competition, so maybe the company has been around longer than SpaceX, and should have been safely considered completely reliable. At another extreme maybe it was a case of good old fashioned projection, and a newer company impressed the SpaceX leadership as being as smart with as strict quality control, when in reality it wasn't. Either way, I think SpaceX must have been relishing one relatively small thing they were convinced they didn't have to worry about on top of everything inhouse. I wouldn't have used the word Elon Musk did, "complacent", which at first seemed could be taken offensively by all the SpaceXers who had been bending over backwards to ensure quality control from their respective stations, but the word can also mean merely "contented".

[Kabloona]

I wouldn't have used the word Elon Musk did, "complacent", which at first seemed could be taken offensively by all the SpaceXers who had been bending over backwards to ensure quality control from their respective stations, but the word can also mean merely "contented".

 I feel "complacent" probably is the right word, and the SXers whose systems all worked well after being thoroughly acceptance tested were probably relieved that it wasn't "their" strut that failed.

Yes, there's team spirit, but there's also individual accountability, and someone inside SX made the engineering/cost decision not to require 100% load testing of each strut as an acceptance criterion. And after the failure I'll bet there were a lot of SXer's breathing a quiet sigh of relief that they were not "that person."

So I feel it was complacent to rely on (1) design margin and (2) materials certification as a substitute for 100% load testing, especially when failure of one strut would cause loss of mission, and when load testing would have added a trivial amount to the cost of each rocket.

[Coastal Ron]

That's enlightening that the buyer has to require and set the testing criteria, but by saying "apparently", it sounds like you are only assuming SpaceX didn't require any load testing at all.

We're having to read between the lines a bit. What Elon said was that (1) they did not load test *every* strut produced, and (2) they relied instead on materials certifications (ie from the raw materials supplier, not the strut manufacturer). He also said that they had designed the strut with a large safety factor (I don't remember the number, but it was something like a factor of 5, which on a rocket is a huge factor (for man-rated systems, it's usually 1.4)

Boy, I don't agree with any of that.  Specifically:

1.  SpaceX said they were relying on vendor certification, meaning SpaceX was not doing any load testing.  Now they are, even though they are using a new vendor.

2.  You are wrong about certifications, they can be for raw material, but also processed material, components, assemblies and finished products.  One government contractor I worked for, every electronic system we shipped had to have a certain amount of testing and certification done.  Component level inspection & certification for critical items is not unusual either.

3.  To my knowledge Musk has not stated that they designed the strut, and most of what I have read has assumed that the strut was purchased from one of many aerospace strut suppliers.  A strut is a pretty simple part, so being able to offer different lengths, diameters and strengths as a catalog item would not be surprising.

In my experience, what that means is:
...
-As a result, SpaceX probably felt comfortable not requiring 100% acceptance load testing of every strut, but instead relied on a materials certification from the raw material supplier that showed the metallurgical content of each batch of steel along with ultimate tensile strength test samples from each batch.

Musk specifically said that the parts were supposed to be certified.  That's pretty clear that they expected 100% quality parts, and that the vendor would be doing whatever was necessary to ensure/certify that.

Whether or not SpaceX required the vendor to do some random load testing from each batch is unknown and not really relevant anyway, because as they learned, one flaw in one casting can ruin your whole rocket.

Why are you assuming this was a cast part?  You don't use cast parts when you want strength, you use forged or stamped parts.

That's why you do 100% load testing of all structure. I'm quite surprised by this lapse in their QA criteria, but no doubt they won't make that mistake again.

100% load testing of all structures?  What industry were you in?  Certainly not aerospace.

No, you misunderstand what a "materials certification" is. It's a piece of paper from the raw materials vendor that the strut maker bought the steel bar stock (or whatever) from. It gives traceability that shows where the raw steel came from, its metallurgical properties, ultimate strength, etc. its basically a "birth certificate" that shows where the raw materials came from and proves they have "good genes."...

I don't know what industry you have been in, but what you described is not the norm.  I don't know of any industry that I've been in where the raw material certification is the only certification.

My interpretation is that SpaceX was paying for each part to be certified by the supplier, and that it was the suppliers responsibility to ensure that each part met the minimum spec. (i.e. 10,000 lbs).  Now, post accident, they are doing what would be expected, which is that they are being very cautious so that the same problem does not bring down another customers payload, so they changed suppliers and are being doubly cautious by testing the parts themselves (I would assume they are also asking for supplier certification too).

[Dante80]

That's why you do 100% load testing of all structure.

That is not what is done in the aerospace industry (I think).

[Kabloona]

2.  You are wrong about certifications, they can be for raw material, but also processed material, components, assemblies and finished products.  One government contractor I worked for, every electronic system we shipped had to have a certain amount of testing and certification done.  Component level inspection & certification for critical items is not unusual either.

I never said there weren't other types of certification. But in this case we're taking about a "materials" certification, because this is what Elon actually said:

we're going to move to individually testing each strut independent of any material certifications.

http://shitelonsays.com/transcript/elon-musk-talks-failed-crs-7-dragon-mission-2015-07-20

In the NASA/Air Force launch vehicle world, a "materials" certification is exactly what I described above, in this case a piece of paper from the raw steel supplier certifying the steel composition and properties. A materials certification is *not* produced by the vendor who makes the part (or in this case the strut). It comes from the raw material supplier.

For example, here's NASA/JSC's materials control plan for flight hardware,
https://standards.nasa.gov/documents/viewdoc/3315899/3315899, requiring certification of materials "composition and properties" (para 4.6.a). That's what a materials certification is.

Maybe the strut vendor did some random sample load testing in each batch of struts, but such testing should not be confused with "materials certification" because they are two completely different things, and someone like Elon would never use the very specific term "materials certification" to mean random sample load testing on a batch of finished struts.

So I stand by my interpretation that the strut vendor was simply making the struts and providing SpaceX with the materials certification they got from their raw materials supplier to prove that the steel they were using was of adequate strength, and that is the "materials certification" Elon referred to.

[cambrianera]

That's why you do 100% load testing of all structure.

That is not what is done in the aerospace industry (I think).

100% testing is not that uncommon.
Every ASME U stamp and/or PED certified pressure vessel has to be pressure tested.

(modified, while uncommon, gas pressure test is foreseen)

[Kabloona]

That's why you do 100% load testing of all structure.

That is not what is done in the aerospace industry (I think).

100% testing is not that uncommon.
Every ASME U stamp and/or PED certified pressure vessel has to be hydrostatically pressure tested.

Yes, but Coastal Ron is correct, launch vehicle structure is usually not acceptance load tested. After the design is load tested in qual, materials certs and build paperwork and inspections are the usual QA criteria for unit acceptance. Some critical structures like propellant tanks are, however, proof tested as part of acceptance.

Sorry, don't know what I was smoking when I wrote that.

But to give a relevant example, on Transfer Orbit Stage we bought from a vendor the separation system (two metal interface rings joined by frangible plates that would be broken at sep by a linear shaped charge), the frangible plates were of particular concern because they were designated "fracture critical," ie a crack in a plate could cause the system to fail with resulting loss of mission/crew.

Fracture critical parts like that all required the "materials certification" paper that I described above from the raw metals suppliers. Those materials certs were incorporated into the build logs and were carefully reviewed by NASA. The materials certs were one of the first things they looked for.

[mfck]

(...)
Sorry, don't know what I was smoking when I wrote that.

Was it certified? I feel your QA process needs a review

[Kabloona]

(...)
Sorry, don't know what I was smoking when I wrote that.

Was it certified? I feel your QA process needs a review

No, but some people have told me I'm certifiable. 

[Coastal Ron]

2.  You are wrong about certifications, they can be for raw material, but also processed material, components, assemblies and finished products.  One government contractor I worked for, every electronic system we shipped had to have a certain amount of testing and certification done.  Component level inspection & certification for critical items is not unusual either.

I never said there weren't other types of certification. But in this case we're taking about a "materials" certification, because this is what Elon actually said:

we're going to move to individually testing each strut independent of any material certifications.

You are interpreting a lot into that comment.  For instance, you don't know the nomenclature of SpaceX internal processes, which for them the term "Material Certification" could mean that the component being received is certified, not that just the raw material it was made out of was certified.  Remember that they receive both raw material and finished material items, so the same terminology could be used for both - which was the case for the companies I've worked at.

And this is important because if you assume that "Material Certification" only means the raw material portion of a finished purchased part, and not that the finished part itself meets ALL the specifications of the engineering documentation that it was designed to handle, then I'd say it was a miracle that any of their rockets flew.  Because they would have used the same processes and procedures with EVERYTHING they bought (and likely everything they built in-house too).