SpaceX Systems Engineering

[Kabloona]

I just stumbled across this SpaceX Systems Engineering presentation and found it interesting as a former systems engineer myself.

https://www.aiaa.org/uploadedFiles/Events/Conferences/2012_Conferences/2012-Complex-Aerospace-Systems-Exchange-Event/Detailed_Program/CASE2012_2-4_Muratore_presentation.pdf

One interesting bit is the chart that shows the flow of qual and acceptance testing. There's a box labeled "structures component acceptance testing." And a bullet on page 16 says "All hardware acceptance tested." But we know from the CRS-7 failure investigation that the helium tank struts were not acceptance (load) tested.

And not acceptance testing the struts ran counter to their philosophy of extensive testing. So it's interesting that not adhering to their own philosophy in the case of the struts came back to bite them.

[Dante80]

Nice presentation, thanks for sharing it.

A question. How accurate are the below cost numbers?

[Kabloona]

I have no insight into those numbers, but a post on parabolicarc says NASA has verified the Falcon 1 and Falcon 9 development cost numbers:

SpaceX has publicly indicated that the development cost for Falcon 9 launch vehicle was approximately $300 million. Additionally, approximately $90 million was spent developing the Falcon 1 launch vehicle which did contribute to some extent to the Falcon 9, for a total of $390 million. NASA has verified these costs.

http://www.parabolicarc.com/2011/05/31/nasa-analysis-falcon-9-cheaper-traditional-approach/

And according to that same article, NASA's own analysis said it would have cost them between $1.7B and $4.0B to develop F9. So SpaceX beat the NASA development cost estimate by at least a factor of 5, and possibly a factor of 10+.

Extremely impressive.

[mfck]

Very interesting presentation. Thanks for posting it.


Looking still at the price numbers in that screenshot, would it be safe to assume an F9 pad is an order of magnitude more, i.e. in the low hundreds of $M and that subsequent F9 integration hangars are at least half the price?

Also, what would be the educated guess about how much that $390M has grown since the presentation - F9 1.1, F9 FT, etc. and what would be the cost of the CRS-7 investigation and direct changes as consequence (not the costs associated with not flying)?

[guckyfan]

Very interesting presentation. Thanks for posting it.


Looking still at the price numbers in that screenshot, would it be safe to assume an F9 pad is an order of magnitude more, i.e. in the low hundreds of $M and that subsequent F9 integration hangars are at least half the price?

The quoted cost for the Brownsville site are just short of 100 million $. That's for a FH launch pad, TE, HIF, control center and satellite integration facility with handling of hypergolic fuels.

[CyndyC]

I just stumbled across this SpaceX Systems Engineering presentation and found it interesting as a former systems engineer myself.

https://www.aiaa.org/uploadedFiles/Events/Conferences/2012_Conferences/2012-Complex-Aerospace-Systems-Exchange-Event/Detailed_Program/CASE2012_2-4_Muratore_presentation.pdf

One interesting bit is the chart that shows the flow of qual and acceptance testing. There's a box labeled "structures component acceptance testing." And a bullet on page 16 says "All hardware acceptance tested." But we know from the CRS-7 failure investigation that the helium tank struts were not acceptance (load) tested.

And not acceptance testing the struts ran counter to their philosophy of extensive testing. So it's interesting that not adhering to their own philosophy in the case of the struts came back to bite them.

Your experience with component testing must have been under an alternate definition. According to definitions for "acceptance testing" online, the general and most common usage, and probably the way SpaceX meant it, refers to testing by the manufacturer prior to delivery to the customer. Labautopaedia.org goes on to point out that a substantial portion of a customer's payment goes toward the manufacturer's testing, and that portion of the payment is not due & paid unless and until the testing is successful.

The discussion on the subject of product testing was one of the long ones in post CRS-7 threads, so I don't want restart too much of it by responding, but wherever the previous discussion left off, I don't think the supplier was ever identified. I don't think much more can be said w/o knowing that identity, and how reliable the company could have or should have been considered. As far as manufacturing techniques alone were to blame, I didn't have to go any further than my 100th frozen Tuscan Chicken casserole, which had every ingredient but the chicken. SOMEbody kept the production line going just a touch too long.

[Kabloona]

I just stumbled across this SpaceX Systems Engineering presentation and found it interesting as a former systems engineer myself.

https://www.aiaa.org/uploadedFiles/Events/Conferences/2012_Conferences/2012-Complex-Aerospace-Systems-Exchange-Event/Detailed_Program/CASE2012_2-4_Muratore_presentation.pdf

One interesting bit is the chart that shows the flow of qual and acceptance testing. There's a box labeled "structures component acceptance testing." And a bullet on page 16 says "All hardware acceptance tested." But we know from the CRS-7 failure investigation that the helium tank struts were not acceptance (load) tested.

And not acceptance testing the struts ran counter to their philosophy of extensive testing. So it's interesting that not adhering to their own philosophy in the case of the struts came back to bite them.

Your experience with component testing must have been under an alternate definition. According to definitions for "acceptance testing" online, the general and most common usage, and probably the way SpaceX meant it, refers to testing by the manufacturer prior to delivery to the customer. Labautopaedia.org goes on to point out that a substantial portion of a customer's payment goes toward the manufacturer's testing, and that portion of the payment is not due & paid unless and until the testing is successful.

I'm quite familiar with common aerospace practice for component subcontracts and acceptance testing, and I'm trying but failing to understand why you think my post somehow contradicts what you wrote above.

When I said "not adhering to their own philosophy," what I meant was that apparently they did not require load testing by the vendor as part of the acceptance criteria. It's up to to buyer to specify what their acceptance criteria are, not the vendor. So by not writing a load testing acceptance requirement into the procurement contract, SpaceX did not adhere to their "test what you fly" philosophy.

So I think we are in agreement. 

[CyndyC]

One interesting bit is the chart that shows the flow of qual and acceptance testing. There's a box labeled "structures component acceptance testing." And a bullet on page 16 says "All hardware acceptance tested." But we know from the CRS-7 failure investigation that the helium tank struts were not acceptance (load) tested.

And not acceptance testing the struts ran counter to their philosophy of extensive testing. So it's interesting that not adhering to their own philosophy in the case of the struts came back to bite them.

Your experience with component testing must have been under an alternate definition. According to definitions for "acceptance testing" online, the general and most common usage, and probably the way SpaceX meant it, refers to testing by the manufacturer prior to delivery to the customer. Labautopaedia.org goes on to point out that a substantial portion of a customer's payment goes toward the manufacturer's testing, and that portion of the payment is not due & paid unless and until the testing is successful.

I'm quite familiar with common aerospace practice for component subcontracts and acceptance testing, and I'm trying but failing to understand why you think my post somehow contradicts what you wrote above.

When I said "not adhering to their own philosophy," what I meant was that apparently they did not require load testing by the vendor as part of the acceptance criteria. It's up to to buyer to specify what their acceptance criteria are, not the vendor. So by not writing a load testing acceptance requirement into the procurement contract, SpaceX did not adhere to their "test what you fly" philosophy.

So I think we are in agreement. 

You are in more of a rank & position to call SpaceX on this subject than I am, so I don't want to try saying I completely agree, at least not without more information. That's enlightening that the buyer has to require and set the testing criteria, but by saying "apparently", it sounds like you are only assuming SpaceX didn't require any load testing at all. Someone in the prior discussions brought up "batch testing" in manufacturing, so although that person was only assuming too, at least that much testing might have been required by SX & then done by the manufacturer. Maybe not though. BTW, isn't turning an argument around to convince the opposite gender has agreed with you supposed to be my line?

[Coastal Ron]

That's enlightening that the buyer has to require and set the testing criteria, but by saying "apparently", it sounds like you are only assuming SpaceX didn't require any load testing at all. Someone in the prior discussions brought up "batch testing" in manufacturing, so although that person was only assuming too, at least that much testing might have been required by SX & then done by the manufacturer. Maybe not though.

SpaceX, and Musk, have stated that the strut was supposed to be "certified" by the supplier.  Keeping in mind that the minimum strength of the strut was supposed to be a breaking point of 10,000 lbs of force, it normally would be up to the supplier to figure out how to ensure that 100% of the product that they supply would not fail below 10,000 lbs of force.  The parts may actually be designed for a strength far higher than 10,000 lbs of force, and the supplier may have calculated that it was not possible to manufacture finished parts that failed below 10,000 lbs of force.  If so then obviously they were wrong.

And that there was a certification process tells me that SpaceX was paying to ensure that 100% of the parts they received would not fail below 10,000 lbs of force.  SpaceX supplier QA would know that "batch testing" does not ensure 100% good parts, so we can rule that out.  And since there were multiple parts that SpaceX found that failed internal SpaceX testing post-failure, that tells me that the supplier was not being forthcoming with SpaceX about their testing program.  Read into that whatever you want.

I've experience enough production stoppages because of supplier issues to understand how difficult it is to ensure 100% reliability from suppliers.  And interestingly reusability is really the best way for SpaceX to find out how well their quality is in-house, since you don't know how close to failure you really are until you can examine something after it's been used as intended.  And I say that because we'll never really know if it was a part failure, or an assembly failure that led to the CRS-7 accident - all we know are that sub-standard parts were found in SpaceX inventory that could have caused the accident.  So reusability will strengthen their internal quality.

[Kabloona]

That's enlightening that the buyer has to require and set the testing criteria, but by saying "apparently", it sounds like you are only assuming SpaceX didn't require any load testing at all.

We're having to read between the lines a bit. What Elon said was that (1) they did not load test *every* strut produced, and (2) they relied instead on materials certifications (ie from the raw materials supplier, not the strut manufacturer). He also said that they had designed the strut with a large safety factor (I don't remember the number, but it was something like a factor of 5, which on a rocket is a huge factor (for man-rated systems, it's usually 1.4)

In my experience, what that means is:

-SpaceX designed the strut and had the vendor qualify the design by testing a bunch of struts to the breaking point, thereby verifying they had a lot of margin.

-As a result, SpaceX probably felt comfortable not requiring 100% acceptance load testing of every strut, but instead relied on a materials certification from the raw material supplier that showed the metallurgical content of each batch of steel along with ultimate tensile strength test samples from each batch.

Whether or not SpaceX required the vendor to do some random load testing from each batch is unknown and not really relevant anyway, because as they learned, one flaw in one casting can ruin your whole rocket. That's why you do 100% load testing of all structure. I'm quite surprised by this lapse in their QA criteria, but no doubt they won't make that mistake again.

BTW, isn't turning an argument around to convince the opposite gender has agreed with you supposed to be my line?

Turnabout is fair play. 

SpaceX, and Musk, have stated that the strut was supposed to be "certified" by the supplier.

No, you misunderstand what a "materials certification" is. It's a piece of paper from the raw materials vendor that the strut maker bought the steel bar stock (or whatever) from. It gives traceability that shows where the raw steel came from, its metallurgical properties, ultimate strength, etc. its basically a "birth certificate" that shows where the raw materials came from and proves they have "good genes."

The vendor then machines the raw steel into a finished product and hands the materials certification over to the buyer (SpaceX) to prove that the raw materials that the part is made of were good.

A materials certification is *not* a certification by the vendor that their strut passed a load test, or any other test. It's simply a "confidence builder" piece of paper that they get from the raw materials supplier and pass on to SpaceX.

Normally a materials cert is not sufficient proof that the part meets spec. It's one element of a QA traceability chain, but it certainly doesn't prove that any manufactured part will meet spec, which is why it's so surprising that SpaceX didn't require 100% load testing of the struts.

[CyndyC]

Whether or not SpaceX required the vendor to do some random load testing from each batch is unknown and not really relevant anyway, because as they learned, one flaw in one casting can ruin your whole rocket. That's why you do 100% load testing of all structure. I'm quite surprised by this lapse in their QA criteria, but no doubt they won't make that mistake again.

That's why I brought up the identity of the supplier earlier, which I think could explain some things one way or the other. At times it has appeared rocketry is a brotherhood behind the scenes, not pure competition, so maybe the company has been around longer than SpaceX, and should have been safely considered completely reliable. At another extreme maybe it was a case of good old fashioned projection, and a newer company impressed the SpaceX leadership as being as smart with as strict quality control, when in reality it wasn't. Either way, I think SpaceX must have been relishing one relatively small thing they were convinced they didn't have to worry about on top of everything inhouse. I wouldn't have used the word Elon Musk did, "complacent", which at first seemed could be taken offensively by all the SpaceXers who had been bending over backwards to ensure quality control from their respective stations, but the word can also mean merely "contented".

[Kabloona]

I wouldn't have used the word Elon Musk did, "complacent", which at first seemed could be taken offensively by all the SpaceXers who had been bending over backwards to ensure quality control from their respective stations, but the word can also mean merely "contented".

 I feel "complacent" probably is the right word, and the SXers whose systems all worked well after being thoroughly acceptance tested were probably relieved that it wasn't "their" strut that failed.

Yes, there's team spirit, but there's also individual accountability, and someone inside SX made the engineering/cost decision not to require 100% load testing of each strut as an acceptance criterion. And after the failure I'll bet there were a lot of SXer's breathing a quiet sigh of relief that they were not "that person."

So I feel it was complacent to rely on (1) design margin and (2) materials certification as a substitute for 100% load testing, especially when failure of one strut would cause loss of mission, and when load testing would have added a trivial amount to the cost of each rocket.

[Coastal Ron]

That's enlightening that the buyer has to require and set the testing criteria, but by saying "apparently", it sounds like you are only assuming SpaceX didn't require any load testing at all.

We're having to read between the lines a bit. What Elon said was that (1) they did not load test *every* strut produced, and (2) they relied instead on materials certifications (ie from the raw materials supplier, not the strut manufacturer). He also said that they had designed the strut with a large safety factor (I don't remember the number, but it was something like a factor of 5, which on a rocket is a huge factor (for man-rated systems, it's usually 1.4)

Boy, I don't agree with any of that.  Specifically:

1.  SpaceX said they were relying on vendor certification, meaning SpaceX was not doing any load testing.  Now they are, even though they are using a new vendor.

2.  You are wrong about certifications, they can be for raw material, but also processed material, components, assemblies and finished products.  One government contractor I worked for, every electronic system we shipped had to have a certain amount of testing and certification done.  Component level inspection & certification for critical items is not unusual either.

3.  To my knowledge Musk has not stated that they designed the strut, and most of what I have read has assumed that the strut was purchased from one of many aerospace strut suppliers.  A strut is a pretty simple part, so being able to offer different lengths, diameters and strengths as a catalog item would not be surprising.

In my experience, what that means is:
...
-As a result, SpaceX probably felt comfortable not requiring 100% acceptance load testing of every strut, but instead relied on a materials certification from the raw material supplier that showed the metallurgical content of each batch of steel along with ultimate tensile strength test samples from each batch.

Musk specifically said that the parts were supposed to be certified.  That's pretty clear that they expected 100% quality parts, and that the vendor would be doing whatever was necessary to ensure/certify that.

Whether or not SpaceX required the vendor to do some random load testing from each batch is unknown and not really relevant anyway, because as they learned, one flaw in one casting can ruin your whole rocket.

Why are you assuming this was a cast part?  You don't use cast parts when you want strength, you use forged or stamped parts.

That's why you do 100% load testing of all structure. I'm quite surprised by this lapse in their QA criteria, but no doubt they won't make that mistake again.

100% load testing of all structures?  What industry were you in?  Certainly not aerospace.

No, you misunderstand what a "materials certification" is. It's a piece of paper from the raw materials vendor that the strut maker bought the steel bar stock (or whatever) from. It gives traceability that shows where the raw steel came from, its metallurgical properties, ultimate strength, etc. its basically a "birth certificate" that shows where the raw materials came from and proves they have "good genes."...

I don't know what industry you have been in, but what you described is not the norm.  I don't know of any industry that I've been in where the raw material certification is the only certification.

My interpretation is that SpaceX was paying for each part to be certified by the supplier, and that it was the suppliers responsibility to ensure that each part met the minimum spec. (i.e. 10,000 lbs).  Now, post accident, they are doing what would be expected, which is that they are being very cautious so that the same problem does not bring down another customers payload, so they changed suppliers and are being doubly cautious by testing the parts themselves (I would assume they are also asking for supplier certification too).

[Dante80]

That's why you do 100% load testing of all structure.

That is not what is done in the aerospace industry (I think).

[Kabloona]

2.  You are wrong about certifications, they can be for raw material, but also processed material, components, assemblies and finished products.  One government contractor I worked for, every electronic system we shipped had to have a certain amount of testing and certification done.  Component level inspection & certification for critical items is not unusual either.

I never said there weren't other types of certification. But in this case we're taking about a "materials" certification, because this is what Elon actually said:

we're going to move to individually testing each strut independent of any material certifications.

http://shitelonsays.com/transcript/elon-musk-talks-failed-crs-7-dragon-mission-2015-07-20

In the NASA/Air Force launch vehicle world, a "materials" certification is exactly what I described above, in this case a piece of paper from the raw steel supplier certifying the steel composition and properties. A materials certification is *not* produced by the vendor who makes the part (or in this case the strut). It comes from the raw material supplier.

For example, here's NASA/JSC's materials control plan for flight hardware,
https://standards.nasa.gov/documents/viewdoc/3315899/3315899, requiring certification of materials "composition and properties" (para 4.6.a). That's what a materials certification is.

Maybe the strut vendor did some random sample load testing in each batch of struts, but such testing should not be confused with "materials certification" because they are two completely different things, and someone like Elon would never use the very specific term "materials certification" to mean random sample load testing on a batch of finished struts.

So I stand by my interpretation that the strut vendor was simply making the struts and providing SpaceX with the materials certification they got from their raw materials supplier to prove that the steel they were using was of adequate strength, and that is the "materials certification" Elon referred to.

[cambrianera]

That's why you do 100% load testing of all structure.

That is not what is done in the aerospace industry (I think).

100% testing is not that uncommon.
Every ASME U stamp and/or PED certified pressure vessel has to be pressure tested.

(modified, while uncommon, gas pressure test is foreseen)

[Kabloona]

That's why you do 100% load testing of all structure.

That is not what is done in the aerospace industry (I think).

100% testing is not that uncommon.
Every ASME U stamp and/or PED certified pressure vessel has to be hydrostatically pressure tested.

Yes, but Coastal Ron is correct, launch vehicle structure is usually not acceptance load tested. After the design is load tested in qual, materials certs and build paperwork and inspections are the usual QA criteria for unit acceptance. Some critical structures like propellant tanks are, however, proof tested as part of acceptance.

Sorry, don't know what I was smoking when I wrote that.

But to give a relevant example, on Transfer Orbit Stage we bought from a vendor the separation system (two metal interface rings joined by frangible plates that would be broken at sep by a linear shaped charge), the frangible plates were of particular concern because they were designated "fracture critical," ie a crack in a plate could cause the system to fail with resulting loss of mission/crew.

Fracture critical parts like that all required the "materials certification" paper that I described above from the raw metals suppliers. Those materials certs were incorporated into the build logs and were carefully reviewed by NASA. The materials certs were one of the first things they looked for.

[mfck]

(...)
Sorry, don't know what I was smoking when I wrote that.

Was it certified? I feel your QA process needs a review

[Kabloona]

(...)
Sorry, don't know what I was smoking when I wrote that.

Was it certified? I feel your QA process needs a review

No, but some people have told me I'm certifiable. 

[Coastal Ron]

2.  You are wrong about certifications, they can be for raw material, but also processed material, components, assemblies and finished products.  One government contractor I worked for, every electronic system we shipped had to have a certain amount of testing and certification done.  Component level inspection & certification for critical items is not unusual either.

I never said there weren't other types of certification. But in this case we're taking about a "materials" certification, because this is what Elon actually said:

we're going to move to individually testing each strut independent of any material certifications.

You are interpreting a lot into that comment.  For instance, you don't know the nomenclature of SpaceX internal processes, which for them the term "Material Certification" could mean that the component being received is certified, not that just the raw material it was made out of was certified.  Remember that they receive both raw material and finished material items, so the same terminology could be used for both - which was the case for the companies I've worked at.

And this is important because if you assume that "Material Certification" only means the raw material portion of a finished purchased part, and not that the finished part itself meets ALL the specifications of the engineering documentation that it was designed to handle, then I'd say it was a miracle that any of their rockets flew.  Because they would have used the same processes and procedures with EVERYTHING they bought (and likely everything they built in-house too).

[Kabloona]

And this is important because if you assume that "Material Certification" only means the raw material portion of a finished purchased part, and not that the finished part itself meets ALL the specifications of the engineering documentation that it was designed to handle...

That's what "materials certification" means within NASA, and I'd be very surprised if it meant anything different inside SpaceX given how many of their employees have prior experience with NASA standards and the close coordination SpaceX has with their customer...not to mention the requirements NASA is/will be levying on manned F9's.

The way the finished part is determined to meet ALL the specifications is by testing and/or inspection that verifies each spec requirement, and that isn't called "material certification," it's called "acceptance testing," ie verifying that the received part meets all specs before the buyer formally accepts the item.

 If it does, it passes "acceptance," and the acceptance test paperwork, along with the materials certs, is eventually incorporated into the vehicle build log. If it doesn't pass acceptance tests, the part is rejected.

But on all the programs I've worked (Pegasus, Taurus, Transfer Orbit Stage), the "materials certifications" from vendors were always certification only of raw materials properties and never implied verification of part functionality, etc. Those spec requirements were always verified by separate acceptance tests or inspections.

[AncientU]

I have no insight into those numbers, but a post on parabolicarc says NASA has verified the Falcon 1 and Falcon 9 development cost numbers:

SpaceX has publicly indicated that the development cost for Falcon 9 launch vehicle was approximately $300 million. Additionally, approximately $90 million was spent developing the Falcon 1 launch vehicle which did contribute to some extent to the Falcon 9, for a total of $390 million. NASA has verified these costs.

http://www.parabolicarc.com/2011/05/31/nasa-analysis-falcon-9-cheaper-traditional-approach/

And according to that same article, NASA's own analysis said it would have cost them between $1.7B and $4.0B to develop F9. So SpaceX beat the NASA development cost estimate by at least a factor of 5, and possibly a factor of 10+.

Extremely impressive.

The interesting thing is that they are accounting for F-1 contributions to F-9 development; also including Dragon costs.  We've heard time and again that NASA doesn't have to include past development costs in the SLS/Orion accounting -- things such as the engines, the solids, the tankage, the capsule, the heat shield, the ACS, ... wait, that's almost all of SLS technology developed before the first SLS weld.

If we'd account NASA costs with the same measure, the number would be astronomically higher!

[oldAtlas_Eguy]

In looking at the design development process SpaceX uses it reminds me of the tremendous differences in the Spiral vs Waterfall software development methods. Spiral is design-implement-test-deploy-design-implement-test-deploy... where Waterfall is study-analyze-design-implement-test-deploy-maintain. If the design falls short it is very difficult to overcome these design faults in maintenance phase of the traditional Waterfall method where only the lowest levels are reworked to meet the higher level designs without starting all over, whereas in the Spiral methodology redesign at higher levels is part of the process of evolving the design to make it better and better.

Also the time between design start and deploy is very short in the spiral vs the long time for the waterfall. Since time is money also each iteration in the Spiral is a lot cheaper than the waterfall to get to a deployed system. The drawback is that the Spiral method deploys the minimum system and increases it in each iteration where waterfall deploys the complete end post system.

From the historical of SpaceX improving the F9 every 2.5 years v1.0->v1.1->FT they are definitely using the Spiral vs the design and done Waterfall that NASA and other LV developers use.

[JamesH]

In looking at the design development process SpaceX uses it reminds me of the tremendous differences in the Spiral vs Waterfall software development methods. Spiral is design-implement-test-deploy-design-implement-test-deploy... where Waterfall is study-analyze-design-implement-test-deploy-maintain. If the design falls short it is very difficult to overcome these design faults in maintenance phase of the traditional Waterfall method where only the lowest levels are reworked to meet the higher level designs without starting all over, whereas in the Spiral methodology redesign at higher levels is part of the process of evolving the design to make it better and better.

Also the time between design start and deploy is very short in the spiral vs the long time for the waterfall. Since time is money also each iteration in the Spiral is a lot cheaper than the waterfall to get to a deployed system. The drawback is that the Spiral method deploys the minimum system and increases it in each iteration where waterfall deploys the complete end post system.

From the historical of SpaceX improving the F9 every 2.5 years v1.0->v1.1->FT they are definitely using the Spiral vs the design and done Waterfall that NASA and other LV developers use.

With Musk coming directly from a software background, this approach should be of no surprise. I'm guessing, but I would think Musk looked at the incumbents (mostly/all waterfall?), and thought they were beatable by a more agile company.

Time will tell, but he's made a good start.

[Coastal Ron]

And this is important because if you assume that "Material Certification" only means the raw material portion of a finished purchased part, and not that the finished part itself meets ALL the specifications of the engineering documentation that it was designed to handle...

That's what "materials certification" means within NASA...

Which is what I mean.  You are trying to extrapolate how the private sector (and actually a unique sub-segment of it) works by applying government rules to it.  That doesn't work.

...and I'd be very surprised if it meant anything different inside SpaceX given how many of their employees have prior experience with NASA standards and the close coordination SpaceX has with their customer...not to mention the requirements NASA is/will be levying on manned F9's.

What percentage of their early employees are you figuring came from NASA supplier quality?  And you're assuming the NASA sets SpaceX standards, which we know from other interactions SpaceX has had with the government doesn't happen - SpaceX does what SpaceX feels is the best way (as do most government contractors).

The way the finished part is determined to meet ALL the specifications is by testing and/or inspection that verifies each spec requirement, and that isn't called "material certification," it's called "acceptance testing," ie verifying that the received part meets all specs before the buyer formally accepts the item.

Again, without knowing the nomenclature SpaceX uses for all of it's processes it's impossible to equate anything to anything.

But on all the programs I've worked (Pegasus, Taurus, Transfer Orbit Stage), the "materials certifications" from vendors were always certification only of raw materials properties and never implied verification of part functionality, etc. Those spec requirements were always verified by separate acceptance tests or inspections.

You're talking about internal processes.  The question you raise about "Material Certification" is whether the supplier does any inspection or validation for what they produce.  You imply "NO", that all they would do is supply a raw material certification and leave it at that - if the part is wrong, then the customer has to catch it.  I don't believe this is what happens.

[sdsds]

Yes, there's team spirit, but there's also individual accountability, and someone inside SX made the engineering/cost decision not to require 100% load testing of each strut as an acceptance criterion. And after the failure I'll bet there were a lot of SXer's breathing a quiet sigh of relief that they were not "that person."

From my experience the other possibility is that no one was "that person." Somehow a set of acceptance test criteria became the default (perhaps they were offered by the external vendor) and no one internally was tasked with asking, "Are we sure this set of criteria meet SpaceX engineering standards?"

Or is there an indication that someone at SpaceX once explicitly asserted that 100% load testing was not required for that strut?

[Kabloona]

You are trying to extrapolate how the private sector (and actually a unique sub-segment of it) works by applying government rules to it.  That doesn't work.

It does when SpaceX's main customer is NASA. They have to speak the same language to some extent, and meet NASA requirements for man-rating F9. What you're suggesting is that SpaceX doesn't understand what NASA means by a "materials certification" or has a different definition of it, when "materials certification" is going to be a huge paperwork requirement levied by NASA for manned F9 and crewed Dragon flights. In my view it's highly unlikely that SpaceX doesn't understand or has a different definition.

The question you raise about "Material Certification" is whether the supplier does any inspection or validation for what they produce.  You imply "NO", that all they would do is supply a raw material certification and leave it at that - if the part is wrong, then the customer has to catch it.  I don't believe this is what happens.

What I said earlier was:

Maybe the strut vendor did some random sample load testing in each batch of struts, but such testing should not be confused with "materials certification" because they are two completely different things, and someone like Elon would never use the very specific term "materials certification" to mean random sample load testing on a batch of finished struts.

And according to Elon, whatever inspection or testing was done, they did not require 100% load testing of *every* strut. Which is kind of the bottom line. Whatever they did, whatever materials certs they provided to SpaceX from the raw materials supplier, it was not sufficient to weed out a bad strut.

So I'm not sure what we're debating any more except the definition of a "materials certification," and I'll let it go at that.

[AncientU]

From what I've heard quoted, SpaceX isn't focused on process -- in fact, I believe process falls into the same category as acronyms... avoid them.  Focus of of EM's hiring is on product, not process, and hands-on performance.

When SpaceX was getting certified by the USAF, it was ding'd for not even having a Systems Engineering function.  Strikes me as funny we'd have a thread on it. 

[Kabloona]

Or is there an indication that someone at SpaceX once explicitly asserted that 100% load testing was not required for that strut?

Every procurement contract has the acceptance criteria written into it. Someone at SpaceX wrote that contract, with the acceptance criteria for the strut either specified or omitted. Usually a contracts person writes the contract and incorporates the acceptance criteria as written/specified by the cognizant engineer.

You can debate whether this was a sin of omission or a sin of commission, but that's beside the point. The point is that it's up to SpaceX to write the required acceptance criteria into the contract. Obviously they didn't require 100% load testing. That was either a conscious decision or an oversight by one or more people.

[CyndyC]

In looking at the design development process SpaceX uses it reminds me of the tremendous differences in the Spiral vs Waterfall software development methods. Spiral is design-implement-test-deploy-design-implement-test-deploy... where Waterfall is study-analyze-design-implement-test-deploy-maintain. If the design falls short it is very difficult to overcome these design faults in maintenance phase of the traditional Waterfall method where only the lowest levels are reworked to meet the higher level designs without starting all over, whereas in the Spiral methodology redesign at higher levels is part of the process of evolving the design to make it better and better.

Also the time between design start and deploy is very short in the spiral vs the long time for the waterfall. Since time is money also each iteration in the Spiral is a lot cheaper than the waterfall to get to a deployed system. The drawback is that the Spiral method deploys the minimum system and increases it in each iteration where waterfall deploys the complete end post system.

From the historical of SpaceX improving the F9 every 2.5 years v1.0->v1.1->FT they are definitely using the Spiral vs the design and done Waterfall that NASA and other LV developers use.

With Musk coming directly from a software background, this approach should be of no surprise. I'm guessing, but I would think Musk looked at the incumbents (mostly/all waterfall?), and thought they were beatable by a more agile company.

Time will tell, but he's made a good start.

I'm not doubting that SpaceX uses Spiral, from the way Atlas described it and Waterfall, but it would be interesting to know how much Elon Musk has had the time & inclination to stay hands on with the computer programming at SpaceX (and at Tesla for that matter). Even though some of the basics haven't changed, don't forget much of his software background was old school, 80s & 90s. PayPal coding may have been the last of it, and I think he had moved on to designing rockets & electric cars before PayPal was sold to eBay in 2002.

[Coastal Ron]

You are trying to extrapolate how the private sector (and actually a unique sub-segment of it) works by applying government rules to it.  That doesn't work.

It does when SpaceX's main customer is NASA.

The main customer for Dragon maybe, but not Falcon 9.  And most of the early NASA money was not for products or delivery services, it was for completing engineering or program milestones for Dragon related development (i.e. COTS, CCDev, etc.).  NASA had no monetary contribution to any SpaceX rocket.

They have to speak the same language to some extent, and meet NASA requirements for man-rating F9.

Unlike government contracts where a contractor is delivering a product, SpaceX delivers a service, so NASA is never involved in the build side of SpaceX - there is no "NASA Falcon 9".  Hence the "Commercial" in Commercial Cargo and Commercial Crew.  NASA doesn't dictate the processes and procedures that the companies use, and the only "deliverable" is a service.

So why would SpaceX need to craft their entire Falcon 9 quality system around a customer that will never have influence over the Falcon 9?

What you're suggesting is that SpaceX doesn't understand what NASA means by a "materials certification" or has a different definition of it, when "materials certification" is going to be a huge paperwork requirement levied by NASA for manned F9 and crewed Dragon flights. In my view it's highly unlikely that SpaceX doesn't understand or has a different definition.

SpaceX provides a service, not a product.  Why does SpaceX need to deliver any sort of "materials certification"?

[Kabloona]

SpaceX provides a service, not a product.  Why does SpaceX need to deliver any sort of "materials certification"?

We've already seen the in-depth review process that SpaceX had to go through with the Air Force to certify the F9 for DoD. There's no way they got the F9 certified without being able to show the Air Force they have basic QA processes like requiring materials certifications for the raw materials in critical components like tanks. That's just a basic part of a QA program, and if you can't even do a basic thing like verify the source, composition and strength of your various raw materials, there is simply no way the Air Force is going to trust you with a $1B payload.

Same thing for NASA. No way they are going to allow their astronauts on an F9 unless they can verify it meets all their man-rating requirements, including having materials certifications and traceability per the usual NASA standards for manned spaceflight.

The materials certs may not be actual contract deliverables, but SpaceX will certainly have to show NASA that they have adequate QA processes in place for manned missions, and materials certifications are typically one piece of that process.

[Coastal Ron]

SpaceX provides a service, not a product.  Why does SpaceX need to deliver any sort of "materials certification"?

We've already seen the in-depth review process that SpaceX had to go through with the Air Force to certify the F9 for DoD. There's no way they got the F9 certified without being able to show the Air Force they have basic QA processes like requiring materials certifications for the raw materials in critical components like tanks. That's just a basic part of a QA program, and if you can't even do a basic thing like verify the source, composition and strength of your various raw materials, there is simply no way the Air Force is going to trust you with a $1B payload.

And since they were certified, they obviously they were able to show that they have processes to validate their quality, and that those systems are institutional.

But you keep only focusing on raw material.  You seem to completely exclude the possibility that they would require component testing from a supplier - and I think it's because you don't think "material certification" can mean something different than what you have experienced.  That is a fragile line of reasoning...

Same thing for NASA. No way they are going to allow their astronauts on an F9 unless they can verify it meets all their man-rating requirements, including having materials certifications and traceability per the usual NASA standards for manned spaceflight.

NASA has no visibility into the component level parts of a Falcon 9 - it is a commercial service, not a product that NASA is buying.  Do think NASA inspectors are on the Falcon 9 production line?

[whitelancer64]

Or is there an indication that someone at SpaceX once explicitly asserted that 100% load testing was not required for that strut?

*snip* The point is that it's up to SpaceX to write the required acceptance criteria into the contract. Obviously they didn't require 100% load testing. That was either a conscious decision or an oversight by one or more people.

Nor should they have. Batch testing should be more than sufficient to catch gross defects in lots of parts, since the part was over-designed by a factor of 3.

[Nomadd]

Or is there an indication that someone at SpaceX once explicitly asserted that 100% load testing was not required for that strut?

*snip* The point is that it's up to SpaceX to write the required acceptance criteria into the contract. Obviously they didn't require 100% load testing. That was either a conscious decision or an oversight by one or more people.

Nor should they have. Batch testing should be more than sufficient to catch gross defects in lots of parts, since the part was over-designed by a factor of 3.

When a defect, like the one being discussed, can take out a rocket if it appears once in 5,000 parts, your statement doesn't seem to make much sense. Over design or margin is irrelevant. A gross defect like the one they had can take out a part no matter how strong it's suppose to be.

[CyndyC]

As much as I wanted to see and previously thought I saw that SpaceX was off the hook, Kabloona's and now Nomadd's viewpoints seem a bit more realistic. They also shed some new light that is a better match for the gravity of Elon Musk's demeanor during the post CRS-7 public update a month later.

I had time to look up a few things today that MIGHT help draw some conclusions, however much is going to be the public's business in the end anyway. I won't try to sort out a bunch of quotes from the previous posts, but addressing Coastal Ron's insistence that SpaceX is only a service and that NASA would have nothing to do with the rocket itself, there were headlines last May that might have gone under the radar, because they came out only about 10 days before SpaceX was certified by the Air Force. This from another news site "The Falcon 9 is now certified by NASA as a “Category 2” launch vehicle in the agency’s nomenclature....The milestone clears the Falcon 9 rocket to launch NASA’s “medium-risk” science missions."

The cert process began in 2012, originally so that SpaceX could launch Jason 3 "in July [2015]". It goes on, "SpaceX originally bid to launch Jason 3 with the older Falcon 9 v1.0 version of its workhorse rocket, but switched to the more powerful Falcon 9 v1.1, featuring upgraded Merlin engines and other changes. NASA officials said the rocket swap forced engineers to redo part of the certification work, which includes management, process and engineering audits of the contractor."

The certification was concluded by NASA's Launch Services Program, LSP, Jim's dept incidentally. But there was also this, "SpaceX has successfully launched six resupply missions to the International Space Station under contract to NASA, but the agency arranged the launches through the space station program and bypassed the certification needed for science missions."

Both the LSP cert requirements and the ISS "arrangement" requirements can probably be found on the internet for anyone motivated to do more digging. Seems there might be some question as to which rocket under which clearance was used for CRS-7, considering the timing.

[Kabloona]

You seem to completely exclude the possibility that they would require component testing from a supplier...

I don't know why you keep saying that, when at least twice above I've said that it's possible SpaceX required random sample load testing of several struts from each batch.

But they did not require 100% load testing of each strut. That was the critical requirement that should have been written into the contract as a required acceptance test for each strut, and was not. And that lapse was the "complacency" Elon referred to.

But you keep only focusing on raw material. 

Only to explain what the term "materials certification" means to NASA and the two launch vehicle companies I've worked for/with. It means certification of the properties of the raw material that went into the part.  It does not mean certification that the finished part meets all its functional requirements. That's why it's called a "materials" certification and not a "part" or "component" certification.

NASA has no visibility into the component level parts of a Falcon 9 - it is a commercial service, not a product that NASA is buying.

How will NASA possibly certify F9 for man-rating, with "no visibility" into component level parts?  And "visibility" does not necessarily mean having a NASA employee on the production line, or requiring all QA paperwork as a deliverable. But it does mean that NASA will have satisfied themselves that the F9 design, production, test and QA processes meet all their man-rating requirements, which means they have "visibility" into every single piece part on the rocket.

You seem fixated on the "buying a service, not a product" mantra without understanding that NASA is not going to put astronauts on any new commercial launch vehicle (yes, Soyuz is an exception) without "certifying" that it meets all their man-rating requirements, and such certification will necessarily require "visibility" into pretty much everything that goes into/onto an F9. And same with Dragon, of course.

[Jcc]

The fact that SpaceX did not try to sue the strut supplier indicates they recognize their own responsibility for not requiring 100% load testing for acceptance. We can assume the provider fulfilled  their obligation to deliver the product with specified testing.

[deltaV]

The fact that SpaceX did not try to sue the strut supplier indicates they recognize their own responsibility for not requiring 100% load testing for acceptance. We can assume the provider fulfilled  their obligation to deliver the product with specified testing.

Or they negotiated some compensation but agreed to keep fact that confidential. It's not safe to assume that we know about everything that occurs. A transfer of money is a lot easier to hide than a rocket.

[CyndyC]

The fact that SpaceX did not try to sue the strut supplier indicates they recognize their own responsibility for not requiring 100% load testing for acceptance. We can assume the provider fulfilled  their obligation to deliver the product with specified testing.

Or they negotiated some compensation but agreed to keep fact that confidential. It's not safe to assume that we know about everything that occurs. A transfer of money is a lot easier to hide than a rocket.

But how would or could a little strut manufacturer possibly compensate for the loss of a rocket & payload worth millions of dollars?

[deltaV]

The fact that SpaceX did not try to sue the strut supplier indicates they recognize their own responsibility for not requiring 100% load testing for acceptance. We can assume the provider fulfilled  their obligation to deliver the product with specified testing.

Or they negotiated some compensation but agreed to keep fact that confidential. It's not safe to assume that we know about everything that occurs. A transfer of money is a lot easier to hide than a rocket.

But how would or could a little strut manufacturer possibly compensate for the loss of a rocket & payload worth millions of dollars?

What's your source for the strut manufacturer being "little"?

[Jim]

1.  But you keep only focusing on raw material.  You seem to completely exclude the possibility that they would require component testing from a supplier - and I think it's because you don't think "material certification" can mean something different than what you have experienced.  That is a fragile line of reasoning...


2.  NASA has no visibility into the component level parts of a Falcon 9 - it is a commercial service, not a product that NASA is buying.  Do think NASA inspectors are on the Falcon 9 production line?

1.  Wrong.  Material cert is for the material not components.   Kabloona is 100% correct

2.  Yes, NASA does.  NASA doesn't look at each individual item, but it has looked at the processes that create the items.

Eample:

Bakery.  NASA does look at each cake but it looks at where the eggs, flour and sugar come from.  NASA would then look at how they are combined into cakes. 

Also, when it comes to a specific cake for NASA (aka Jason-3 booster), NASA watches how it is built. 

[llanitedave]

NASA would definitely not want one of my wife's cakes, then.  S'ok.  More for me!

[CyndyC]

But how would or could a little strut manufacturer possibly compensate for the loss of a rocket & payload worth millions of dollars?

What's your source for the strut manufacturer being "little"?

Albert Einstein, Theory of Relativity

[mfck]

But how would or could a little strut manufacturer possibly compensate for the loss of a rocket & payload worth millions of dollars?

What's your source for the strut manufacturer being "little"?

Albert Einstein, Theory of Relativity

First Principles all the way, huh?

[Kabloona]

Some new and interesting videos related to SpaceX's systems engineering approach. And Dan Rasky has a few other SpaceX- and Blue- Origin-related videos worth checking out on YouTube.

[Craftyatom]

Some new and interesting videos related to SpaceX's systems engineering approach. And Dan Rasky has a few other SpaceX- and Blue- Origin-related videos worth checking out on YouTube.

Youtube videos aren't showing for me, so here are some links for those having the same issue:
youtube.com/watch?v=xCHuT7r38Es
youtube.com/watch?v=SMLDAgDNOhk
youtube.com/watch?v=I0qgBuIO6n4

[catdlr]

People need to stop posting videos with the shared youtube link (i.e.,  youtube.be), and use the link yoututbe.com/watch?v= instead.  Chris did installed a conversion marco for this, but it seems that post above got past it.

[Roy_H]

The fact that SpaceX did not try to sue the strut supplier indicates they recognize their own responsibility for not requiring 100% load testing for acceptance. We can assume the provider fulfilled  their obligation to deliver the product with specified testing.

Or they negotiated some compensation but agreed to keep fact that confidential. It's not safe to assume that we know about everything that occurs. A transfer of money is a lot easier to hide than a rocket.

But how would or could a little strut manufacturer possibly compensate for the loss of a rocket & payload worth millions of dollars?

Totally out of the question. Warranties are for the product, and never cover losses as a result of use. The company would be required to replace the defective bolts/struts, nothing more.

[savuporo]

What i have learned on this forum is that, most of the avid followers of the company favor replacing CDRs with LDRs ( where C used to stand for critical, and L will be laudatory )