SpaceX Systems Engineering

[Kabloona]

And this is important because if you assume that "Material Certification" only means the raw material portion of a finished purchased part, and not that the finished part itself meets ALL the specifications of the engineering documentation that it was designed to handle...

That's what "materials certification" means within NASA, and I'd be very surprised if it meant anything different inside SpaceX given how many of their employees have prior experience with NASA standards and the close coordination SpaceX has with their customer...not to mention the requirements NASA is/will be levying on manned F9's.

The way the finished part is determined to meet ALL the specifications is by testing and/or inspection that verifies each spec requirement, and that isn't called "material certification," it's called "acceptance testing," ie verifying that the received part meets all specs before the buyer formally accepts the item.

 If it does, it passes "acceptance," and the acceptance test paperwork, along with the materials certs, is eventually incorporated into the vehicle build log. If it doesn't pass acceptance tests, the part is rejected.

But on all the programs I've worked (Pegasus, Taurus, Transfer Orbit Stage), the "materials certifications" from vendors were always certification only of raw materials properties and never implied verification of part functionality, etc. Those spec requirements were always verified by separate acceptance tests or inspections.

[AncientU]

I have no insight into those numbers, but a post on parabolicarc says NASA has verified the Falcon 1 and Falcon 9 development cost numbers:

SpaceX has publicly indicated that the development cost for Falcon 9 launch vehicle was approximately $300 million. Additionally, approximately $90 million was spent developing the Falcon 1 launch vehicle which did contribute to some extent to the Falcon 9, for a total of $390 million. NASA has verified these costs.

http://www.parabolicarc.com/2011/05/31/nasa-analysis-falcon-9-cheaper-traditional-approach/

And according to that same article, NASA's own analysis said it would have cost them between $1.7B and $4.0B to develop F9. So SpaceX beat the NASA development cost estimate by at least a factor of 5, and possibly a factor of 10+.

Extremely impressive.

The interesting thing is that they are accounting for F-1 contributions to F-9 development; also including Dragon costs.  We've heard time and again that NASA doesn't have to include past development costs in the SLS/Orion accounting -- things such as the engines, the solids, the tankage, the capsule, the heat shield, the ACS, ... wait, that's almost all of SLS technology developed before the first SLS weld.

If we'd account NASA costs with the same measure, the number would be astronomically higher!

[oldAtlas_Eguy]

In looking at the design development process SpaceX uses it reminds me of the tremendous differences in the Spiral vs Waterfall software development methods. Spiral is design-implement-test-deploy-design-implement-test-deploy... where Waterfall is study-analyze-design-implement-test-deploy-maintain. If the design falls short it is very difficult to overcome these design faults in maintenance phase of the traditional Waterfall method where only the lowest levels are reworked to meet the higher level designs without starting all over, whereas in the Spiral methodology redesign at higher levels is part of the process of evolving the design to make it better and better.

Also the time between design start and deploy is very short in the spiral vs the long time for the waterfall. Since time is money also each iteration in the Spiral is a lot cheaper than the waterfall to get to a deployed system. The drawback is that the Spiral method deploys the minimum system and increases it in each iteration where waterfall deploys the complete end post system.

From the historical of SpaceX improving the F9 every 2.5 years v1.0->v1.1->FT they are definitely using the Spiral vs the design and done Waterfall that NASA and other LV developers use.

[JamesH]

In looking at the design development process SpaceX uses it reminds me of the tremendous differences in the Spiral vs Waterfall software development methods. Spiral is design-implement-test-deploy-design-implement-test-deploy... where Waterfall is study-analyze-design-implement-test-deploy-maintain. If the design falls short it is very difficult to overcome these design faults in maintenance phase of the traditional Waterfall method where only the lowest levels are reworked to meet the higher level designs without starting all over, whereas in the Spiral methodology redesign at higher levels is part of the process of evolving the design to make it better and better.

Also the time between design start and deploy is very short in the spiral vs the long time for the waterfall. Since time is money also each iteration in the Spiral is a lot cheaper than the waterfall to get to a deployed system. The drawback is that the Spiral method deploys the minimum system and increases it in each iteration where waterfall deploys the complete end post system.

From the historical of SpaceX improving the F9 every 2.5 years v1.0->v1.1->FT they are definitely using the Spiral vs the design and done Waterfall that NASA and other LV developers use.

With Musk coming directly from a software background, this approach should be of no surprise. I'm guessing, but I would think Musk looked at the incumbents (mostly/all waterfall?), and thought they were beatable by a more agile company.

Time will tell, but he's made a good start.

[Coastal Ron]

And this is important because if you assume that "Material Certification" only means the raw material portion of a finished purchased part, and not that the finished part itself meets ALL the specifications of the engineering documentation that it was designed to handle...

That's what "materials certification" means within NASA...

Which is what I mean.  You are trying to extrapolate how the private sector (and actually a unique sub-segment of it) works by applying government rules to it.  That doesn't work.

...and I'd be very surprised if it meant anything different inside SpaceX given how many of their employees have prior experience with NASA standards and the close coordination SpaceX has with their customer...not to mention the requirements NASA is/will be levying on manned F9's.

What percentage of their early employees are you figuring came from NASA supplier quality?  And you're assuming the NASA sets SpaceX standards, which we know from other interactions SpaceX has had with the government doesn't happen - SpaceX does what SpaceX feels is the best way (as do most government contractors).

The way the finished part is determined to meet ALL the specifications is by testing and/or inspection that verifies each spec requirement, and that isn't called "material certification," it's called "acceptance testing," ie verifying that the received part meets all specs before the buyer formally accepts the item.

Again, without knowing the nomenclature SpaceX uses for all of it's processes it's impossible to equate anything to anything.

But on all the programs I've worked (Pegasus, Taurus, Transfer Orbit Stage), the "materials certifications" from vendors were always certification only of raw materials properties and never implied verification of part functionality, etc. Those spec requirements were always verified by separate acceptance tests or inspections.

You're talking about internal processes.  The question you raise about "Material Certification" is whether the supplier does any inspection or validation for what they produce.  You imply "NO", that all they would do is supply a raw material certification and leave it at that - if the part is wrong, then the customer has to catch it.  I don't believe this is what happens.

[sdsds]

Yes, there's team spirit, but there's also individual accountability, and someone inside SX made the engineering/cost decision not to require 100% load testing of each strut as an acceptance criterion. And after the failure I'll bet there were a lot of SXer's breathing a quiet sigh of relief that they were not "that person."

From my experience the other possibility is that no one was "that person." Somehow a set of acceptance test criteria became the default (perhaps they were offered by the external vendor) and no one internally was tasked with asking, "Are we sure this set of criteria meet SpaceX engineering standards?"

Or is there an indication that someone at SpaceX once explicitly asserted that 100% load testing was not required for that strut?

[Kabloona]

You are trying to extrapolate how the private sector (and actually a unique sub-segment of it) works by applying government rules to it.  That doesn't work.

It does when SpaceX's main customer is NASA. They have to speak the same language to some extent, and meet NASA requirements for man-rating F9. What you're suggesting is that SpaceX doesn't understand what NASA means by a "materials certification" or has a different definition of it, when "materials certification" is going to be a huge paperwork requirement levied by NASA for manned F9 and crewed Dragon flights. In my view it's highly unlikely that SpaceX doesn't understand or has a different definition.

The question you raise about "Material Certification" is whether the supplier does any inspection or validation for what they produce.  You imply "NO", that all they would do is supply a raw material certification and leave it at that - if the part is wrong, then the customer has to catch it.  I don't believe this is what happens.

What I said earlier was:

Maybe the strut vendor did some random sample load testing in each batch of struts, but such testing should not be confused with "materials certification" because they are two completely different things, and someone like Elon would never use the very specific term "materials certification" to mean random sample load testing on a batch of finished struts.

And according to Elon, whatever inspection or testing was done, they did not require 100% load testing of *every* strut. Which is kind of the bottom line. Whatever they did, whatever materials certs they provided to SpaceX from the raw materials supplier, it was not sufficient to weed out a bad strut.

So I'm not sure what we're debating any more except the definition of a "materials certification," and I'll let it go at that.

[AncientU]

From what I've heard quoted, SpaceX isn't focused on process -- in fact, I believe process falls into the same category as acronyms... avoid them.  Focus of of EM's hiring is on product, not process, and hands-on performance.

When SpaceX was getting certified by the USAF, it was ding'd for not even having a Systems Engineering function.  Strikes me as funny we'd have a thread on it. 

[Kabloona]

Or is there an indication that someone at SpaceX once explicitly asserted that 100% load testing was not required for that strut?

Every procurement contract has the acceptance criteria written into it. Someone at SpaceX wrote that contract, with the acceptance criteria for the strut either specified or omitted. Usually a contracts person writes the contract and incorporates the acceptance criteria as written/specified by the cognizant engineer.

You can debate whether this was a sin of omission or a sin of commission, but that's beside the point. The point is that it's up to SpaceX to write the required acceptance criteria into the contract. Obviously they didn't require 100% load testing. That was either a conscious decision or an oversight by one or more people.

[CyndyC]

In looking at the design development process SpaceX uses it reminds me of the tremendous differences in the Spiral vs Waterfall software development methods. Spiral is design-implement-test-deploy-design-implement-test-deploy... where Waterfall is study-analyze-design-implement-test-deploy-maintain. If the design falls short it is very difficult to overcome these design faults in maintenance phase of the traditional Waterfall method where only the lowest levels are reworked to meet the higher level designs without starting all over, whereas in the Spiral methodology redesign at higher levels is part of the process of evolving the design to make it better and better.

Also the time between design start and deploy is very short in the spiral vs the long time for the waterfall. Since time is money also each iteration in the Spiral is a lot cheaper than the waterfall to get to a deployed system. The drawback is that the Spiral method deploys the minimum system and increases it in each iteration where waterfall deploys the complete end post system.

From the historical of SpaceX improving the F9 every 2.5 years v1.0->v1.1->FT they are definitely using the Spiral vs the design and done Waterfall that NASA and other LV developers use.

With Musk coming directly from a software background, this approach should be of no surprise. I'm guessing, but I would think Musk looked at the incumbents (mostly/all waterfall?), and thought they were beatable by a more agile company.

Time will tell, but he's made a good start.

I'm not doubting that SpaceX uses Spiral, from the way Atlas described it and Waterfall, but it would be interesting to know how much Elon Musk has had the time & inclination to stay hands on with the computer programming at SpaceX (and at Tesla for that matter). Even though some of the basics haven't changed, don't forget much of his software background was old school, 80s & 90s. PayPal coding may have been the last of it, and I think he had moved on to designing rockets & electric cars before PayPal was sold to eBay in 2002.

[Coastal Ron]

You are trying to extrapolate how the private sector (and actually a unique sub-segment of it) works by applying government rules to it.  That doesn't work.

It does when SpaceX's main customer is NASA.

The main customer for Dragon maybe, but not Falcon 9.  And most of the early NASA money was not for products or delivery services, it was for completing engineering or program milestones for Dragon related development (i.e. COTS, CCDev, etc.).  NASA had no monetary contribution to any SpaceX rocket.

They have to speak the same language to some extent, and meet NASA requirements for man-rating F9.

Unlike government contracts where a contractor is delivering a product, SpaceX delivers a service, so NASA is never involved in the build side of SpaceX - there is no "NASA Falcon 9".  Hence the "Commercial" in Commercial Cargo and Commercial Crew.  NASA doesn't dictate the processes and procedures that the companies use, and the only "deliverable" is a service.

So why would SpaceX need to craft their entire Falcon 9 quality system around a customer that will never have influence over the Falcon 9?

What you're suggesting is that SpaceX doesn't understand what NASA means by a "materials certification" or has a different definition of it, when "materials certification" is going to be a huge paperwork requirement levied by NASA for manned F9 and crewed Dragon flights. In my view it's highly unlikely that SpaceX doesn't understand or has a different definition.

SpaceX provides a service, not a product.  Why does SpaceX need to deliver any sort of "materials certification"?

[Kabloona]

SpaceX provides a service, not a product.  Why does SpaceX need to deliver any sort of "materials certification"?

We've already seen the in-depth review process that SpaceX had to go through with the Air Force to certify the F9 for DoD. There's no way they got the F9 certified without being able to show the Air Force they have basic QA processes like requiring materials certifications for the raw materials in critical components like tanks. That's just a basic part of a QA program, and if you can't even do a basic thing like verify the source, composition and strength of your various raw materials, there is simply no way the Air Force is going to trust you with a $1B payload.

Same thing for NASA. No way they are going to allow their astronauts on an F9 unless they can verify it meets all their man-rating requirements, including having materials certifications and traceability per the usual NASA standards for manned spaceflight.

The materials certs may not be actual contract deliverables, but SpaceX will certainly have to show NASA that they have adequate QA processes in place for manned missions, and materials certifications are typically one piece of that process.

[Coastal Ron]

SpaceX provides a service, not a product.  Why does SpaceX need to deliver any sort of "materials certification"?

We've already seen the in-depth review process that SpaceX had to go through with the Air Force to certify the F9 for DoD. There's no way they got the F9 certified without being able to show the Air Force they have basic QA processes like requiring materials certifications for the raw materials in critical components like tanks. That's just a basic part of a QA program, and if you can't even do a basic thing like verify the source, composition and strength of your various raw materials, there is simply no way the Air Force is going to trust you with a $1B payload.

And since they were certified, they obviously they were able to show that they have processes to validate their quality, and that those systems are institutional.

But you keep only focusing on raw material.  You seem to completely exclude the possibility that they would require component testing from a supplier - and I think it's because you don't think "material certification" can mean something different than what you have experienced.  That is a fragile line of reasoning...

Same thing for NASA. No way they are going to allow their astronauts on an F9 unless they can verify it meets all their man-rating requirements, including having materials certifications and traceability per the usual NASA standards for manned spaceflight.

NASA has no visibility into the component level parts of a Falcon 9 - it is a commercial service, not a product that NASA is buying.  Do think NASA inspectors are on the Falcon 9 production line?

[whitelancer64]

Or is there an indication that someone at SpaceX once explicitly asserted that 100% load testing was not required for that strut?

*snip* The point is that it's up to SpaceX to write the required acceptance criteria into the contract. Obviously they didn't require 100% load testing. That was either a conscious decision or an oversight by one or more people.

Nor should they have. Batch testing should be more than sufficient to catch gross defects in lots of parts, since the part was over-designed by a factor of 3.

[Nomadd]

Or is there an indication that someone at SpaceX once explicitly asserted that 100% load testing was not required for that strut?

*snip* The point is that it's up to SpaceX to write the required acceptance criteria into the contract. Obviously they didn't require 100% load testing. That was either a conscious decision or an oversight by one or more people.

Nor should they have. Batch testing should be more than sufficient to catch gross defects in lots of parts, since the part was over-designed by a factor of 3.

When a defect, like the one being discussed, can take out a rocket if it appears once in 5,000 parts, your statement doesn't seem to make much sense. Over design or margin is irrelevant. A gross defect like the one they had can take out a part no matter how strong it's suppose to be.

[CyndyC]

As much as I wanted to see and previously thought I saw that SpaceX was off the hook, Kabloona's and now Nomadd's viewpoints seem a bit more realistic. They also shed some new light that is a better match for the gravity of Elon Musk's demeanor during the post CRS-7 public update a month later.

I had time to look up a few things today that MIGHT help draw some conclusions, however much is going to be the public's business in the end anyway. I won't try to sort out a bunch of quotes from the previous posts, but addressing Coastal Ron's insistence that SpaceX is only a service and that NASA would have nothing to do with the rocket itself, there were headlines last May that might have gone under the radar, because they came out only about 10 days before SpaceX was certified by the Air Force. This from another news site "The Falcon 9 is now certified by NASA as a “Category 2” launch vehicle in the agency’s nomenclature....The milestone clears the Falcon 9 rocket to launch NASA’s “medium-risk” science missions."

The cert process began in 2012, originally so that SpaceX could launch Jason 3 "in July [2015]". It goes on, "SpaceX originally bid to launch Jason 3 with the older Falcon 9 v1.0 version of its workhorse rocket, but switched to the more powerful Falcon 9 v1.1, featuring upgraded Merlin engines and other changes. NASA officials said the rocket swap forced engineers to redo part of the certification work, which includes management, process and engineering audits of the contractor."

The certification was concluded by NASA's Launch Services Program, LSP, Jim's dept incidentally. But there was also this, "SpaceX has successfully launched six resupply missions to the International Space Station under contract to NASA, but the agency arranged the launches through the space station program and bypassed the certification needed for science missions."

Both the LSP cert requirements and the ISS "arrangement" requirements can probably be found on the internet for anyone motivated to do more digging. Seems there might be some question as to which rocket under which clearance was used for CRS-7, considering the timing.

[Kabloona]

You seem to completely exclude the possibility that they would require component testing from a supplier...

I don't know why you keep saying that, when at least twice above I've said that it's possible SpaceX required random sample load testing of several struts from each batch.

But they did not require 100% load testing of each strut. That was the critical requirement that should have been written into the contract as a required acceptance test for each strut, and was not. And that lapse was the "complacency" Elon referred to.

But you keep only focusing on raw material. 

Only to explain what the term "materials certification" means to NASA and the two launch vehicle companies I've worked for/with. It means certification of the properties of the raw material that went into the part.  It does not mean certification that the finished part meets all its functional requirements. That's why it's called a "materials" certification and not a "part" or "component" certification.

NASA has no visibility into the component level parts of a Falcon 9 - it is a commercial service, not a product that NASA is buying.

How will NASA possibly certify F9 for man-rating, with "no visibility" into component level parts?  And "visibility" does not necessarily mean having a NASA employee on the production line, or requiring all QA paperwork as a deliverable. But it does mean that NASA will have satisfied themselves that the F9 design, production, test and QA processes meet all their man-rating requirements, which means they have "visibility" into every single piece part on the rocket.

You seem fixated on the "buying a service, not a product" mantra without understanding that NASA is not going to put astronauts on any new commercial launch vehicle (yes, Soyuz is an exception) without "certifying" that it meets all their man-rating requirements, and such certification will necessarily require "visibility" into pretty much everything that goes into/onto an F9. And same with Dragon, of course.

[Jcc]

The fact that SpaceX did not try to sue the strut supplier indicates they recognize their own responsibility for not requiring 100% load testing for acceptance. We can assume the provider fulfilled  their obligation to deliver the product with specified testing.

[deltaV]

The fact that SpaceX did not try to sue the strut supplier indicates they recognize their own responsibility for not requiring 100% load testing for acceptance. We can assume the provider fulfilled  their obligation to deliver the product with specified testing.

Or they negotiated some compensation but agreed to keep fact that confidential. It's not safe to assume that we know about everything that occurs. A transfer of money is a lot easier to hide than a rocket.

[CyndyC]

The fact that SpaceX did not try to sue the strut supplier indicates they recognize their own responsibility for not requiring 100% load testing for acceptance. We can assume the provider fulfilled  their obligation to deliver the product with specified testing.

Or they negotiated some compensation but agreed to keep fact that confidential. It's not safe to assume that we know about everything that occurs. A transfer of money is a lot easier to hide than a rocket.

But how would or could a little strut manufacturer possibly compensate for the loss of a rocket & payload worth millions of dollars?