OIG Report on CRS-7 Failure

[Brovane]

Since the last topic on this got locked because of straying off topic I hope we can keep this one on topic and not stray into articles about the failure and stick to what was in the report. 

https://oig.nasa.gov/audits/reports/FY16/IG-16-025.pdf

What I find interesting in reading the report was the comments on the SpaceX Procedures during manufacturing.

From what I understand the USAF spent a lot of time reviewing SpaceX's procedures for F9 production as part of the EELV certification process.  Which is supposed to be fairly rigorous.  Did the USAF decide that the type of procedures below are ok that NASA has called out as a issue?  I cannot imagine the EELV auditors not seeing SpaceX personnel standing on flight hardware during assembly. 

 LSP also highlighted improper material selection and such practices as individuals standing on flight hardware during the assembly process, as possible contributing factors.

SpaceX has taken action to correct the deficiencies that led to the failed strut assembly and to address
NASA’s concerns by conducting inspections, replacing suspect parts, and conducting additional testing.
The company also reviewed the certifications of all spaceflight hardware and altered its quality control
processes to better align with NASA technical standards. In order to track completion of its corrective
actions, SpaceX is updating its process for identifying and resolving work-related tasks, which allows for
improved auditing, prioritizing, and tracking of fracturable hardware.

To administer its updated quality control process, SpaceX has reorganized into three teams called
“Design Reliability,” “Build Reliability,” and “Flight Reliability.” Besides monitoring corrective actions
taken as a result of the SPX-7 failure, these teams are tracking the significant upgrades SpaceX has made
to the Falcon 9 launch system for future launches, including increased thrust capability with a new fuel
mixture and corrective actions on software implementation plans, which are both rated as low risks by
the ISS Program.   

[Jim]

  Which is supposed to be fairly rigorous.  Did the USAF decide that the type of procedures below are ok that NASA has called out as a issue?  I cannot imagine the EELV auditors not seeing SpaceX personnel standing on flight hardware during assembly. 
 

The USAF process was no more rigorous than NASA's.  They are not hovering over every worker.   Certification looks at processes and completed work.  Not the work as it being done.

And standing on flight hardware isn't a standard practice.  Especially, one that can seen anytime that one walks on to the production floor.  Likely it was observed as a matter of chance, both in the actual occasion of standing on flight hardware and the observer being in the right place at the right time.

[Brovane]

  Which is supposed to be fairly rigorous.  Did the USAF decide that the type of procedures below are ok that NASA has called out as a issue?  I cannot imagine the EELV auditors not seeing SpaceX personnel standing on flight hardware during assembly. 
 

The USAF process was no more rigorous than NASA's.  They are not hovering over every worker.   Certification looks at processes and completed work.  Not the work as it being done.

And standing on flight hardware isn't a standard practice.  Especially, one that can seen anytime that one walks on to the production floor.  Likely it was observed as a matter of chance, both in the actual occasion of standing on flight hardware and the observer being in the right place at the right time.

Though the CRS payloads are not officially classified to a payload risk, they are considered like class D payloads which allow the highest risk for LV's. 

The USAF considers NSS payloads as class A payloads (The USAF doesn't officially place a risk category on payloads) which means EELV certified launch vehicles would basically be the equivalent of category 3 LV's for a NASA from a certification process if we looked at launch risk. 

Considering the difference in risk category between EELV payloads and CRS payloads wouldn't that imply that the USAF would be more rigorous than the process NASA uses for reviewing CRS LV's?   

[cuddihy]

I look forward to the day when launch operations are so routine that standing on flight hardware is nothing out of the ordinary--as long as the part in question is fairly sturdy.

Throwing it into the summary seems like a cheap shot of questionable relevance to the actual QA practices SpaceX has, but in my experience it is exactly the kind of thing the IG always puts in its summary report to support their theory/ narrative for the casual / management / political leaders who only read the summary. It's an anecdote supporting a minor part of the story, nothing more.

Frankly, as IG reports go, this one is actually pretty benign. When the IG gets called in and invests government time and thoughtful people into doing an investigation, they will always find something suboptimal in their view...the thrust of this one is about as positive about NASA's management, and about SpaceX and Orbital's performance, as you can expect.

I'm impressed by NASA's agility and good contract management in this case that the IG reports on. Gives me a lot more confidence as a taxpayer in NASA as an effective organization.

[Jim]

Considering the difference in risk category between EELV payloads and CRS payloads wouldn't that imply that the USAF would be more rigorous than the process NASA uses for reviewing CRS LV's?   

Mixing apples and oranges.  USAF has yet to use an F9 and NASA had certified F9 for category 3 and flown one.

[su27k]

I look forward to the day when launch operations are so routine that standing on flight hardware is nothing out of the ordinary--as long as the part in question is fairly sturdy.

Or having the whole manufacturing process instrumented to the point where such out of specs practice would raise alarm instantly. I mean everyone is already doing close out photos, seems to me video everything and analyze it would be the logical next step.

[meberbs]

Considering the difference in risk category between EELV payloads and CRS payloads wouldn't that imply that the USAF would be more rigorous than the process NASA uses for reviewing CRS LV's?

Considering that NASA will also using the Falcon 9 for crew, I don't think this is valid. I don't see anything in the remaining milestones to approve Falcon 9 for crew, so this likely has already happened, though it may be still left as part of one of the final approval milestones.

...
Frankly, as IG reports go, this one is actually pretty benign. When the IG gets called in and invests government time and thoughtful people into doing an investigation, they will always find something suboptimal in their view...the thrust of this one is about as positive about NASA's management, and about SpaceX and Orbital's performance, as you can expect.

I'm impressed by NASA's agility and good contract management in this case that the IG reports on. Gives me a lot more confidence as a taxpayer in NASA as an effective organization.

I still have to finish reading the report, but I agree with this. Something went wrong, so any reasonable investigation should have suggestions for improvement on both the NASA and SpaceX ends. This report does not seem excessively critical, and doesn't seem to suggest that the entire contracting arrangement was inappropriate or anything like that.

[Brovane]

Considering the difference in risk category between EELV payloads and CRS payloads wouldn't that imply that the USAF would be more rigorous than the process NASA uses for reviewing CRS LV's?   

Mixing apples and oranges.  USAF has yet to use an F9 and NASA had certified F9 for category 3 and flown one.

Which Class-A NASA payload has already flown on the F9? 

I was also not aware that NASA had certified F9 as a category 3 LV.  The last mention I can find is for category 2 certification.  When did NASA certify the F9 as category 3 LV? 

[abaddon]

Which Class-A NASA payload has already flown on the F9? 

I was also not aware that NASA had certified F9 as a category 3 LV.  The last mention I can find is for category 2 certification.  When did NASA certify the F9 as category 3 LV?

NASA LSP contracted the Jason-3 launch with SpaceX.  It was a Category 2 required payload, not 3.  I couldn't find a reference to the class but my recollection was it was B or C.  NASA LSP has also contracted the TESS launch with SpaceX; that is a Category 2 required payload, Class C.

I don't believe NASA has announced certification of the Falcon 9 as available for a payload requiring Category 3 certification, or available for Class A payloads.

[abaddon]

As an aside, in reading this report, it seems NASA got a much better deal from SpaceX than OrbitalATK, both overall for CRS1 and also in concessions following the respective LOVs from each organization.  That certainly seems to be the determination of the OIG.  It is perhaps unfair to come to that conclusion, but it seems to be clear from the information in the report.  Is there anything I'm missing here?

[abaddon]

Mixing apples and oranges.  USAF has yet to use an F9 and NASA had certified F9 for category 3 and flown one.

Technically, the USAF purchased the DSCOVR launch for NASA/NOAA, but it was under the OSP-3 program, which allows for higher-risk vehicles.  As noted above, NASA has yet to certify Falcon 9 for payloads requiring Category 3 classification; or if they have, I have apparently missed it.

[LouScheffer]

My confidence (perhaps misplaced) is higher for a fix being the right one, if the rocket has significant commercial business (as SpaceX does, but Orbital does not).  I think this helps since then there are two more groups asking hard questions about the problem and the fix - the customers and the insurers.   Furthermore I suspect commercial contracts are less forgiving in case of launch mishap (If I remember right, Orbital got most of the payment from NASA even when their launch failed.  I assume this applies to all vendors.)

Also, the report makes a big deal about potential conflict of interest.  I don't see this - it seems to me that everyone's interests are the same.  It does the company no good to claim it's not their fault if their rocket fails again.  To me, a bigger worry with the company leading the investigation is technical blind spots.   Sometimes if you know how something should work, you see that rather than how it might fail.  Surely there must be at least some history on this - when review boards have failed (identified the wrong root cause), why did they make this mistake?  Has lack of independence ever been responsible for this?

[baldusi]

Closest I can think of is the Taurus fairing failures. First one got it wrong, the second kind of too, but they kept investigating something like three years and finally identified the problem.
It was a secondary effect on a batch of aluminum stock used to machine the frangible joints. These metallurgy problems changes the was the fracture propagated, and thus never fully separated.
It was, apparently, a new mode of failure in the metallurgy that they didn't controlled for because they didn't understood how it contributed to fracture propagation.

[catdlr]

NASA negotiates cheaper mission prices after last year's SpaceX failure

http://www.latimes.com/business/la-fi-nasa-spacex-report-20160630-snap-story.html

NASA negotiated discounted mission prices with SpaceX after one of the Hawthorne company’s rockets broke apart last June while laden with supplies for the International Space Station, according to a report by the space agency’s office of inspector general.

[Robotbeat]

NASA negotiates cheaper mission prices after last year's SpaceX failure

http://www.latimes.com/business/la-fi-nasa-spacex-report-20160630-snap-story.html

NASA negotiated discounted mission prices with SpaceX after one of the Hawthorne company’s rockets broke apart last June while laden with supplies for the International Space Station, according to a report by the space agency’s office of inspector general.

This is a good thing, IMO.

It keeps NASA happy using SpaceX (which is good for SpaceX), it gives the US citizen a better deal, and it provides SpaceX incentive to think long and hard about how to improve reliability (not that they really needed any extra motivation).

[LouScheffer]

Also, the report makes a big deal about potential conflict of interest.  I don't see this - it seems to me that everyone's interests are the same.  It does the company no good to claim it's not their fault if their rocket fails again.  To me, a bigger worry with the company leading the investigation is technical blind spots.   Sometimes if you know how something should work, you see that rather than how it might fail.  Surely there must be at least some history on this - when review boards have failed (identified the wrong root cause), why did they make this mistake?  Has lack of independence ever been responsible for this?

Here's an interesting article about why AC-70 failure board failed.  The author (a government employee) implies that private industry simply does not have the competence in this area:

However, the AC-70 failure was different in a very basic but non-obvious way. The failure investigation was conducted by a private firm, GD, rather than led by the government. This was a first, and it produced another first, a failed investigation.  [...]  Not surprisingly, the government was somewhat more involved in the AC-71 investigation.

However, in the comments to the article, another person who was there blamed the failure on the lead investigator being wedded to a particular hypothesis. 

Both of these seem like reasonable worries - any given company does not do a lot of accident investigation, and being you need an objective view of all possible causes of the failure.   But neither of these seems like a conflict of interest.

[Jim]

Here's an interesting article about why AC-70 failure board failed.  The author (a government employee) implies that private industry simply does not have the competence in this area:

However, the AC-70 failure was different in a very basic but non-obvious way. The failure investigation was conducted by a private firm, GD, rather than led by the government. This was a first, and it produced another first, a failed investigation.  [...]  Not surprisingly, the government was somewhat more involved in the AC-71 investigation.

That was maybe true in the 90's but not now. Industry is competent enough to do accident investigations itself.

[Coastal Ron]

I look forward to the day when launch operations are so routine that standing on flight hardware is nothing out of the ordinary--as long as the part in question is fairly sturdy.

Or having the whole manufacturing process instrumented to the point where such out of specs practice would raise alarm instantly. I mean everyone is already doing close out photos, seems to me video everything and analyze it would be the logical next step.

HD video can't show the strengths or weaknesses within a component (such as an under-strength strut), or the incorrect software in a electronic module.

And in general, you can't inspect quality in.

Now that SpaceX is recovering some of their 1st stages they will be able to have more validation of their processes and procedures.  Having better feedback loops is what will probably be their biggest path to higher quality.

[Comga]

Here's an interesting article about why AC-70 failure board failed.  The author (a government employee) implies that private industry simply does not have the competence in this area:

However, the AC-70 failure was different in a very basic but non-obvious way. The failure investigation was conducted by a private firm, GD, rather than led by the government. This was a first, and it produced another first, a failed investigation.  [...]  Not surprisingly, the government was somewhat more involved in the AC-71 investigation.

That was maybe true in the 90's but not now. Industry is competent enough to do accident investigations itself.

And it's not like "government" accident investigation boards are infallible.  Just attended a lecture on the incident where water leaking into ISS astronaut Luca Parmitano's spacesuit during a spacewalk nearly drowned him.  Despite being the highest criticality, the previous anomaly investigation got it totally wrong.  (They thought it was the water bottle leaking, when it was actually from the life support system.)

All accident investigations deal with limited information and limited resources.  More time can always be spent, and will be seen as necessary in hindsight whenever a wrong conclusion is later proven.  Conversely, no review board will ever say that an investigation team spent too much time or effort.

[LouScheffer]

I look forward to the day when launch operations are so routine that standing on flight hardware is nothing out of the ordinary--as long as the part in question is fairly sturdy.

Or do it like airplanes.  You can step on certain parts, like the wings, provided you avoid the "no step" area.  And there must be some explicit analysis to show it *is* safe to step on the allowed areas.

It sometime surprising where stepping is allowed.  Here are folks standing on the mirror of an optical telescope, and cleaning it with a mop.