Author Topic: Evaluation of the Evolved Expendable Launch Vehicle Program Quality Management  (Read 2348 times)

The Department of Defense Inspector General conducted an assessment of whether ULA, SpaceX and AJR performed adequate quality assurance management for the EELV program recently. In short, here are the findings:

Quote
We found that ULA, SpaceX, and AR did not perform adequate quality assurance management of the EELV program, as evidenced by the 181 non-conformities to December 20, 2017 the AS9100C at the EELV contractor production facilities. 
For example, we found that ULA, SpaceX, and AR failed to comply with AS9100C, section 7.5.5, Preservation of Product. 
At ULA, we found nonconformities related to Electrostatic Sensitive Device (ESD) protection in the avionics production area.  We found ungrounded ESD workstations, untested wrist straps, missing ESD protective covers, non-ESD-approved containers and materials, and uncontrolled humidity levels.  Inadequate ESD controls and mitigation could result in the premature failure of electronic components in the EELV system.  At SpaceX, we found an inadequately protected Merlin engine on the test stand.  The Merlin engine exhaust ports and vent tubes should have been protected with specific covers.  Furthermore, we found bottles of soda and personal items in FOD-controlled areas.  At AR, we found that the RL-10 engine test stand, used to test both the Delta IV and Atlas V second stage engine, had significant FOD issues, including loose bolts, nuts, tape, foil, tie wraps, and animal feces.  Inadequate control of FOD significantly increases the risk of damage to EELV hardware, which can lead to costly rework and schedule impact.  ULA’s, SpaceX’s, and AR’s inadequate quality assurance management could increase program costs, delay launch
schedules, and increase the risk of mission failure.
 

In general, it seems all have tried to cut corners in one way or another.

Link to the PDF is here: https://media.defense.gov/2017/Dec/22/2001860659/-1/-1/1/DODIG-2018-045_REDACTED.PDF

And thank you to Zucal for finding it initially.
« Last Edit: 12/22/2017 11:46 PM by tvg98 »

Offline Targeteer

  • Senior Member
  • *****
  • Posts: 3871
  • near hangar 18
  • Liked: 982
  • Likes Given: 478
http://www.dodig.mil/reports.html/Article/1403539/evaluation-of-the-evolved-expendable-launch-vehicle-program-quality-management/

Evaluation of the Evolved Expendable Launch Vehicle Program Quality Management System

DODIG 2018-0453

Dec. 20, 2017 — 

Background

The DoD created the Evolved Expendable Launch Vehicle (EELV) program in response to the National Defense Authorization Act for Fiscal Year 1994, which directed the Secretary of Defense to develop and submit to Congress a plan for the “modernization of space launch capabilities for the DoD or, if appropriate, for the Government as a whole.” The EELV System Program Office (SPO) acquires launch services for U.S. military and intelligence spacecraft from ULA and SpaceX. AR provides ULA the RL-10 engine for use on the Delta IV and Atlas V. The EELV SPO delegates day-to-day contract and quality assurance management to the Defense Contract Management Agency (DCMA) through a memorandum of agreement (MOA).

Findings

We found that ULA, SpaceX, and AR did not perform adequate quality assurance management of the EELV program, as evidenced by the 181 nonconformities to the AS9100C at the EELV contractor production facilities. For example, we found that ULA, SpaceX, and AR failed to comply with AS9100C, section 7.5.5, Preservation of Product. At ULA, we found nonconformities related to Electrostatic Sensitive Device (ESD) protection in the avionics production area. We found ungrounded ESD workstations, untested wrist straps, missing ESD protective covers, non‑ESD approved containers and materials, and uncontrolled humidity levels. Inadequate ESD controls and mitigation could result in the premature failure of electronic components in the EELV system.

At SpaceX, we found an inadequately protected Merlin engine on the test stand. The Merlin engine exhaust ports and vent tubes should have been protected with specific covers. Furthermore, we found bottles of soda and personal items in FOD-controlled areas. At AR, we found that the RL-10 engine test stand, used to test both the Delta IV and Atlas V second stage engine, had significant FOD issues, including loose bolts, nuts, tape, foil, tie wraps, and animal feces. Inadequate control of FOD significantly increases the risk of damage to EELV hardware, which can lead to costly rework and schedule impact.

ULA’s, SpaceX’s, and AR’s inadequate quality assurance management could increase program costs, delay launch schedules, and increase the risk of mission failure.

Recommendations

We recommend that the Director, EELV SPO, and the Director, DCMA, conduct a root cause analysis and implement corrective actions for the 181 nonconformities identified during our evaluation and provide the DoD Office of Inspector General a copy of the root cause analysis and corrective action plan within 90 days of the issuance of this report.

Additionally, we recommend that the Director, EELV SPO, and the Director, DCMA, develop a corrective action plan to improve EELV quality assurance management to ensure that the EELV contractors comply with all AS9100C requirements and provide the DoD Office of Inspector General a copy of the corrective action plan within 90 days of the issuance of this report.
Best quote heard during an inspection, "I was unaware that I was the only one who was aware."

Offline rakaydos

  • Full Member
  • ****
  • Posts: 478
  • Liked: 190
  • Likes Given: 4
Quote
At SpaceX, we found an inadequately protected Merlin engine on the test stand.  The Merlin engine exhaust ports and vent tubes should have been protected with specific covers.  Furthermore, we found bottles of soda and personal items in FOD-controlled areas

At AR, we found that the RL-10 engine test stand, used to test both the Delta IV and Atlas V second stage engine, had significant FOD issues, including loose bolts, nuts, tape, foil, tie wraps, and animal feces. 
 

In general, it seems all have tried to cut corners in one way or another.

The comparison here is... wow. "Both sides the same," am I right?
« Last Edit: 01/04/2018 07:44 AM by Lar »

Online Lar

  • Fan boy at large
  • Global Moderator
  • Senior Member
  • *****
  • Posts: 10632
  • Saw Gemini live on TV
  • A large LEGO storage facility ... in Michigan
  • Liked: 7486
  • Likes Given: 5258
fix quotes please people...

Edit: I fixed some, but don't count on mods to fix your quotes. If preview shows a problem (you DO use preview, right??) then fix it before you post. If your post looks wrong, then fix it before you move on. Thank you.
« Last Edit: 01/04/2018 07:46 AM by Lar »
"I think it would be great to be born on Earth and to die on Mars. Just hopefully not at the point of impact." -Elon Musk
"We're a little bit like the dog who caught the bus" - Musk after CRS-8 S1 successfully landed on ASDS OCISLY

Offline Testraindrop

  • Member
  • Posts: 12
  • Germany
  • Liked: 13
  • Likes Given: 24
SpaceX'  "issues" should be an easy fix, don't forget to put protective covers on every engine and put a sign to leave personal items outside of FOD-controlled areas...

Offline brihath

  • Member
  • Full Member
  • ****
  • Posts: 877
  • Liked: 18
  • Likes Given: 20
SpaceX'  "issues" should be an easy fix, don't forget to put protective covers on every engine and put a sign to leave personal items outside of FOD-controlled areas...

Actually, putting up signs rarely fixes deficiencies like this.  Lean Six Sigma programs to include well developed procedures that are followed without exception, Standard Work and 5-S can provide a more lasting solution.  When properly implemented, they work.  I have seen measurable improvements when I have implemented these routines.

Online Lar

  • Fan boy at large
  • Global Moderator
  • Senior Member
  • *****
  • Posts: 10632
  • Saw Gemini live on TV
  • A large LEGO storage facility ... in Michigan
  • Liked: 7486
  • Likes Given: 5258
I read somewhere (not sure how much credence to give it) that sometimes easy to fix flaws are left deliberately to give the inspectors something to find, on the theory that they will find something and easy things are better. 

I'm really dubious of that but wanted to toss it out there. Perhaps if true, it is the sort of thing that could backfire. Because here, it seems easy to fix things are nevertheless being considered quite negative.
"I think it would be great to be born on Earth and to die on Mars. Just hopefully not at the point of impact." -Elon Musk
"We're a little bit like the dog who caught the bus" - Musk after CRS-8 S1 successfully landed on ASDS OCISLY

Online AncientU

  • Senior Member
  • *****
  • Posts: 6233
  • Liked: 4055
  • Likes Given: 5567
At what level does DoD have a say in how these vendors conduct their activities?
FoD rules, for instance, seem appropriate where you have high velocity air intakes (USAF ground ops), but what suction is there around an engine test stand?
"If we shared everything [we are working on] people would think we are insane!"
-- SpaceX friend of mlindner

Online AncientU

  • Senior Member
  • *****
  • Posts: 6233
  • Liked: 4055
  • Likes Given: 5567
Quote
At SpaceX, we found an inadequately protected Merlin engine on the test stand.  The Merlin engine exhaust ports and vent tubes should have been protected with specific covers.  Furthermore, we found bottles of soda and personal items in FOD-controlled areas

At AR, we found that the RL-10 engine test stand, used to test both the Delta IV and Atlas V second stage engine, had significant FOD issues, including loose bolts, nuts, tape, foil, tie wraps, and animal feces. 
 

In general, it seems all have tried to cut corners in one way or another.

The comparison here is... wow. "Both sides the same," am I right?

Not really enough information available to say 'the same.'  Cannot determine the extent of end-to-end process covered at each vendor.  For instance, the RD-180 and RS-68 production and test facilities were not inspected, though this accounts for half of the engines flown by ULA (not counting solids, thrusters, etc.) -- AJR deficiencies are part of the ULA scope.  Need also to understand what extent the end-to-end production process was reviewed -- SpaceX produces much in house, while ULA subs out much of its hardware production (sub-contractors for ULA were not included except for AJR/RL-10s).  Also don't know that the same inspectors went to each vendor facility and measured with the same ruler...

All one can say from this limited info is that none of these vendors are perfect or above reproach.
« Last Edit: 01/04/2018 08:19 PM by AncientU »
"If we shared everything [we are working on] people would think we are insane!"
-- SpaceX friend of mlindner

Tags: