There are far, far cheaper upper stages that could be used for the rocket's primary function to launch the Orion spacecraft to lunar orbit, including United Launch Alliance's reliable (and ready) Centaur V upper stage.
With Starship and New Glenn, NASA will also soon have two very powerful commercial super heavy lift rockets to draw upon.
While NASA requires its aerospace contractors to have quality assurance programs that comply with SAE International’s AS9100D standards on quality management systems, we found Boeing’s quality management system at Michoud does not adhere to these standards or NASA requirements. NASA engages DCMA to conduct surveillance of Boeing’s core and upper stage manufacturing efforts at Michoud, and when deficiencies in quality are found, DCMA issues Corrective Action Requests (CAR) to the contractor. CARs are labeled Level I through IV, with Level I the least serious deficiency. From September 2021 to September 2023, DCMA issued Boeing 71 Level I and II CARs, as well as a draft Level III CAR. According to DCMA officials, this is a high number of CARs for a space flight system at this stage in development and reflects a recurring and degraded state of product quality control. Boeing’s process to address deficiencies to date has been ineffective, and the company has generally been nonresponsive in taking corrective actions when the same quality control issues reoccur.Quality control issues at Michoud are largely due to the lack of a sufficient number of trained and experienced aerospace workers at Boeing. To mitigate these challenges, Boeing provides training and work orders to its employees. Considering the significant quality control deficiencies at Michoud, we found these efforts to be inadequate. For example, during our visit to Michoud in April 2023, we observed a liquid oxygen fuel tank dome—a critical component of the SLS Core Stage 3—segregated and pending disposition on whether and how it can safely be used going forward due to welds that did not meet NASA specifications. According to NASA officials, the welding issues arose due to Boeing’s inexperienced technicians and inadequate work order planning and supervision. The lack of a trained and qualified workforce increases the risk that Boeing will continue to manufacture parts and components that do not adhere to NASA requirements and industry standards.We project SLS Block 1B costs will reach approximately $5.7 billion before the system is scheduled to launch in 2028. This is $700 million more than NASA’s 2023 Agency Baseline Commitment, which established a cost and schedule baseline at nearly $5 billion. EUS development accounts for more than half of this cost, which we estimate will increase from an initial cost of $962 million in 2017 to nearly $2.8 billion through 2028. Boeing’s delivery of the EUS to NASA has also been delayed from February 2021 to April 2027, and when combined with other factors, suggests the September 2028 Artemis IV launch date could be delayed as well. Factors contributing to these cost increases and schedule delays include redirection of EUS funds to the core stage during Artemis I production, changing Artemis mission assignments, maintaining an extended workforce 7 years more than planned, manufacturing issues, and supply chain challenges...... Additionally, Boeing Defense, Space & Security’s EVMS, used by NASA for its Stages contract to measure cost and schedule progress, has been disapproved by the U.S. Department of Defense since 2020. DCMA has issued several Level II and III CARs for this EVMS, including a Level III CAR related to visibility into cost, schedule, and resource needs for several Boeing contracts, including Stages.
Regarding poor weld quality: the OIG seems to place responsibility on the workforce. Didn't Boeing invest in robotic weld tooling?
Budget Issues: The development costs for the EUS have tripled from the original estimate of $962 million in 2017 to $2.8 billion. The overall Block 1B development costs are expected to reach $5.7 billion, $700 million more than NASA’s recent estimate.
From the OIG report’s executive summary:QuoteWhile NASA requires its aerospace contractors to have quality assurance programs that comply with SAE International’s AS9100D standards on quality management systems, we found Boeing’s quality management system at Michoud does not adhere to these standards or NASA requirements. NASA engages DCMA to conduct surveillance of Boeing’s core and upper stage manufacturing efforts at Michoud, and when deficiencies in quality are found, DCMA issues Corrective Action Requests (CAR) to the contractor. CARs are labeled Level I through IV, with Level I the least serious deficiency. From September 2021 to September 2023, DCMA issued Boeing 71 Level I and II CARs, as well as a draft Level III CAR. According to DCMA officials, this is a high number of CARs for a space flight system at this stage in development and reflects a recurring and degraded state of product quality control. Boeing’s process to address deficiencies to date has been ineffective, and the company has generally been nonresponsive in taking corrective actions when the same quality control issues reoccur.Quality control issues at Michoud are largely due to the lack of a sufficient number of trained and experienced aerospace workers at Boeing. To mitigate these challenges, Boeing provides training and work orders to its employees. Considering the significant quality control deficiencies at Michoud, we found these efforts to be inadequate. For example, during our visit to Michoud in April 2023, we observed a liquid oxygen fuel tank dome—a critical component of the SLS Core Stage 3—segregated and pending disposition on whether and how it can safely be used going forward due to welds that did not meet NASA specifications. According to NASA officials, the welding issues arose due to Boeing’s inexperienced technicians and inadequate work order planning and supervision. The lack of a trained and qualified workforce increases the risk that Boeing will continue to manufacture parts and components that do not adhere to NASA requirements and industry standards.We project SLS Block 1B costs will reach approximately $5.7 billion before the system is scheduled to launch in 2028. This is $700 million more than NASA’s 2023 Agency Baseline Commitment, which established a cost and schedule baseline at nearly $5 billion. EUS development accounts for more than half of this cost, which we estimate will increase from an initial cost of $962 million in 2017 to nearly $2.8 billion through 2028. Boeing’s delivery of the EUS to NASA has also been delayed from February 2021 to April 2027, and when combined with other factors, suggests the September 2028 Artemis IV launch date could be delayed as well. Factors contributing to these cost increases and schedule delays include redirection of EUS funds to the core stage during Artemis I production, changing Artemis mission assignments, maintaining an extended workforce 7 years more than planned, manufacturing issues, and supply chain challenges...... Additionally, Boeing Defense, Space & Security’s EVMS, used by NASA for its Stages contract to measure cost and schedule progress, has been disapproved by the U.S. Department of Defense since 2020. DCMA has issued several Level II and III CARs for this EVMS, including a Level III CAR related to visibility into cost, schedule, and resource needs for several Boeing contracts, including Stages.Two important things unsaid in the report:1) What are the flight safety implications? EUS had unqualified welders doing shoddy work to the tune of 70+ corrective actions caught by Defense Contracts Management Agency, which is only looking at contract violations. What build errors outside contract violations have and have not been caught by Boeing and NASA? What is the plan for ensuring that all are found and corrected? What are the programmatic implications of that rework? What are the flight safety implications if that rework is not done (or still done poorly)?Orion/SLS already had a lousy (Shuttle ballpark) projected loss-of-crew estimate going forward, made worse by the system’s very infrequent use and what that does to workforce skills retention. What does this shoddy work and rework imply for flight safety? Are we at the point where alternatives to Orion/SLS need to be pursued for flight safety reasons alone?2) How far up the management chain does knowledge of these shenanigans go? At some point in those 70+ corrective actions from the Defense Contracts Management Agency, Steve Snell (Boeing’s EUS program manager) and James Savage (Boeing’s EUS chief engineer) must have known that they had hired/were continuing to hire unqualified welders and let them continue to work on EUS builds. Why didn’t they do something about it, like stand down until the project had the right welders with the right qualifications on board? According to the report, this shoddy work has cost the project years in schedule slips anyway, so why not stop work for a few months until the hiring was resolved? If they claim that they never knew after 70+ corrective actions, then why the heck not? Are they really that out-of-touch with their own project? Either way, will Snell and Savage get booted from the project and replaced with competent, ethical managers?Same goes for the Boeing and NASA managers leading SLS, John Honeycutt and David Dutcher, respectively. What did they know? When did they know it? Why didn’t they do anything about it? Will they get the boot?Same goes for ESDMD leadership. What did Free and Koerner know? When did they know it? Why didn’t they do anything about it?And a bonus third point of outrage — why on God’s green Earth does a major US defense contractor not have a working earned value management system in the 21st freaking century?!?! EVM was invented in 1967. It’s been over half a century, for Heaven’s sake!Cripes...
Quote from: VSECOTSPE on 08/09/2024 05:52 amFrom the OIG report’s executive summary:QuoteWhile NASA requires its aerospace contractors to have quality assurance programs that comply with SAE International’s AS9100D standards on quality management systems, we found Boeing’s quality management system at Michoud does not adhere to these standards or NASA requirements. NASA engages DCMA to conduct surveillance of Boeing’s core and upper stage manufacturing efforts at Michoud, and when deficiencies in quality are found, DCMA issues Corrective Action Requests (CAR) to the contractor. CARs are labeled Level I through IV, with Level I the least serious deficiency. From September 2021 to September 2023, DCMA issued Boeing 71 Level I and II CARs, as well as a draft Level III CAR. According to DCMA officials, this is a high number of CARs for a space flight system at this stage in development and reflects a recurring and degraded state of product quality control. Boeing’s process to address deficiencies to date has been ineffective, and the company has generally been nonresponsive in taking corrective actions when the same quality control issues reoccur.Quality control issues at Michoud are largely due to the lack of a sufficient number of trained and experienced aerospace workers at Boeing. To mitigate these challenges, Boeing provides training and work orders to its employees. Considering the significant quality control deficiencies at Michoud, we found these efforts to be inadequate. For example, during our visit to Michoud in April 2023, we observed a liquid oxygen fuel tank dome—a critical component of the SLS Core Stage 3—segregated and pending disposition on whether and how it can safely be used going forward due to welds that did not meet NASA specifications. According to NASA officials, the welding issues arose due to Boeing’s inexperienced technicians and inadequate work order planning and supervision. The lack of a trained and qualified workforce increases the risk that Boeing will continue to manufacture parts and components that do not adhere to NASA requirements and industry standards.We project SLS Block 1B costs will reach approximately $5.7 billion before the system is scheduled to launch in 2028. This is $700 million more than NASA’s 2023 Agency Baseline Commitment, which established a cost and schedule baseline at nearly $5 billion. EUS development accounts for more than half of this cost, which we estimate will increase from an initial cost of $962 million in 2017 to nearly $2.8 billion through 2028. Boeing’s delivery of the EUS to NASA has also been delayed from February 2021 to April 2027, and when combined with other factors, suggests the September 2028 Artemis IV launch date could be delayed as well. Factors contributing to these cost increases and schedule delays include redirection of EUS funds to the core stage during Artemis I production, changing Artemis mission assignments, maintaining an extended workforce 7 years more than planned, manufacturing issues, and supply chain challenges...... Additionally, Boeing Defense, Space & Security’s EVMS, used by NASA for its Stages contract to measure cost and schedule progress, has been disapproved by the U.S. Department of Defense since 2020. DCMA has issued several Level II and III CARs for this EVMS, including a Level III CAR related to visibility into cost, schedule, and resource needs for several Boeing contracts, including Stages.Two important things unsaid in the report:1) What are the flight safety implications? EUS had unqualified welders doing shoddy work to the tune of 70+ corrective actions caught by Defense Contracts Management Agency, which is only looking at contract violations. What build errors outside contract violations have and have not been caught by Boeing and NASA? What is the plan for ensuring that all are found and corrected? What are the programmatic implications of that rework? What are the flight safety implications if that rework is not done (or still done poorly)?Orion/SLS already had a lousy (Shuttle ballpark) projected loss-of-crew estimate going forward, made worse by the system’s very infrequent use and what that does to workforce skills retention. What does this shoddy work and rework imply for flight safety? Are we at the point where alternatives to Orion/SLS need to be pursued for flight safety reasons alone?2) How far up the management chain does knowledge of these shenanigans go? At some point in those 70+ corrective actions from the Defense Contracts Management Agency, Steve Snell (Boeing’s EUS program manager) and James Savage (Boeing’s EUS chief engineer) must have known that they had hired/were continuing to hire unqualified welders and let them continue to work on EUS builds. Why didn’t they do something about it, like stand down until the project had the right welders with the right qualifications on board? According to the report, this shoddy work has cost the project years in schedule slips anyway, so why not stop work for a few months until the hiring was resolved? If they claim that they never knew after 70+ corrective actions, then why the heck not? Are they really that out-of-touch with their own project? Either way, will Snell and Savage get booted from the project and replaced with competent, ethical managers?Same goes for the Boeing and NASA managers leading SLS, John Honeycutt and David Dutcher, respectively. What did they know? When did they know it? Why didn’t they do anything about it? Will they get the boot?Same goes for ESDMD leadership. What did Free and Koerner know? When did they know it? Why didn’t they do anything about it?And a bonus third point of outrage — why on God’s green Earth does a major US defense contractor not have a working earned value management system in the 21st freaking century?!?! EVM was invented in 1967. It’s been over half a century, for Heaven’s sake!Cripes...These are great questions, VSECOTSPE.One can only hope that someone in a position to do so is able to extract the answers from NASA leadership.
And a bonus third point of outrage — why on God’s green Earth does a major US defense contractor not have a working earned value management system in the 21st freaking century?!?! EVM was invented in 1967. It’s been over half a century, for Heaven’s sake!
This might deserve it's own thread. And I don't know that I have an actual point, just a lot of observations that could lead to a point. Boeing having unskilled labor isn't just a Boeing problem. Another program I work on (that I'm not comfortable outing), is 1) Firm Fixed Price, 2) way behind schedule, 3) engineering talent is leaving the program. It's easy to just blame management, but I also think many other things are happening at the same time. NASA moving to FFP contracts puts tremendous pressure on companies to reduce costs, which translates into lower resources (pay, oversight, etc). Add to that, that new commercial space companies rarely subcontract work (everyone is following SpaceX and Blue in bringing everything that they can in house), and it's hard to be an aerospace engineer these days (unless you live in Seattle, Denver, or LA).It's also easy to point to SpaceX and argue that they are very good at FFP, but as others have mentioned, they are a unicorn. SpaceX doesn't have to make a profit, they get don't get funding at the whim of outside investors, talented engineers seek THEM out to work 60-80 hrs a week for 40 hrs a week salary. SpaceX is a truly incredible company with incredible people. That is not the same bar to hold everyone else to. Continuing with this "ramble" I worry that with so many commercial startups pulling for talent, that we have spread out the talent thin, to the point that you can't concentrate talent onto a strong team to get the work done (again SpaceX does not have this problem).I dunno, maybe I'm way off base here.
Quote from: jarmumd on 08/09/2024 03:23 pmThis might deserve it's own thread. And I don't know that I have an actual point, just a lot of observations that could lead to a point. Boeing having unskilled labor isn't just a Boeing problem. Another program I work on (that I'm not comfortable outing), is 1) Firm Fixed Price, 2) way behind schedule, 3) engineering talent is leaving the program. It's easy to just blame management, but I also think many other things are happening at the same time. NASA moving to FFP contracts puts tremendous pressure on companies to reduce costs, which translates into lower resources (pay, oversight, etc). Add to that, that new commercial space companies rarely subcontract work (everyone is following SpaceX and Blue in bringing everything that they can in house), and it's hard to be an aerospace engineer these days (unless you live in Seattle, Denver, or LA).It's also easy to point to SpaceX and argue that they are very good at FFP, but as others have mentioned, they are a unicorn. SpaceX doesn't have to make a profit, they get don't get funding at the whim of outside investors, talented engineers seek THEM out to work 60-80 hrs a week for 40 hrs a week salary. SpaceX is a truly incredible company with incredible people. That is not the same bar to hold everyone else to. Continuing with this "ramble" I worry that with so many commercial startups pulling for talent, that we have spread out the talent thin, to the point that you can't concentrate talent onto a strong team to get the work done (again SpaceX does not have this problem).I dunno, maybe I'm way off base here.Good observations. I don't think you are offbase. Almost everyone would agree that SX is a unicornHowever, i do think it's a management problem. Rocket Lab seems to be doing OK .. at a much lower scale than SX.A lot of people, including insiders, have criticised Blue prior to the new CEO. LockMart seems to keep it's programs going.Management makes a big difference.
Quote from: freddo411 on 08/09/2024 04:21 pmQuote from: jarmumd on 08/09/2024 03:23 pmThis might deserve it's own thread. And I don't know that I have an actual point, just a lot of observations that could lead to a point. Boeing having unskilled labor isn't just a Boeing problem. Another program I work on (that I'm not comfortable outing), is 1) Firm Fixed Price, 2) way behind schedule, 3) engineering talent is leaving the program. It's easy to just blame management, but I also think many other things are happening at the same time. NASA moving to FFP contracts puts tremendous pressure on companies to reduce costs, which translates into lower resources (pay, oversight, etc). Add to that, that new commercial space companies rarely subcontract work (everyone is following SpaceX and Blue in bringing everything that they can in house), and it's hard to be an aerospace engineer these days (unless you live in Seattle, Denver, or LA).It's also easy to point to SpaceX and argue that they are very good at FFP, but as others have mentioned, they are a unicorn. SpaceX doesn't have to make a profit, they get don't get funding at the whim of outside investors, talented engineers seek THEM out to work 60-80 hrs a week for 40 hrs a week salary. SpaceX is a truly incredible company with incredible people. That is not the same bar to hold everyone else to. Continuing with this "ramble" I worry that with so many commercial startups pulling for talent, that we have spread out the talent thin, to the point that you can't concentrate talent onto a strong team to get the work done (again SpaceX does not have this problem).I dunno, maybe I'm way off base here.Good observations. I don't think you are offbase. Almost everyone would agree that SX is a unicornHowever, i do think it's a management problem. Rocket Lab seems to be doing OK .. at a much lower scale than SX.A lot of people, including insiders, have criticised Blue prior to the new CEO. LockMart seems to keep it's programs going.Management makes a big difference.Finding really good management is just as hard, if not harder than finding qualified engineers and shop laborers. None of this will be easy to fix. This is a problem in lots of industries, not just aerospace.
If it's broader than aerospace, is it partially a generational shift thing? (The more numerous baby boom generation being mostly retired now, etc)
Quote from: Vultur on 08/09/2024 07:45 pmIf it's broader than aerospace, is it partially a generational shift thing? (The more numerous baby boom generation being mostly retired now, etc)When the economy is good, companies struggle to find workers, which makes it a) high cost to the company to fire an ineffective worker (because it's cheaper to retrain than to find and hire a replacement), and b) low cost to the worker to get fired (because everyone is hiring so it's easy to find another job). This makes the penalty for shoddy workmanship almost non-existent.