Ars Technica article on SLS/Boeing

[Vultur]

 https://arstechnica.com/space/2024/08/a-new-report-finds-boeings-rockets-are-built-with-an-unqualified-work-force/

This is rather disturbing.

[VSECOTSPE]

Ugh.  Just ugh.

[seb21051]

The NASA Inspector General's Report that spawned the Ars Report:

https://oig.nasa.gov/wp-content/uploads/2024/08/ig-24-015.pdf

The hits just keep on coming.

If I was Boeing's new CEO, I might well be tempted to get rid of the top 25 management levels and their inhabitants.

[catdlr]

Here’s a summary of the article:

NASA’s Exploration Upper Stage Program Delays and Budget Overruns

Program Overview: NASA’s program to develop a new upper stage for the Space Launch System (SLS) rocket, known as the Exploration Upper Stage (EUS), is significantly behind schedule and over budget. Initially planned for completion in 2017, the project is now seven years late and costs have ballooned.

Budget Issues: The development costs for the EUS have tripled from the original estimate of $962 million in 2017 to $2.8 billion. The overall Block 1B development costs are expected to reach $5.7 billion, $700 million more than NASA’s recent estimate.

Quality Control Concerns: The report highlights poor quality control practices by Boeing, the project’s prime contractor. Issues include insufficient aerospace production experience among the workforce and “unsatisfactory” welding operations, leading to delays and increased costs.

Impact on Artemis Program: The EUS is crucial for NASA’s Artemis program, specifically for the Artemis IV mission scheduled for 2028. However, due to various issues, including the readiness of other mission components, this timeline is unlikely to be met.

Recommendations and Responses: NASA’s inspector general recommended financial penalties for Boeing’s noncompliance with quality standards. However, NASA’s deputy associate administrator declined, citing existing contract provisions for financial ramifications.

NASA initially estimated the development costs for the Exploration Upper Stage (EUS) at $962 million in 2017. However, a new report predicts the cost will be $2.8 billion, nearly three times the original estimate. The delays in the EUS development are almost year-for-year, with the project now expected to be completed by 2028 instead of the original 2021 deadline.

The increased costs benefit Boeing due to the cost-plus contract, which covers all expenses plus a fee. This may explain the prolonged development timeline. Critics argue that the Space Launch System (SLS) works well as it is, and there are cheaper alternatives for the upper stage, such as United Launch Alliance’s Centaur V. Additionally, upcoming commercial rockets like Starship and New Glenn could serve as powerful alternatives.

[deltaV]

There are far, far cheaper upper stages that could be used for the rocket's primary function to launch the Orion spacecraft to lunar orbit, including United Launch Alliance's reliable (and ready) Centaur V upper stage.

Last time I made back-of-the-envelope performance estimates of alternative SLS upper stages, I got SLS + EUS 42 tonnes to TLI, SLS + BE3U (I forget how many BE3U) ~35 tonnes, SLS + Centaur V ~34 tonnes, SLS + Falcon Upper Stage ~33 tonnes, SLS + ICPS 27 tonnes. SLS + EUS may actually have more performance per dollar than SLS + Centaur V, simply because the rest of SLS is even more overpriced than EUS is.

With Starship and New Glenn, NASA will also soon have two very powerful commercial super heavy lift rockets to draw upon.

Yeah, Starship and New Glenn (with propellant transfer) are the primary SLS + EUS alternatives, not SLS with another upper stage (or other boosters as sometimes proposed).

[VSECOTSPE]

From the OIG report’s executive summary:

While NASA requires its aerospace contractors to have quality assurance programs that comply with SAE International’s AS9100D standards on quality management systems, we found Boeing’s quality management system at Michoud does not adhere to these standards or NASA requirements. NASA engages DCMA to conduct surveillance of Boeing’s core and upper stage manufacturing efforts at Michoud, and when deficiencies in quality are found, DCMA issues Corrective Action Requests (CAR) to the contractor. CARs are labeled Level I through IV, with Level I the least serious deficiency. From September 2021 to September 2023, DCMA issued Boeing 71 Level I and II CARs, as well as a draft Level III CAR. According to DCMA officials, this is a high number of CARs for a space flight system at this stage in development and reflects a recurring and degraded state of product quality control. Boeing’s process to address deficiencies to date has been ineffective, and the company has generally been nonresponsive in taking corrective actions when the same quality control issues reoccur.

Quality control issues at Michoud are largely due to the lack of a sufficient number of trained and experienced aerospace workers at Boeing. To mitigate these challenges, Boeing provides training and work orders to its employees. Considering the significant quality control deficiencies at Michoud, we found these efforts to be inadequate. For example, during our visit to Michoud in April 2023, we observed a liquid oxygen fuel tank dome—a critical component of the SLS Core Stage 3—segregated and pending disposition on whether and how it can safely be used going forward due to welds that did not meet NASA specifications. According to NASA officials, the welding issues arose due to Boeing’s inexperienced technicians and inadequate work order planning and supervision. The lack of a trained and qualified workforce increases the risk that Boeing will continue to manufacture parts and components that do not adhere to NASA requirements and industry standards.

We project SLS Block 1B costs will reach approximately $5.7 billion before the system is scheduled to launch in 2028. This is $700 million more than NASA’s 2023 Agency Baseline Commitment, which established a cost and schedule baseline at nearly $5 billion. EUS development accounts for more than half of this cost, which we estimate will increase from an initial cost of $962 million in 2017 to nearly $2.8 billion through 2028. Boeing’s delivery of the EUS to NASA has also been delayed from February 2021 to April 2027, and when combined with other factors, suggests the September 2028 Artemis IV launch date could be delayed as well. Factors contributing to these cost increases and schedule delays include redirection of EUS funds to the core stage during Artemis I production, changing Artemis mission assignments, maintaining an extended workforce 7 years more than planned, manufacturing issues, and supply chain challenges...

... Additionally, Boeing Defense, Space & Security’s EVMS, used by NASA for its Stages contract to measure cost and schedule progress, has been disapproved by the U.S. Department of Defense since 2020. DCMA has issued several Level II and III CARs for this EVMS, including a Level III CAR related to visibility into cost, schedule, and resource needs for several Boeing contracts, including Stages.

Two important things unsaid in the report:

1) What are the flight safety implications? EUS had unqualified welders doing shoddy work to the tune of 70+ corrective actions caught by Defense Contracts Management Agency, which is only looking at contract violations.  What build errors outside contract violations have and have not been caught by Boeing and NASA?  What is the plan for ensuring that all are found and corrected?  What are the programmatic implications of that rework?  What are the flight safety implications if that rework is not done (or still done poorly)?

Orion/SLS already had a lousy (Shuttle ballpark) projected loss-of-crew estimate going forward, made worse by the system’s very infrequent use and what that does to workforce skills retention.  What does this shoddy work and rework imply for flight safety?  Are we at the point where alternatives to Orion/SLS need to be pursued for flight safety reasons alone?

2) How far up the management chain does knowledge of these shenanigans go?  At some point in those 70+ corrective actions from the Defense Contracts Management Agency, Steve Snell (Boeing’s EUS program manager) and James Savage (Boeing’s EUS chief engineer) must have known that they had hired/were continuing to hire unqualified welders and let them continue to work on EUS builds.  Why didn’t they do something about it, like stand down until the project had the right welders with the right qualifications on board?  According to the report, this shoddy work has cost the project years in schedule slips anyway, so why not stop work for a few months until the hiring was resolved?  If they claim that they never knew after 70+ corrective actions, then why the heck not?  Are they really that out-of-touch with their own project?   Either way, will Snell and Savage get booted from the project and replaced with competent, ethical managers?

Same goes for the Boeing and NASA managers leading SLS, John Honeycutt and David Dutcher, respectively.  What did they know?  When did they know it?  Why didn’t they do anything about it?  Will they get the boot?

Same goes for ESDMD leadership.  What did Free and Koerner know?  When did they know it?  Why didn’t they do anything about it?

And a bonus third point of outrage — why on God’s green Earth does a major US defense contractor not have a working earned value management system in the 21st freaking century?!?!  EVM was invented in 1967.  It’s been over half a century, for Heaven’s sake!

Cripes...

[sdsds]

Regarding poor weld quality: the OIG seems to place responsibility on the workforce. Didn't Boeing invest in robotic weld tooling?

[russianhalo117]

Regarding poor weld quality: the OIG seems to place responsibility on the workforce. Didn't Boeing invest in robotic weld tooling?

Only for certain tasks. There are plenty of manual welds to be done i.e. welding on the tank walls grid filler pieces across the friction stir welds on either the inside and/or outside the tanks and the segments welded to it. The same welds occur with the core stage. There are differences with EUS versus ICPS in terms of design outside of upscaling plus manufacturing is different to allow use of select core stage tooling to assemble the 8.4m LH2 tank and forward skirt. AFAIU the 5.5m tank uses slightly upscaled yet modernized tooling derived from DCSS/ICPS.

[woods170]

Regarding poor weld quality: the OIG seems to place responsibility on the workforce. Didn't Boeing invest in robotic weld tooling?

No, the OIG clearly signals that the workforce is unqualified for the job. But having an unqualified workforce in the first place, is solely the responsibility of management.

Managment can't hide behind "our welders are unqualified" because everyone will immediately point out that it is management that hired the unqualified welders.


Another thing pointed out by OIG is that there are insufficient numbers of adequately trained employees. That, once again, is the responsibility of management.
The fact that there's not enough employees and that their training is inadequate is not the fault of those employees. It is the fault of management, because they failed to hire sufficient people to do the job and because they failed to hire the right people to do the job.


And quite frankly I can understand why Boeing managment failed at this. All the qualified and motivated people are working at SpaceX, ULA, Blue Origin, Rocketlab, Firefly, etc, etc. With the amount of bad news about Boeing piling up year-after-year I wouldn't want to work for Boeing either, if I were an aerospace engineer (which I'm not btw). Heck, I wouldn't want to work for Boeing in any capacity given their completely floundered reputation.

[Proponent]

Budget Issues: The development costs for the EUS have tripled from the original estimate of $962 million in 2017 to $2.8 billion. The overall Block 1B development costs are expected to reach $5.7 billion, $700 million more than NASA’s recent estimate.

I'll bet that even retired Senator Shelby is impressed by the success of the SLS program in achieving its primary albeit unstated goal. It is the gift that keeps on giving.

EDIT: "be" ->"bet"

[Athelstane]

From the OIG report’s executive summary:

While NASA requires its aerospace contractors to have quality assurance programs that comply with SAE International’s AS9100D standards on quality management systems, we found Boeing’s quality management system at Michoud does not adhere to these standards or NASA requirements. NASA engages DCMA to conduct surveillance of Boeing’s core and upper stage manufacturing efforts at Michoud, and when deficiencies in quality are found, DCMA issues Corrective Action Requests (CAR) to the contractor. CARs are labeled Level I through IV, with Level I the least serious deficiency. From September 2021 to September 2023, DCMA issued Boeing 71 Level I and II CARs, as well as a draft Level III CAR. According to DCMA officials, this is a high number of CARs for a space flight system at this stage in development and reflects a recurring and degraded state of product quality control. Boeing’s process to address deficiencies to date has been ineffective, and the company has generally been nonresponsive in taking corrective actions when the same quality control issues reoccur.

Quality control issues at Michoud are largely due to the lack of a sufficient number of trained and experienced aerospace workers at Boeing. To mitigate these challenges, Boeing provides training and work orders to its employees. Considering the significant quality control deficiencies at Michoud, we found these efforts to be inadequate. For example, during our visit to Michoud in April 2023, we observed a liquid oxygen fuel tank dome—a critical component of the SLS Core Stage 3—segregated and pending disposition on whether and how it can safely be used going forward due to welds that did not meet NASA specifications. According to NASA officials, the welding issues arose due to Boeing’s inexperienced technicians and inadequate work order planning and supervision. The lack of a trained and qualified workforce increases the risk that Boeing will continue to manufacture parts and components that do not adhere to NASA requirements and industry standards.

We project SLS Block 1B costs will reach approximately $5.7 billion before the system is scheduled to launch in 2028. This is $700 million more than NASA’s 2023 Agency Baseline Commitment, which established a cost and schedule baseline at nearly $5 billion. EUS development accounts for more than half of this cost, which we estimate will increase from an initial cost of $962 million in 2017 to nearly $2.8 billion through 2028. Boeing’s delivery of the EUS to NASA has also been delayed from February 2021 to April 2027, and when combined with other factors, suggests the September 2028 Artemis IV launch date could be delayed as well. Factors contributing to these cost increases and schedule delays include redirection of EUS funds to the core stage during Artemis I production, changing Artemis mission assignments, maintaining an extended workforce 7 years more than planned, manufacturing issues, and supply chain challenges...

... Additionally, Boeing Defense, Space & Security’s EVMS, used by NASA for its Stages contract to measure cost and schedule progress, has been disapproved by the U.S. Department of Defense since 2020. DCMA has issued several Level II and III CARs for this EVMS, including a Level III CAR related to visibility into cost, schedule, and resource needs for several Boeing contracts, including Stages.

Two important things unsaid in the report:

1) What are the flight safety implications? EUS had unqualified welders doing shoddy work to the tune of 70+ corrective actions caught by Defense Contracts Management Agency, which is only looking at contract violations.  What build errors outside contract violations have and have not been caught by Boeing and NASA?  What is the plan for ensuring that all are found and corrected?  What are the programmatic implications of that rework?  What are the flight safety implications if that rework is not done (or still done poorly)?

Orion/SLS already had a lousy (Shuttle ballpark) projected loss-of-crew estimate going forward, made worse by the system’s very infrequent use and what that does to workforce skills retention.  What does this shoddy work and rework imply for flight safety?  Are we at the point where alternatives to Orion/SLS need to be pursued for flight safety reasons alone?

2) How far up the management chain does knowledge of these shenanigans go?  At some point in those 70+ corrective actions from the Defense Contracts Management Agency, Steve Snell (Boeing’s EUS program manager) and James Savage (Boeing’s EUS chief engineer) must have known that they had hired/were continuing to hire unqualified welders and let them continue to work on EUS builds.  Why didn’t they do something about it, like stand down until the project had the right welders with the right qualifications on board?  According to the report, this shoddy work has cost the project years in schedule slips anyway, so why not stop work for a few months until the hiring was resolved?  If they claim that they never knew after 70+ corrective actions, then why the heck not?  Are they really that out-of-touch with their own project?   Either way, will Snell and Savage get booted from the project and replaced with competent, ethical managers?

Same goes for the Boeing and NASA managers leading SLS, John Honeycutt and David Dutcher, respectively.  What did they know?  When did they know it?  Why didn’t they do anything about it?  Will they get the boot?

Same goes for ESDMD leadership.  What did Free and Koerner know?  When did they know it?  Why didn’t they do anything about it?

And a bonus third point of outrage — why on God’s green Earth does a major US defense contractor not have a working earned value management system in the 21st freaking century?!?!  EVM was invented in 1967.  It’s been over half a century, for Heaven’s sake!

Cripes...

These are great questions, VSECOTSPE.

One can only hope that someone in a position to do so is able to extract the answers from NASA leadership.

[woods170]

From the OIG report’s executive summary:

While NASA requires its aerospace contractors to have quality assurance programs that comply with SAE International’s AS9100D standards on quality management systems, we found Boeing’s quality management system at Michoud does not adhere to these standards or NASA requirements. NASA engages DCMA to conduct surveillance of Boeing’s core and upper stage manufacturing efforts at Michoud, and when deficiencies in quality are found, DCMA issues Corrective Action Requests (CAR) to the contractor. CARs are labeled Level I through IV, with Level I the least serious deficiency. From September 2021 to September 2023, DCMA issued Boeing 71 Level I and II CARs, as well as a draft Level III CAR. According to DCMA officials, this is a high number of CARs for a space flight system at this stage in development and reflects a recurring and degraded state of product quality control. Boeing’s process to address deficiencies to date has been ineffective, and the company has generally been nonresponsive in taking corrective actions when the same quality control issues reoccur.

Quality control issues at Michoud are largely due to the lack of a sufficient number of trained and experienced aerospace workers at Boeing. To mitigate these challenges, Boeing provides training and work orders to its employees. Considering the significant quality control deficiencies at Michoud, we found these efforts to be inadequate. For example, during our visit to Michoud in April 2023, we observed a liquid oxygen fuel tank dome—a critical component of the SLS Core Stage 3—segregated and pending disposition on whether and how it can safely be used going forward due to welds that did not meet NASA specifications. According to NASA officials, the welding issues arose due to Boeing’s inexperienced technicians and inadequate work order planning and supervision. The lack of a trained and qualified workforce increases the risk that Boeing will continue to manufacture parts and components that do not adhere to NASA requirements and industry standards.

We project SLS Block 1B costs will reach approximately $5.7 billion before the system is scheduled to launch in 2028. This is $700 million more than NASA’s 2023 Agency Baseline Commitment, which established a cost and schedule baseline at nearly $5 billion. EUS development accounts for more than half of this cost, which we estimate will increase from an initial cost of $962 million in 2017 to nearly $2.8 billion through 2028. Boeing’s delivery of the EUS to NASA has also been delayed from February 2021 to April 2027, and when combined with other factors, suggests the September 2028 Artemis IV launch date could be delayed as well. Factors contributing to these cost increases and schedule delays include redirection of EUS funds to the core stage during Artemis I production, changing Artemis mission assignments, maintaining an extended workforce 7 years more than planned, manufacturing issues, and supply chain challenges...

... Additionally, Boeing Defense, Space & Security’s EVMS, used by NASA for its Stages contract to measure cost and schedule progress, has been disapproved by the U.S. Department of Defense since 2020. DCMA has issued several Level II and III CARs for this EVMS, including a Level III CAR related to visibility into cost, schedule, and resource needs for several Boeing contracts, including Stages.

Two important things unsaid in the report:

1) What are the flight safety implications? EUS had unqualified welders doing shoddy work to the tune of 70+ corrective actions caught by Defense Contracts Management Agency, which is only looking at contract violations.  What build errors outside contract violations have and have not been caught by Boeing and NASA?  What is the plan for ensuring that all are found and corrected?  What are the programmatic implications of that rework?  What are the flight safety implications if that rework is not done (or still done poorly)?

Orion/SLS already had a lousy (Shuttle ballpark) projected loss-of-crew estimate going forward, made worse by the system’s very infrequent use and what that does to workforce skills retention.  What does this shoddy work and rework imply for flight safety?  Are we at the point where alternatives to Orion/SLS need to be pursued for flight safety reasons alone?

2) How far up the management chain does knowledge of these shenanigans go?  At some point in those 70+ corrective actions from the Defense Contracts Management Agency, Steve Snell (Boeing’s EUS program manager) and James Savage (Boeing’s EUS chief engineer) must have known that they had hired/were continuing to hire unqualified welders and let them continue to work on EUS builds.  Why didn’t they do something about it, like stand down until the project had the right welders with the right qualifications on board?  According to the report, this shoddy work has cost the project years in schedule slips anyway, so why not stop work for a few months until the hiring was resolved?  If they claim that they never knew after 70+ corrective actions, then why the heck not?  Are they really that out-of-touch with their own project?   Either way, will Snell and Savage get booted from the project and replaced with competent, ethical managers?

Same goes for the Boeing and NASA managers leading SLS, John Honeycutt and David Dutcher, respectively.  What did they know?  When did they know it?  Why didn’t they do anything about it?  Will they get the boot?

Same goes for ESDMD leadership.  What did Free and Koerner know?  When did they know it?  Why didn’t they do anything about it?

And a bonus third point of outrage — why on God’s green Earth does a major US defense contractor not have a working earned value management system in the 21st freaking century?!?!  EVM was invented in 1967.  It’s been over half a century, for Heaven’s sake!

Cripes...

These are great questions, VSECOTSPE.

One can only hope that someone in a position to do so is able to extract the answers from NASA leadership.

That "someone" would either be the President, the Vice President or an honest committee from the U.S. Senate.
So, it's won't be happening IMO.

[VSECOTSPE]

NASA IG Blasts Boeing on Block IB

https://spacepolicyonline.com/news/nasa-ig-blasts-boeing-on-block-1b/

[jarmumd]

This might deserve it's own thread.  And I don't know that I have an actual point, just a lot of observations that could lead to a point.  Boeing having unskilled labor isn't just a Boeing problem.  Another program I work on (that I'm not comfortable outing), is 1) Firm Fixed Price, 2) way behind schedule, 3) engineering talent is leaving the program.  It's easy to just blame management, but I also think many other things are happening at the same time.  NASA moving to FFP contracts puts tremendous pressure on companies to reduce costs, which translates into lower resources (pay, oversight, etc).  Add to that, that new commercial space companies rarely subcontract work (everyone is following SpaceX and Blue in bringing everything that they can in house), and it's hard to be an aerospace engineer these days (unless you live in Seattle, Denver, or LA).

It's also easy to point to SpaceX and argue that they are very good at FFP, but as others have mentioned, they are a unicorn.  SpaceX doesn't have to make a profit, they get don't get funding at the whim of outside investors, talented engineers seek THEM out to work 60-80 hrs a week for 40 hrs a week salary.  SpaceX is a truly incredible company with incredible people.  That is not the same bar to hold everyone else to. 

Continuing with this "ramble" I worry that with so many commercial startups pulling for talent, that we have spread out the talent thin, to the point that you can't concentrate talent onto a strong team to get the work done (again SpaceX does not have this problem).

I dunno, maybe I'm way off base here.

[ccdengr]

And a bonus third point of outrage — why on God’s green Earth does a major US defense contractor not have a working earned value management system in the 21st freaking century?!?!  EVM was invented in 1967.  It’s been over half a century, for Heaven’s sake!

Likely because, like a lot of management buzzword stuff, it's one thing to require it and another thing to figure out what it actually means, and then convince auditors that you are "fully compliant" with an enormous stack of hard to interpret, if not mutually contradictory, requirements.

But I'm just an engineer, not a manager, so what do I know?  I still have PTSD from trying to support an IS09000 audit in a small organization.

That said, I am surprised that a huge organization like Boeing can't devote the resources to at least look like they are meeting this requirement.

[freddo411]

This might deserve it's own thread.  And I don't know that I have an actual point, just a lot of observations that could lead to a point.  Boeing having unskilled labor isn't just a Boeing problem.  Another program I work on (that I'm not comfortable outing), is 1) Firm Fixed Price, 2) way behind schedule, 3) engineering talent is leaving the program.  It's easy to just blame management, but I also think many other things are happening at the same time.  NASA moving to FFP contracts puts tremendous pressure on companies to reduce costs, which translates into lower resources (pay, oversight, etc).  Add to that, that new commercial space companies rarely subcontract work (everyone is following SpaceX and Blue in bringing everything that they can in house), and it's hard to be an aerospace engineer these days (unless you live in Seattle, Denver, or LA).

It's also easy to point to SpaceX and argue that they are very good at FFP, but as others have mentioned, they are a unicorn.  SpaceX doesn't have to make a profit, they get don't get funding at the whim of outside investors, talented engineers seek THEM out to work 60-80 hrs a week for 40 hrs a week salary.  SpaceX is a truly incredible company with incredible people.  That is not the same bar to hold everyone else to. 

Continuing with this "ramble" I worry that with so many commercial startups pulling for talent, that we have spread out the talent thin, to the point that you can't concentrate talent onto a strong team to get the work done (again SpaceX does not have this problem).

I dunno, maybe I'm way off base here.

Good observations.   I don't think you are offbase.   Almost everyone would agree that SX is a unicorn

However, i do think it's a management problem.   Rocket Lab seems to be doing OK .. at a much lower scale than SX.

A lot of people, including insiders, have criticised Blue prior to the new CEO.   

LockMart seems to keep it's programs going.

Management makes a big difference.

[Eric Hedman]

This might deserve it's own thread.  And I don't know that I have an actual point, just a lot of observations that could lead to a point.  Boeing having unskilled labor isn't just a Boeing problem.  Another program I work on (that I'm not comfortable outing), is 1) Firm Fixed Price, 2) way behind schedule, 3) engineering talent is leaving the program.  It's easy to just blame management, but I also think many other things are happening at the same time.  NASA moving to FFP contracts puts tremendous pressure on companies to reduce costs, which translates into lower resources (pay, oversight, etc).  Add to that, that new commercial space companies rarely subcontract work (everyone is following SpaceX and Blue in bringing everything that they can in house), and it's hard to be an aerospace engineer these days (unless you live in Seattle, Denver, or LA).

It's also easy to point to SpaceX and argue that they are very good at FFP, but as others have mentioned, they are a unicorn.  SpaceX doesn't have to make a profit, they get don't get funding at the whim of outside investors, talented engineers seek THEM out to work 60-80 hrs a week for 40 hrs a week salary.  SpaceX is a truly incredible company with incredible people.  That is not the same bar to hold everyone else to. 

Continuing with this "ramble" I worry that with so many commercial startups pulling for talent, that we have spread out the talent thin, to the point that you can't concentrate talent onto a strong team to get the work done (again SpaceX does not have this problem).

I dunno, maybe I'm way off base here.

Good observations.   I don't think you are offbase.   Almost everyone would agree that SX is a unicorn

However, i do think it's a management problem.   Rocket Lab seems to be doing OK .. at a much lower scale than SX.

A lot of people, including insiders, have criticised Blue prior to the new CEO.   

LockMart seems to keep it's programs going.

Management makes a big difference.

Finding really good management is just as hard, if not harder than finding qualified engineers and shop laborers.  None of this will be easy to fix.  This is a problem in lots of industries, not just aerospace.

[Vultur]

This might deserve it's own thread.  And I don't know that I have an actual point, just a lot of observations that could lead to a point.  Boeing having unskilled labor isn't just a Boeing problem.  Another program I work on (that I'm not comfortable outing), is 1) Firm Fixed Price, 2) way behind schedule, 3) engineering talent is leaving the program.  It's easy to just blame management, but I also think many other things are happening at the same time.  NASA moving to FFP contracts puts tremendous pressure on companies to reduce costs, which translates into lower resources (pay, oversight, etc).  Add to that, that new commercial space companies rarely subcontract work (everyone is following SpaceX and Blue in bringing everything that they can in house), and it's hard to be an aerospace engineer these days (unless you live in Seattle, Denver, or LA).

It's also easy to point to SpaceX and argue that they are very good at FFP, but as others have mentioned, they are a unicorn.  SpaceX doesn't have to make a profit, they get don't get funding at the whim of outside investors, talented engineers seek THEM out to work 60-80 hrs a week for 40 hrs a week salary.  SpaceX is a truly incredible company with incredible people.  That is not the same bar to hold everyone else to. 

Continuing with this "ramble" I worry that with so many commercial startups pulling for talent, that we have spread out the talent thin, to the point that you can't concentrate talent onto a strong team to get the work done (again SpaceX does not have this problem).

I dunno, maybe I'm way off base here.

Good observations.   I don't think you are offbase.   Almost everyone would agree that SX is a unicorn

However, i do think it's a management problem.   Rocket Lab seems to be doing OK .. at a much lower scale than SX.

A lot of people, including insiders, have criticised Blue prior to the new CEO.   

LockMart seems to keep it's programs going.

Management makes a big difference.

Finding really good management is just as hard, if not harder than finding qualified engineers and shop laborers.  None of this will be easy to fix.  This is a problem in lots of industries, not just aerospace.

If it's broader than aerospace, is it partially a generational shift thing? (The more numerous baby boom generation being mostly retired now, etc)

[KF3K]

If it's broader than aerospace, is it partially a generational shift thing? (The more numerous baby boom generation being mostly retired now, etc)

When the economy is good, companies struggle to find workers, which makes it a) high cost to the company to fire an ineffective worker (because it's cheaper to retrain than to find and hire a replacement), and b) low cost to the worker to get fired (because everyone is hiring so it's easy to find another job).  This makes the penalty for shoddy workmanship almost non-existent.

As a Gen Xer, I'd love to blame the young'uns for this, but I think it's more of a zero-interest-rate phenomenon, which affects workers across all generations.

[Coastal Ron]

If it's broader than aerospace, is it partially a generational shift thing? (The more numerous baby boom generation being mostly retired now, etc)

When the economy is good, companies struggle to find workers, which makes it a) high cost to the company to fire an ineffective worker (because it's cheaper to retrain than to find and hire a replacement), and b) low cost to the worker to get fired (because everyone is hiring so it's easy to find another job).  This makes the penalty for shoddy workmanship almost non-existent.

Economy boom and bust cycles have always been around, so this must be something else.

It could be that because Boeing took over the U.S. Government-owned Michoud facility outside of New Orleans well after the Lockheed Martin shut down the Shuttle External Tank (ET) production, that the general area lost a significant number of workers that had the skills needed for the SLS production.

Also, maybe there isn't a large need for aerospace workers in the area, so it is hard to recruit skilled workers for niche job categories?

[Vultur]

If it's broader than aerospace, is it partially a generational shift thing? (The more numerous baby boom generation being mostly retired now, etc)

When the economy is good, companies struggle to find workers, which makes it a) high cost to the company to fire an ineffective worker (because it's cheaper to retrain than to find and hire a replacement), and b) low cost to the worker to get fired (because everyone is hiring so it's easy to find another job).  This makes the penalty for shoddy workmanship almost non-existent.

As a Gen Xer, I'd love to blame the young'uns for this, but I think it's more of a zero-interest-rate phenomenon, which affects workers across all generations.

I actually wasn't blaming the younger generations, I was asking about whether there is a smaller number of relevant workers in the post baby boom cohort (now in their mature career phase) vs the baby boom cohort (now mostly retired). Especially in places like Louisiana which (unlike neighboring TX) is not seeing a lot of people moving to it, Louisiana population has actually slightly declined in recent years.

The question wasn't about generational work ethic, it was about workforce available.

[DaveS]

Also keeo in mind that the area did lose a significant amount of the experienced work force after hurricane Katrina back in 2005. It was found that part of the root cause for the ET hydrogen leaks (STS-119, 126 and 133) was that they all used carrier assemblies that had been improperly aligned during integration with the intertank at MAF due to an inexperienced workforce. So things can be traced back quite far back.

[Eric Hedman]

If it's broader than aerospace, is it partially a generational shift thing? (The more numerous baby boom generation being mostly retired now, etc)

When the economy is good, companies struggle to find workers, which makes it a) high cost to the company to fire an ineffective worker (because it's cheaper to retrain than to find and hire a replacement), and b) low cost to the worker to get fired (because everyone is hiring so it's easy to find another job).  This makes the penalty for shoddy workmanship almost non-existent.

As a Gen Xer, I'd love to blame the young'uns for this, but I think it's more of a zero-interest-rate phenomenon, which affects workers across all generations.

I deal with lots of companies in the sheet metal fabrication industry.  There is another big problem that has existed in welding for the last thirty years.  Not nearly enough people want to do it.  A friend of mine from mine ended up running a program at a local technical college training students in industrial arts including welding, machining, bending, cutting (Turret punches, lasers,etc.), additive manufacturing, etc.  They have been trying for a long time to recruit high school students for their programs with just not enough success to meet industry demand.

[sdsds]

It's easy to blame corporate management for not attracting and retaining a skilled workforce. Organized labor plays a role too. In the Seattle area, the International Association of Machinists District 751 (i.e. the Boeing machinists union) offers the Machinists Institute to provide education and training to build the local workforce.

https://www.machinistsinstitute.org/about-machinists-institute
https://www.iam751.org/

Are the welders at Michoud represented by a union?

[dfp21]

What happens to a diligent worker at Boeing who complains about poor quality?  (You can google that if you don't already know).

Yes it is BOEING MANAGEMENT CULTURE that produces low-to-mediocre quality products at high-to-insane prices.

[VSECOTSPE]

This might deserve it's own thread.  And I don't know that I have an actual point, just a lot of observations that could lead to a point.  Boeing having unskilled labor isn't just a Boeing problem.  Another program I work on (that I'm not comfortable outing), is 1) Firm Fixed Price, 2) way behind schedule, 3) engineering talent is leaving the program.  It's easy to just blame management, but I also think many other things are happening at the same time.  NASA moving to FFP contracts puts tremendous pressure on companies to reduce costs, which translates into lower resources (pay, oversight, etc).  Add to that, that new commercial space companies rarely subcontract work (everyone is following SpaceX and Blue in bringing everything that they can in house), and it's hard to be an aerospace engineer these days (unless you live in Seattle, Denver, or LA).

It's also easy to point to SpaceX and argue that they are very good at FFP, but as others have mentioned, they are a unicorn.  SpaceX doesn't have to make a profit, they get don't get funding at the whim of outside investors, talented engineers seek THEM out to work 60-80 hrs a week for 40 hrs a week salary.  SpaceX is a truly incredible company with incredible people.  That is not the same bar to hold everyone else to. 

No doubt, there are usually strains in most labor markets, and they may be especially acute in the space sector these days.  That said, EUS is critical path element of the agency’s huge flagship program, and it involves human space flight.  This isn’t a some smallsat that will cost investors millions if it fails.  This is a launch vehicle that will cost taxpayers billions, delay the nation’s most visible civil space goals for years, and kill or nearly kill four people if it fails.  I think they buried the lede, but the IG report points out the flight safety implications of the bad welds in a couple places:

Nonetheless, according to both DCMA and NASA Safety and Mission Assurance officials, the unusually high number of stamp warranty CARs reflects the seriousness of the nonconformity, and changing certified work order data without retention of historical information could increase the risk to the flight vehicle...

Moreover, quality control deficiencies, if not identified and corrected, could increase safety risk to the integrated spacecraft...

To borrow from the famous book on the Challenger accident, this is normalization of deviance.  It’s one thing to keep building after DCMA identifies a few deficiencies/nonconformities.   It’s another after DCMA identifies over 70+ deficiencies/nonconformities over the course of more than a year..  On a project this big or on a project involving astronauts, there is no excuse not to stand down until the workforce is properly trained and qualified or until qualified workers can be brought on board (or both).  There’s a half-dozen ways Boeing and NASA could have addressed this:  on-the-job training, offering high wages/relocation bonuses/etc. to qualified transfers from within the company, offering high wages/relocation bonuses/etc. to qualified outside hires, headhunters, subcontracting to qualified providers, etc.  Instead, management repeatedly decided to keep plugging away with an inadequate workforce, which has resulted in crap like this (again from the IG report), on top of the bad welds:

For example, in September 2021, a Level II CAR noted that foreign object debris was identified inside the SLS Core. Stage 2 liquid hydrogen fuel tank.  DCMA found contamination of metal shavings, Teflon, and other debris... on the forward dome panels inside of the tank. Foreign object debris can damage hardware and potentially injure flight crew when entrapped within crewed flight articles...

For example, Boeing officials incorrectly approved hardware processing under unacceptable environmental conditions, accepted and presented damaged seals to NASA for inspection, and used outdated versions of work orders. DCMA also found that Boeing personnel made numerous administrative errors through changes to certified work order data without proper documentation and retention of historical information necessary to trace the changes...

My old man was a muscle car enthusiast in his spare time.  Did lots of different types of welding.  He kept his engines, cars, and garage in better condition than this for decades.  Heck, I visited salvage yards with my dad that kept better records than this.  I’ve visited a small factory for Ford Cobra replicas that was cleaner and better organized than this.

My maternal grandfather and his brother (my great uncle) were metalworkers by trade who did welding and pipe fitting for an Army munitions plant.  Worked around insane amounts of high-explosives for decades.  Never created incidents like this.

A lot of this isn’t even about an improperly trained workforce that doesn’t know how to weld.  It’s also just plain lazy and sloppy work, period.

Again, a few instances in a bad week or month are one thing.  But 70+ over more than a year is evidence of bad management that is either too stupid or too venal (or both) to know better.  Everyone in the management chain that repeatedly failed to take adequate action should be removed from the program and replaced by managers with proven competence and backbone.  And NASA leadership will still need to bring in some independent technical authority to make sure the build has not been compromised in ways that that DCMA would not detect.

Apologies for my harsh tone.  It’s directed at the EUS project, not the poster.

Likely because, like a lot of management buzzword stuff, it's one thing to require it and another thing to figure out what it actually means,

EVM is not a buzzword.  It’s just an objective system for tracking whether a project is on schedule and cost.  I taught it to myself back in the late 90s when I was keeping tabs on NASA programs at OMB (and back when NASA programs actually used it).  It was formulated in the 60s and formalized by DOD in 1967.  Folks outside the industry may not be familiar with it and even folks inside the industry who have not managed projects or done program accounting may not understand it.  But it’s the best tool for tracking program progress and finding where program management attention is needed that we’ve got.  There’s no excuse for an aerospace giant like Boeing to have lost the ability to track EVM for the past four years.

https://en.m.wikipedia.org/wiki/Earned_value_management