Boeing's Starliner (CST-100) - Discussion Thread 5

[anof]

So, just to make sure that I'm on top of this and my reading comprehension hasn't slipped any:

1) The problems with Starliner are almost entirely due to control system software QA shortfalls;

I would disagree with this, if you're thinking of S/W QA in the traditional, independent department, sense. The problem is much more ingrained in their development and testing culture: it seems that Boeing's development team did an incomplete set of peer reviews, low level unit testing, and mid-level integration testing.  It also seems like they did little to no actual hardware in the loop systems integration testing, instead using software simulations and assuming those were valid.  I'm sure they did more work than the IRT report summaries we read imply, but for sure there were a lot of holes to fill in.

I would also add that NASA trusted Boeing and performed less oversight.

[DigitalMan]

Also, NASA hasn't done the safety review yet (as of the last press conference on Starliner). I don't see them proceeding without it.

[butters]

In addition to the software and software testing shortfalls, Boeing has also had to modify the electronics hardware in the radio transceiver systems to resolve the problems they had trying to communicate with the spacecraft over TDRS after launch.

[Nomadd]

 The chute failure on the abort test had nothing to do with software. But all the problems seemed to have the same basic cause, which you might call an attitude problem. They weren't from a design issue or other glitch, like a valve not performing or material incompatibility. They were from screwups that plain shouldn't have happened, and fixing that is a lot harder than fixing a piece of hardware.
 The 737 issues were also not just software by a longshot. A lack of redundant sensors, lack of notification that sensors had a problem and lack of proper documentation and training all contributed.
 But, I suspect the main issue is the same for both projects. A lack of Joe Sutters with the cajones to stand up to management.

[intelati]

But, I suspect the main issue is the same for both projects. A lack of Joe Sutters with the cajones to stand up to management.

Somewhat off topic, but just did a quick dive into Sutter.

from a 2010 article, there's this scary passage forshadowing these issues.

Boeing does need to keep a closer eye on its outsourced engineering work, Sutter said. “Engineers tend to drift off and do their own thing if you let them. The only way you maintain discipline is to watch them like a hawk. And that hasn’t always happened at Boeing.”

https://blog.seattlepi.com/aerospace/2010/07/16/boeings-sutter-737-replacement-timing-depends-on-engines/

[freddo411]

But, I suspect the main issue is the same for both projects. A lack of Joe Sutters with the cajones to stand up to management.

Somewhat off topic, but just did a quick dive into Sutter.

from a 2010 article, there's this scary passage forshadowing these issues.

Boeing does need to keep a closer eye on its outsourced engineering work, Sutter said. “Engineers tend to drift off and do their own thing if you let them. The only way you maintain discipline is to watch them like a hawk. And that hasn’t always happened at Boeing.”

https://blog.seattlepi.com/aerospace/2010/07/16/boeings-sutter-737-replacement-timing-depends-on-engines/

I'm not sure how others read that quote, but I find it insulting to engineers and engineering discipline.   I wouldn't want to work at a company that even had the whiff of that kind of attitude toward engineering.   

Virtually every engineer I know is uniquely qualified and called upon to make critical, rational tradeoff decisions in most tasks.   Supervision is a collaborative effort to define goals.   If things descend into a "build this just like I said" scenario, you lose the talent and expertise of the engineer.

Makes me wonder if there was an engineer who might have said:  "We should run an all up integrated test with the booster" ....    Judging by posters on NSF, there is a high likelyhood there was.

[Frogstar_Robot]

I think I understand the quote. He is referring to outsourced engineering work. Outsourcing contains many hidden dangers while the management are only looking at the bottom line. I have been on both sides of the fence.

In my experience, companies are not good at capturing requirements. In house, this is mitigated because the engineers usually have long experience of the application area, and have close contact with other stakeholders. This gets worse over time to a point where in-house specs are very poor to non-existent.

When a company tries to outsource a project, they fall flat on their face, because the out-sourced engineers no longer have access to all the company knowledge. They work from an incomplete and inadequate spec, and the contact with the customer is through "liaison" who add an extra level of bureaucracy. When the customer says, "oh we meant X not Y", the supplier regards that as a contract change that should be billed for, which obviously the customer thinks the supplier should pay for since "they got it wrong".

That of course, assume that the supplier's engineers are competent. If not, then all bets are off...

[niwax]

But, I suspect the main issue is the same for both projects. A lack of Joe Sutters with the cajones to stand up to management.

Somewhat off topic, but just did a quick dive into Sutter.

from a 2010 article, there's this scary passage forshadowing these issues.

Boeing does need to keep a closer eye on its outsourced engineering work, Sutter said. “Engineers tend to drift off and do their own thing if you let them. The only way you maintain discipline is to watch them like a hawk. And that hasn’t always happened at Boeing.”

https://blog.seattlepi.com/aerospace/2010/07/16/boeings-sutter-737-replacement-timing-depends-on-engines/

I highly recommend his autobiography that centers around the development of the 747, the last chapter deals with his work on the Rogers commission after Challenger and has a lot more foreshadowing about planes, space and NASA.

[intelati]

I think I understand the quote. He is referring to outsourced engineering work. Outsourcing contains many hidden dangers while the management are only looking at the bottom line. I have been on both sides of the fence.

In my experience, companies are not good at capturing requirements. In house, this is mitigated because the engineers usually have long experience of the application area, and have close contact with other stakeholders. This gets worse over time to a point where in-house specs are very poor to non-existent.

When a company tries to outsource a project, they fall flat on their face, because the out-sourced engineers no longer have access to all the company knowledge. They work from an incomplete and inadequate spec, and the contact with the customer is through "liaison" who add an extra level of bureaucracy. When the customer says, "oh we meant X not Y", the supplier regards that as a contract change that should be billed for, which obviously the customer thinks the supplier should pay for since "they got it wrong".

That of course, assume that the supplier's engineers are competent. If not, then all bets are off...

Look at the post just a couple above mine. The quote rang a bell with me in this context.

So, just to make sure that I'm on top of this and my reading comprehension hasn't slipped any:

1) The problems with Starliner are almost entirely due to control system software QA shortfalls;

I would disagree with this, if you're thinking of S/W QA in the traditional, independent department, sense. The problem is much more ingrained in their development and testing culture: it seems that Boeing's development team did an incomplete set of peer reviews, low level unit testing, and mid-level integration testing.

But yes, I certainly will check out his Autobiography

[Nomadd]

Makes me wonder if there was an engineer who might have said:  "We should run an all up integrated test with the booster" ....    Judging by posters on NSF, there is a high likelyhood there was.

Of course there was. The problem was a lack of someone willing to risk his job by insisting. Sutter thought he'd be fired more than once for flat out refusing to go along with management idiocy.
 I have a little more than an armchair opinion on the matter myself. Part of the reason I've had more jobs than I can easily count. That, and an unfortunate lack of diplomacy at times.
 I'd like to think there's a solution, but when it involves the people making the decisions giving up power, It's hard to be optimistic.

 As far as the thread subject, I hope they're doing a little more than just piling on procedures. But finding and enabling the few people who can make projects like these shine doesn't seem to be that common now days.
 I'd do it, but it turns out my roof isn't all that waterproof with a 60mph wind driving the rain, so I have to work on that instead.

[LouScheffer]

Makes me wonder if there was an engineer who might have said:  "We should run an all up integrated test with the booster" ....    Judging by posters on NSF, there is a high likelyhood there was.

Of course there was. The problem was a lack of someone willing to risk his job by insisting. Sutter thought he'd be fired more than once by flat out refusing to go along with management idiocy.

Older engineers have a lot of tactical disadvantages.  The last language they programmed in was likely Fortran, and they are entirely unfamiliar with Slack and GitHub.  And they have higher salaries.

But this situation is exactly where they can earn their keep.  This is not the first time that folks have tried to save money by skipping the integrated test, convinced themselves it was OK, and paid by loss of mission.  Remember the Ariane 5 first flight?  Remembering where things screwed up, and standing up for not repeating the same mistake, is the job of senior engineering staff.

[savantu]

The bar has been set very high after yesterday's succesful landing.

[john smith 19]

The bar has been set very high after yesterday's succesful landing.

True, but not unfairlyhigh.

How many times have they (and LM) played the "We are the safe pair of hands" card?

Going along with that line of reasoning (without checking actual performance against claims gives the illusion of security, but not the reality. I'd suggest all of those faults should have been picked up in testing, not during an actual missions.

Making this stuff work is tough.

[ThePonjaX]

Makes me wonder if there was an engineer who might have said:  "We should run an all up integrated test with the booster" ....    Judging by posters on NSF, there is a high likelyhood there was.

Of course there was. The problem was a lack of someone willing to risk his job by insisting. Sutter thought he'd be fired more than once by flat out refusing to go along with management idiocy.

Older engineers have a lot of tactical disadvantages.  The last language they programmed in was likely Fortran, and they are entirely unfamiliar with Slack and GitHub.  And they have higher salaries.

But this situation is exactly where they can earn their keep.  This is not the first time that folks have tried to save money by skipping the integrated test, convinced themselves it was OK, and paid by loss of mission.  Remember the Ariane 5 first flight?  Remembering where things screwed up, and standing up for not repeating the same mistake, is the job of senior engineering staff.

I'm a kind of old engineer ... in my fifties , more than 30 years working on IT and I can tell you ( a put some post before on these) the problem is other:
They believe the software isn't a main component of the ship so they outsource the work trying to save money and if the hardware is ready all it's ok.
Of course these is plain wrong. Software can compensate a a hardware problem not the other way around. Design, write and test software is complicated and expensive.  You can improve the hardware without change a part but better software can improve the performance on the same hardware search for tesla improvements , or Fuji Cameras , after years the old hardware is more capable because the better software.
So the problem is old thinking at management and engineering level.
By the way I use github, python and cassandra datatabases, so the old engineers can improve too 

[john smith 19]

They believe the software isn't a main component of the ship so they outsource the work trying to save money and if the hardware is ready all it's ok.

IIRC Ford had a gearbox leak they issued a software patch to mitigate it.

This was sometime in the 90's looking at an old back issue of PopSci

This sounds like more an issue with MBA types with zero understanding of technology. 

[king1999]

They believe the software isn't a main component of the ship so they outsource the work trying to save money and if the hardware is ready all it's ok.

IIRC Ford had a gearbox leak they issued a software patch to mitigate it.

This was sometime in the 90's looking at an old back issue of PopSci

This sounds like more an issue with MBA types with zero understanding of technology. 

Yep, for the MBA types, software people are expensive and not as essential as HW people because they can't see the software!

[niwax]

They believe the software isn't a main component of the ship so they outsource the work trying to save money and if the hardware is ready all it's ok.

IIRC Ford had a gearbox leak they issued a software patch to mitigate it.

This was sometime in the 90's looking at an old back issue of PopSci

This sounds like more an issue with MBA types with zero understanding of technology. 

Yep, for the MBA types, software people are expensive and not as essential as HW people because they can't see the software!

I've seen the other end of that, working on a gearbox (not at Ford!), that, three generations on, was still based on the same code from the 90s. I had to write a static analysis tool to tear apart some ten thousand lines of spaghetti code in a central component that no one ever bothered (or budgeted) to update and was now surprised it violated real time constraints when dealing with twice as many speeds. The code had last change comments that were older than me...

[rpapo]

They believe the software isn't a main component of the ship so they outsource the work trying to save money and if the hardware is ready all it's ok.

IIRC Ford had a gearbox leak they issued a software patch to mitigate it.

This was sometime in the 90's looking at an old back issue of PopSci

This sounds like more an issue with MBA types with zero understanding of technology. 

Yep, for the MBA types, software people are expensive and not as essential as HW people because they can't see the software!

I've seen the other end of that, working on a gearbox (not at Ford!), that, three generations on, was still based on the same code from the 90s. I had to write a static analysis tool to tear apart some ten thousand lines of spaghetti code in a central component that no one ever bothered (or budgeted) to update and was now surprised it violated real time constraints when dealing with twice as many speeds. The code had last change comments that were older than me...

Funny how that goes.  Software never rusts, they say, so there are those that think it never needs maintenance either.

[Nomadd]

I've seen the other end of that, working on a gearbox (not at Ford!), that, three generations on, was still based on the same code from the 90s. I had to write a static analysis tool to tear apart some ten thousand lines of spaghetti code in a central component that no one ever bothered (or budgeted) to update and was now surprised it violated real time constraints when dealing with twice as many speeds. The code had last change comments that were older than me...

You should read the story of the Zip2 founder's attempt to write the software. It took a few more realistic employees a while to unravel the giant hairball he came up with and reduce it to digestible chunks.
 What ever happened to that guy?

[zubenelgenubi]

Member:
Great historical sources and anecdotes.

Moderator:
Back to Starliner, please.
If there is a desire for more anecdotes, I would suggest posting in the party thread?