Author Topic: Boeing's Starliner (CST-100) - Discussion Thread 5  (Read 631432 times)

Offline anof

  • Member
  • Posts: 63
  • Michigan
  • Liked: 54
  • Likes Given: 106
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #40 on: 07/29/2020 01:48 pm »
So, just to make sure that I'm on top of this and my reading comprehension hasn't slipped any:

1) The problems with Starliner are almost entirely due to control system software QA shortfalls;

I would disagree with this, if you're thinking of S/W QA in the traditional, independent department, sense. The problem is much more ingrained in their development and testing culture: it seems that Boeing's development team did an incomplete set of peer reviews, low level unit testing, and mid-level integration testing.  It also seems like they did little to no actual hardware in the loop systems integration testing, instead using software simulations and assuming those were valid.  I'm sure they did more work than the IRT report summaries we read imply, but for sure there were a lot of holes to fill in.


I would also add that NASA trusted Boeing and performed less oversight.

Offline DigitalMan

  • Full Member
  • ****
  • Posts: 1683
  • Liked: 1184
  • Likes Given: 76
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #41 on: 07/29/2020 03:50 pm »
Also, NASA hasn't done the safety review yet (as of the last press conference on Starliner). I don't see them proceeding without it.

Online butters

  • Senior Member
  • *****
  • Posts: 2399
  • Liked: 1694
  • Likes Given: 600
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #42 on: 07/29/2020 04:01 pm »
In addition to the software and software testing shortfalls, Boeing has also had to modify the electronics hardware in the radio transceiver systems to resolve the problems they had trying to communicate with the spacecraft over TDRS after launch.

Offline Nomadd

  • Senior Member
  • *****
  • Posts: 8845
  • Lower 48
  • Liked: 60456
  • Likes Given: 1308
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #43 on: 07/29/2020 04:45 pm »
 The chute failure on the abort test had nothing to do with software. But all the problems seemed to have the same basic cause, which you might call an attitude problem. They weren't from a design issue or other glitch, like a valve not performing or material incompatibility. They were from screwups that plain shouldn't have happened, and fixing that is a lot harder than fixing a piece of hardware.
 The 737 issues were also not just software by a longshot. A lack of redundant sensors, lack of notification that sensors had a problem and lack of proper documentation and training all contributed.
 But, I suspect the main issue is the same for both projects. A lack of Joe Sutters with the cajones to stand up to management.
« Last Edit: 07/29/2020 04:49 pm by Nomadd »
Those who danced were thought to be quite insane by those who couldn't hear the music.

Offline intelati

Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #44 on: 07/29/2020 05:46 pm »
But, I suspect the main issue is the same for both projects. A lack of Joe Sutters with the cajones to stand up to management.

Somewhat off topic, but just did a quick dive into Sutter.

from a 2010 article, there's this scary passage forshadowing these issues.

Quote
Boeing does need to keep a closer eye on its outsourced engineering work, Sutter said. “Engineers tend to drift off and do their own thing if you let them. The only way you maintain discipline is to watch them like a hawk. And that hasn’t always happened at Boeing.”

https://blog.seattlepi.com/aerospace/2010/07/16/boeings-sutter-737-replacement-timing-depends-on-engines/
Starships are meant to fly

Offline freddo411

  • Full Member
  • ****
  • Posts: 1053
  • Liked: 1198
  • Likes Given: 3422
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #45 on: 07/29/2020 06:24 pm »
But, I suspect the main issue is the same for both projects. A lack of Joe Sutters with the cajones to stand up to management.

Somewhat off topic, but just did a quick dive into Sutter.

from a 2010 article, there's this scary passage forshadowing these issues.

Quote
Boeing does need to keep a closer eye on its outsourced engineering work, Sutter said. “Engineers tend to drift off and do their own thing if you let them. The only way you maintain discipline is to watch them like a hawk. And that hasn’t always happened at Boeing.”

https://blog.seattlepi.com/aerospace/2010/07/16/boeings-sutter-737-replacement-timing-depends-on-engines/

I'm not sure how others read that quote, but I find it insulting to engineers and engineering discipline.   I wouldn't want to work at a company that even had the whiff of that kind of attitude toward engineering.   

Virtually every engineer I know is uniquely qualified and called upon to make critical, rational tradeoff decisions in most tasks.   Supervision is a collaborative effort to define goals.   If things descend into a "build this just like I said" scenario, you lose the talent and expertise of the engineer.

Makes me wonder if there was an engineer who might have said:  "We should run an all up integrated test with the booster" ....    Judging by posters on NSF, there is a high likelyhood there was.
« Last Edit: 07/29/2020 06:24 pm by freddo411 »

Offline Frogstar_Robot

  • Full Member
  • ****
  • Posts: 498
  • Liked: 723
  • Likes Given: 138
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #46 on: 07/29/2020 06:42 pm »
I think I understand the quote. He is referring to outsourced engineering work. Outsourcing contains many hidden dangers while the management are only looking at the bottom line. I have been on both sides of the fence.

In my experience, companies are not good at capturing requirements. In house, this is mitigated because the engineers usually have long experience of the application area, and have close contact with other stakeholders. This gets worse over time to a point where in-house specs are very poor to non-existent.

When a company tries to outsource a project, they fall flat on their face, because the out-sourced engineers no longer have access to all the company knowledge. They work from an incomplete and inadequate spec, and the contact with the customer is through "liaison" who add an extra level of bureaucracy. When the customer says, "oh we meant X not Y", the supplier regards that as a contract change that should be billed for, which obviously the customer thinks the supplier should pay for since "they got it wrong".

That of course, assume that the supplier's engineers are competent. If not, then all bets are off...
Rule 1: Be civil. Respect other members.
Rule 3: No "King of the Internet" attitudes.

Online niwax

  • Full Member
  • ****
  • Posts: 1424
  • Germany
    • SpaceX Booster List
  • Liked: 2042
  • Likes Given: 166
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #47 on: 07/29/2020 06:45 pm »
But, I suspect the main issue is the same for both projects. A lack of Joe Sutters with the cajones to stand up to management.

Somewhat off topic, but just did a quick dive into Sutter.

from a 2010 article, there's this scary passage forshadowing these issues.

Quote
Boeing does need to keep a closer eye on its outsourced engineering work, Sutter said. “Engineers tend to drift off and do their own thing if you let them. The only way you maintain discipline is to watch them like a hawk. And that hasn’t always happened at Boeing.”

https://blog.seattlepi.com/aerospace/2010/07/16/boeings-sutter-737-replacement-timing-depends-on-engines/

I highly recommend his autobiography that centers around the development of the 747, the last chapter deals with his work on the Rogers commission after Challenger and has a lot more foreshadowing about planes, space and NASA.
Which booster has the most soot? SpaceX booster launch history! (discussion)

Offline intelati

Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #48 on: 07/29/2020 06:53 pm »
I think I understand the quote. He is referring to outsourced engineering work. Outsourcing contains many hidden dangers while the management are only looking at the bottom line. I have been on both sides of the fence.

In my experience, companies are not good at capturing requirements. In house, this is mitigated because the engineers usually have long experience of the application area, and have close contact with other stakeholders. This gets worse over time to a point where in-house specs are very poor to non-existent.

When a company tries to outsource a project, they fall flat on their face, because the out-sourced engineers no longer have access to all the company knowledge. They work from an incomplete and inadequate spec, and the contact with the customer is through "liaison" who add an extra level of bureaucracy. When the customer says, "oh we meant X not Y", the supplier regards that as a contract change that should be billed for, which obviously the customer thinks the supplier should pay for since "they got it wrong".

That of course, assume that the supplier's engineers are competent. If not, then all bets are off...

Look at the post just a couple above mine. The quote rang a bell with me in this context.

So, just to make sure that I'm on top of this and my reading comprehension hasn't slipped any:

1) The problems with Starliner are almost entirely due to control system software QA shortfalls;

I would disagree with this, if you're thinking of S/W QA in the traditional, independent department, sense. The problem is much more ingrained in their development and testing culture: it seems that Boeing's development team did an incomplete set of peer reviews, low level unit testing, and mid-level integration testing.

But yes, I certainly will check out his Autobiography
« Last Edit: 07/29/2020 06:54 pm by intelati »
Starships are meant to fly

Offline Nomadd

  • Senior Member
  • *****
  • Posts: 8845
  • Lower 48
  • Liked: 60456
  • Likes Given: 1308
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #49 on: 07/30/2020 02:37 pm »

Makes me wonder if there was an engineer who might have said:  "We should run an all up integrated test with the booster" ....    Judging by posters on NSF, there is a high likelyhood there was.
Of course there was. The problem was a lack of someone willing to risk his job by insisting. Sutter thought he'd be fired more than once for flat out refusing to go along with management idiocy.
 I have a little more than an armchair opinion on the matter myself. Part of the reason I've had more jobs than I can easily count. That, and an unfortunate lack of diplomacy at times.
 I'd like to think there's a solution, but when it involves the people making the decisions giving up power, It's hard to be optimistic.

 As far as the thread subject, I hope they're doing a little more than just piling on procedures. But finding and enabling the few people who can make projects like these shine doesn't seem to be that common now days.
 I'd do it, but it turns out my roof isn't all that waterproof with a 60mph wind driving the rain, so I have to work on that instead.
« Last Edit: 09/11/2020 03:56 am by Nomadd »
Those who danced were thought to be quite insane by those who couldn't hear the music.

Online LouScheffer

  • Senior Member
  • *****
  • Posts: 3397
  • Liked: 6134
  • Likes Given: 853
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #50 on: 07/30/2020 09:08 pm »

Makes me wonder if there was an engineer who might have said:  "We should run an all up integrated test with the booster" ....    Judging by posters on NSF, there is a high likelyhood there was.
Of course there was. The problem was a lack of someone willing to risk his job by insisting. Sutter thought he'd be fired more than once by flat out refusing to go along with management idiocy.
Older engineers have a lot of tactical disadvantages.  The last language they programmed in was likely Fortran, and they are entirely unfamiliar with Slack and GitHub.  And they have higher salaries.

But this situation is exactly where they can earn their keep.  This is not the first time that folks have tried to save money by skipping the integrated test, convinced themselves it was OK, and paid by loss of mission.  Remember the Ariane 5 first flight?  Remembering where things screwed up, and standing up for not repeating the same mistake, is the job of senior engineering staff.


Offline savantu

  • Full Member
  • **
  • Posts: 227
  • Romania
  • Liked: 293
  • Likes Given: 131
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #51 on: 08/03/2020 12:59 pm »
The bar has been set very high after yesterday's succesful landing.

Offline john smith 19

  • Senior Member
  • *****
  • Posts: 10357
  • Everyplaceelse
  • Liked: 2436
  • Likes Given: 13612
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #52 on: 08/03/2020 01:27 pm »
The bar has been set very high after yesterday's succesful landing.
True, but not unfairlyhigh.

How many times have they (and LM) played the "We are the safe pair of hands" card?

Going along with that line of reasoning (without checking actual performance against claims gives the illusion of security, but not the reality. I'd suggest all of those faults should have been picked up in testing, not during an actual missions. :(

Making this stuff work is tough.
MCT ITS BFR SS. The worlds first Methane fueled FFSC engined CFRP SS structure A380 sized aerospaceplane tail sitter capable of Earth & Mars atmospheric flight.First flight to Mars by end of 2022 TBC. T&C apply. Trust nothing. Run your own #s "Extraordinary claims require extraordinary proof" R. Simberg."Competitve" means cheaper ¬cheap SCramjet proposed 1956. First +ve thrust 2004. US R&D spend to date > $10Bn. #deployed designs. Zero.

Offline ThePonjaX

  • Full Member
  • *
  • Posts: 164
  • BsAs. - Argentina
  • Liked: 245
  • Likes Given: 984
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #53 on: 08/03/2020 04:21 pm »

Makes me wonder if there was an engineer who might have said:  "We should run an all up integrated test with the booster" ....    Judging by posters on NSF, there is a high likelyhood there was.
Of course there was. The problem was a lack of someone willing to risk his job by insisting. Sutter thought he'd be fired more than once by flat out refusing to go along with management idiocy.
Older engineers have a lot of tactical disadvantages.  The last language they programmed in was likely Fortran, and they are entirely unfamiliar with Slack and GitHub.  And they have higher salaries.

But this situation is exactly where they can earn their keep.  This is not the first time that folks have tried to save money by skipping the integrated test, convinced themselves it was OK, and paid by loss of mission.  Remember the Ariane 5 first flight?  Remembering where things screwed up, and standing up for not repeating the same mistake, is the job of senior engineering staff.

I'm a kind of old engineer ... in my fifties , more than 30 years working on IT and I can tell you ( a put some post before on these) the problem is other:
They believe the software isn't a main component of the ship so they outsource the work trying to save money and if the hardware is ready all it's ok.
Of course these is plain wrong. Software can compensate a a hardware problem not the other way around. Design, write and test software is complicated and expensive.  You can improve the hardware without change a part but better software can improve the performance on the same hardware search for tesla improvements , or Fuji Cameras , after years the old hardware is more capable because the better software.
So the problem is old thinking at management and engineering level.
By the way I use github, python and cassandra datatabases, so the old engineers can improve too  ;)

Offline john smith 19

  • Senior Member
  • *****
  • Posts: 10357
  • Everyplaceelse
  • Liked: 2436
  • Likes Given: 13612
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #54 on: 08/05/2020 06:32 am »
They believe the software isn't a main component of the ship so they outsource the work trying to save money and if the hardware is ready all it's ok.
IIRC Ford had a gearbox leak they issued a software patch to mitigate it.

This was sometime in the 90's looking at an old back issue of PopSci

This sounds like more an issue with MBA types with zero understanding of technology.  :(
MCT ITS BFR SS. The worlds first Methane fueled FFSC engined CFRP SS structure A380 sized aerospaceplane tail sitter capable of Earth & Mars atmospheric flight.First flight to Mars by end of 2022 TBC. T&C apply. Trust nothing. Run your own #s "Extraordinary claims require extraordinary proof" R. Simberg."Competitve" means cheaper ¬cheap SCramjet proposed 1956. First +ve thrust 2004. US R&D spend to date > $10Bn. #deployed designs. Zero.

Offline king1999

  • Full Member
  • ****
  • Posts: 441
  • F-Niner Fan
  • Atlanta, GA
  • Liked: 306
  • Likes Given: 1282
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #55 on: 08/05/2020 01:10 pm »
They believe the software isn't a main component of the ship so they outsource the work trying to save money and if the hardware is ready all it's ok.
IIRC Ford had a gearbox leak they issued a software patch to mitigate it.

This was sometime in the 90's looking at an old back issue of PopSci

This sounds like more an issue with MBA types with zero understanding of technology.  :(

Yep, for the MBA types, software people are expensive and not as essential as HW people because they can't see the software!

Online niwax

  • Full Member
  • ****
  • Posts: 1424
  • Germany
    • SpaceX Booster List
  • Liked: 2042
  • Likes Given: 166
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #56 on: 08/05/2020 04:36 pm »
They believe the software isn't a main component of the ship so they outsource the work trying to save money and if the hardware is ready all it's ok.
IIRC Ford had a gearbox leak they issued a software patch to mitigate it.

This was sometime in the 90's looking at an old back issue of PopSci

This sounds like more an issue with MBA types with zero understanding of technology.  :(

Yep, for the MBA types, software people are expensive and not as essential as HW people because they can't see the software!

I've seen the other end of that, working on a gearbox (not at Ford!), that, three generations on, was still based on the same code from the 90s. I had to write a static analysis tool to tear apart some ten thousand lines of spaghetti code in a central component that no one ever bothered (or budgeted) to update and was now surprised it violated real time constraints when dealing with twice as many speeds. The code had last change comments that were older than me...

Offline rpapo

Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #57 on: 08/05/2020 06:22 pm »
They believe the software isn't a main component of the ship so they outsource the work trying to save money and if the hardware is ready all it's ok.
IIRC Ford had a gearbox leak they issued a software patch to mitigate it.

This was sometime in the 90's looking at an old back issue of PopSci

This sounds like more an issue with MBA types with zero understanding of technology.  :(

Yep, for the MBA types, software people are expensive and not as essential as HW people because they can't see the software!

I've seen the other end of that, working on a gearbox (not at Ford!), that, three generations on, was still based on the same code from the 90s. I had to write a static analysis tool to tear apart some ten thousand lines of spaghetti code in a central component that no one ever bothered (or budgeted) to update and was now surprised it violated real time constraints when dealing with twice as many speeds. The code had last change comments that were older than me...
Funny how that goes.  Software never rusts, they say, so there are those that think it never needs maintenance either.
Following the space program since before Apollo 8.

Offline Nomadd

  • Senior Member
  • *****
  • Posts: 8845
  • Lower 48
  • Liked: 60456
  • Likes Given: 1308
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #58 on: 08/05/2020 09:26 pm »
I've seen the other end of that, working on a gearbox (not at Ford!), that, three generations on, was still based on the same code from the 90s. I had to write a static analysis tool to tear apart some ten thousand lines of spaghetti code in a central component that no one ever bothered (or budgeted) to update and was now surprised it violated real time constraints when dealing with twice as many speeds. The code had last change comments that were older than me...
You should read the story of the Zip2 founder's attempt to write the software. It took a few more realistic employees a while to unravel the giant hairball he came up with and reduce it to digestible chunks.
 What ever happened to that guy?
Those who danced were thought to be quite insane by those who couldn't hear the music.

Online zubenelgenubi

  • Global Moderator
  • Senior Member
  • *****
  • Posts: 11345
  • Arc to Arcturus, then Spike to Spica
  • Sometimes it feels like Trantor in the time of Hari Seldon
  • Liked: 7543
  • Likes Given: 73581
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #59 on: 08/06/2020 03:54 pm »
Member:
Great historical sources and anecdotes. 8)

Moderator:
Back to Starliner, please.
If there is a desire for more anecdotes, I would suggest posting in the party thread?
Support your local planetarium! (COVID-panic and forward: Now more than ever.) My current avatar is saying "i wants to go uppies!" Yes, there are God-given rights. Do you wish to gainsay the Declaration of Independence?

Tags:
 

Advertisement NovaTech
Advertisement Northrop Grumman
Advertisement
Advertisement Margaritaville Beach Resort South Padre Island
Advertisement Brady Kenniston
Advertisement NextSpaceflight
Advertisement Nathan Barker Photography
1