Author Topic: Boeing's Starliner (CST-100) - Discussion Thread 5  (Read 638301 times)

Offline gongora

  • Global Moderator
  • Senior Member
  • *****
  • Posts: 10309
  • US
  • Liked: 14123
  • Likes Given: 6029

Offline gongora

  • Global Moderator
  • Senior Member
  • *****
  • Posts: 10309
  • US
  • Liked: 14123
  • Likes Given: 6029
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #1 on: 06/28/2020 01:02 am »
The discussion about lifeboat scenarios for ISS is over in this thread.  As it's not specific to a particular crew vehicle, any further discussion should move to the Commercial Crew discussion threads.

Online abaddon

  • Senior Member
  • *****
  • Posts: 3091
  • Liked: 3968
  • Likes Given: 5386
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #2 on: 07/06/2020 09:07 pm »
Quote
July 6, 2020
MEDIA ADVISORY M20-076

NASA Provides Update on Commercial Crew Program, Close Call Review of Boeing’s Orbital Flight Test

NASA will host a media teleconference at 2:30 p.m. EDT Tuesday, July 7, to discuss the outcome of its High Visibility Close Call review of the December 2019 uncrewed Orbital Flight Test of Boeing’s Starliner spacecraft.
Tuesday is a good sign as bad news tends to be announced on Fridays to bury it in the dead space of the news cycle over the weekend.  Hopefully we'll get some good info and updated dates.
« Last Edit: 07/06/2020 09:07 pm by abaddon »

Offline leovinus

  • Full Member
  • ****
  • Posts: 1156
  • Porto, Portugal
  • Liked: 909
  • Likes Given: 1794
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #3 on: 07/07/2020 06:48 pm »
Eric Berger asks a good question "Can you tell a bit more where NASA oversight dropped the ball on the software overview for Starliner?"

Answer by Steve: None
Answer by Kathy: Nasa perceived less risk on un-crewed flight (and implies that more oversight would come later on the crewed flight)

The real question is still "Why did Boeing not do end-to-end question of all HW/SW in the first place?"

Offline leovinus

  • Full Member
  • ****
  • Posts: 1156
  • Porto, Portugal
  • Liked: 909
  • Likes Given: 1794
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #4 on: 07/07/2020 06:55 pm »
Eric Berger asks a good question "Can you tell a bit more where NASA oversight dropped the ball on the software overview for Starliner?"

Answer by Steve: None
Answer by Kathy: Nasa perceived less risk on un-crewed flight (and implies that more oversight would come later on the crewed flight)

The real question is still "Why did Boeing not do end-to-end question of all HW/SW in the first place?"

... and why did NASA missed that even though it is in the Shuttle best-practice recommendations ?

Offline Lars-J

  • Senior Member
  • *****
  • Posts: 6809
  • California
  • Liked: 8486
  • Likes Given: 5385
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #5 on: 07/07/2020 07:49 pm »
... and why did NASA missed that even though it is in the Shuttle best-practice recommendations ?

That answer is obvious: Because they trusted Boeing.

Offline woods170

  • IRAS fan
  • Senior Member
  • *****
  • Posts: 12130
  • IRAS fan
  • The Netherlands
  • Liked: 18308
  • Likes Given: 12258
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #6 on: 07/07/2020 08:52 pm »
... and why did NASA missed that even though it is in the Shuttle best-practice recommendations ?

That answer is obvious: Because they trusted Boeing.
NASA forgot about the Russian proverb “Trust but verify”.

Offline Vettedrmr

  • Full Member
  • ****
  • Posts: 1540
  • Hot Springs, AR
  • Liked: 2117
  • Likes Given: 3262
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #7 on: 07/07/2020 09:07 pm »
I blew it and didn't get on-line until about 10 minutes left in the Q&A, but the summary from futurespacetourist was damning both for Boeing skipping stuff it regularly does on its aeronautics work, and NASA for, as you say, just trusting Boeing.  But the board's recommendations were almost verbatim what we had to do on every fighter program I worked on (F-16, F-22, F-35), and the plans had to be in place before we ever approved a single line of code.  Way back in the 80s, all the way until I retired in '13.

For example, we introduced an auto-coder for a portion of a safety-critical control system on the F-35 program, and the JSF Program Office required us to fully characterize the coder.  We limited the constructs that could be used, subjected the ones we did use to the full peer review, unit testing, software integration testing, etc. processes that we did for hand-coded software.  That was done BEFORE any of that software was approved for formal testing (much less actual flight).

And the notion of assessing the code to ensure all logical paths that can be tested ARE tested?  Are you kidding?  100% test coverage (documenting those paths that are impossible to take, and why they can't be exercised) is another one of those I did my entire career.

Not saying they actually did this, but it's almost like Boeing approached OFT-1 as an Alpha test, assuming any bugs wouldn't be very bad.

Good news is that OFT wasn't lost, data was gathered, and the problems were bad enough to wake Boeing and NASA up.

Now to go back and listen to the front end of the news conference.

[EDIT] Listened to the conference, question: WHERE WAS BOEING?

Have a good one,
Mike
« Last Edit: 07/07/2020 10:19 pm by Vettedrmr »
Aviation/space enthusiast, retired control system SW engineer, doesn't know anything!

Offline leovinus

  • Full Member
  • ****
  • Posts: 1156
  • Porto, Portugal
  • Liked: 909
  • Likes Given: 1794
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #8 on: 07/07/2020 11:28 pm »
A quote from the update thread from today:


NASA and Boeing Complete Orbital Flight Test Reviews

[snip]

* Develop a best practices document for use by future programs that implement the shared accountability model used in NASA’s Commercial Crew Program.

[snip]

https://www.nasa.gov/feature/nasa-and-boeing-complete-orbital-flight-test-reviews

That is a good recommendation of course but the non-use of existing best practice documents certainly seems a contributing factor to the issues with OFT-1 Starliner.

As an example, there is a Shuttle "best (software) practices" document named "The Legacy of Space Shuttle Flight Software", NTRS report 20110014946,
discussed in the previous thread with, e.g., end-to-end testing recommendations, and it seems like Boeing and NASA forgot about it.

... and why did NASA missed that even though it is in the Shuttle best-practice recommendations ?

That answer is obvious: Because they trusted Boeing.
NASA forgot about the Russian proverb “Trust but verify”.

Agreed. Based on Woods' observation, today's recommendation above should be augmented with something like

.. and NASA will verify.

Offline xyv

  • Full Member
  • **
  • Posts: 230
  • South of Vandenberg
  • Liked: 514
  • Likes Given: 96
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #9 on: 07/08/2020 01:32 am »
Quote
Lueders: SpX has robust way of doing this. After software is done, they go back and ask engineers if it's operating the way they expected. Strong systems engineering approach. To be sure owners of systems know how they operate. Was big learning experience for us. This was a gift

Well this is refreshing.  It appears that NASA is catching on to modern software development.  When have we ever heard NASA lauding Space X for a "strong systems engineering approach"...let alone when contrasting them to Boeing.  I must say I am pleased...Boeing can recover or not... but NASA I think, is still critical and they appear to be learning.

Offline loekf

  • Full Member
  • *
  • Posts: 169
  • Liked: 204
  • Likes Given: 18
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #10 on: 07/08/2020 04:00 pm »
What the ...

https://twitter.com/spcplcyonline/status/1280580812312895488"

This is what in the real world is called the V-model. Implementation is verified against tests from the implementer. Also never test modules in splendid isolation, usually things pop up during integration. Test your own stuff. It doesn't make sense to blindly implement code or hardware, then throw it across the fence.

Makes me wonder how Boeing manages their other product ranges.
(Vaguely remember they outsourced the 737 Max to a bunch of inexperienced Indian SW engineers. Actually I've seen this in other companies before, but you only get good quality if people understand what they are doing)
« Last Edit: 07/08/2020 04:03 pm by loekf »

Offline mn

  • Full Member
  • ****
  • Posts: 1071
  • United States
  • Liked: 926
  • Likes Given: 342
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #11 on: 07/08/2020 05:09 pm »
Quote
After software is done, they go back and ask engineers if it's operating the way they expected

So how do you NOT do this?

How in the world is this considered revolutionary?

I must be not understanding this at all.

Offline lonestriker

  • Full Member
  • ****
  • Posts: 417
  • Houston We've Had A Problem
  • Liked: 820
  • Likes Given: 5155
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #12 on: 07/08/2020 05:17 pm »
What the ...

https://twitter.com/spcplcyonline/status/1280580812312895488"

This is what in the real world is called the V-model. Implementation is verified against tests from the implementer. Also never test modules in splendid isolation, usually things pop up during integration. Test your own stuff. It doesn't make sense to blindly implement code or hardware, then throw it across the fence.

Makes me wonder how Boeing manages their other product ranges.
(Vaguely remember they outsourced the 737 Max to a bunch of inexperienced Indian SW engineers. Actually I've seen this in other companies before, but you only get good quality if people understand what they are doing)

In a follow-up comment, she said something to the effect that the developers also sign-off on the code at SpaceX.  So, even if there's a separate QA or integration team involved, the original software developer still has ultimately responsibility.  This approach is one way software can be effectively written and tested.  You can't "throw stuff over a wall" when you're done with development like old school methods, where after development, it's the QA team's responsibility/fault.

Modern software practices (at least in the good companies I've worked in or with) are such that developers play a role in development, testing, release/rollback, and ultimate retirement of code.  Developers are involved in the full life-cycle of software, not just the front-end work.


Online abaddon

  • Senior Member
  • *****
  • Posts: 3091
  • Liked: 3968
  • Likes Given: 5386
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #13 on: 07/08/2020 05:59 pm »
Modern software practices (at least in the good companies I've worked in or with) are such that developers play a role in development, testing, release/rollback, and ultimate retirement of code.  Developers are involved in the full life-cycle of software, not just the front-end work.
This is still a (somewhat) new concept and involves not only the work being done but the moving away from specialized roles (e.g. dev-qa) to more generalized positions that are expected to perform all of these jobs, as you note.  This has impacts on hiring as it becomes harder to find developers who are interested/willing to specialize in qa and live operations type roles.  Due to market dominance of players (e.g. Google) pushing this paradigm I expect it will become defacto standard sooner than later even at the older guard companies that have been resistant to change.

Offline Vettedrmr

  • Full Member
  • ****
  • Posts: 1540
  • Hot Springs, AR
  • Liked: 2117
  • Likes Given: 3262
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #14 on: 07/08/2020 06:44 pm »
Quote
After software is done, they go back and ask engineers if it's operating the way they expected

So how do you NOT do this?

How in the world is this considered revolutionary?

I must be not understanding this at all.

There are a few reasons. 

1.) large complex systems have to have the s/w work broken down far enough to get stuff done that the individual developers are spending all their time writing, debugging, and doing low level testing, and unless they just do it on their own initiative, the team doesn't have the time to teach overall system functionality.

2.) This is the tougher one.  In big, old, companies there is a pecking order of engineers.  System engineers are at the top of the heap, then electrical, then software, then testing, then QA. We had to tear that hierarchy down more than one time during my career.  And every time we did the overall process improved, lines of communication REALLY improved, and we got better at meeting our milestones.

Not saying that's the case at Boeing, just my experiences.

Have a good one,
Mike
Aviation/space enthusiast, retired control system SW engineer, doesn't know anything!

Offline xyv

  • Full Member
  • **
  • Posts: 230
  • South of Vandenberg
  • Liked: 514
  • Likes Given: 96
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #15 on: 07/09/2020 01:20 am »
2.) This is the tougher one.  In big, old, companies there is a pecking order of engineers.  System engineers are at the top of the heap, then electrical, then software, then testing, then QA. We had to tear that hierarchy down more than one time during my career.  And every time we did the overall process improved, lines of communication REALLY improved, and we got better at meeting our milestones.

By 'old', I think you mean aerospace companies where systems engineers rule.  Most engineering companies do have a pecking order based on the product.  For my company, the systems engineer is up there but the real royalty is the sensor designer - focal plane array and detector.  To make a camera, all other disciplines are required and it is the systems engineer that is the technical lead.  For a google, the software engineer is it - pretty much the alpha and the omega.  My experience with companies like Boeing is that their culture is based on flight - the wing designer (aerodynamicist) is the golden child - we used to call them 'metal benders'.  They buy avionics, engines and a whole litany of sub components but own the airplane.  Software just hasn't gotten respect as a discipline to attract the best.  You want an X37b to re-enter and land... no problem (I know, legacy North American but same kind of a company...) but if you want to attract the best software engineers how are you going to compete with companies where software is the product?
« Last Edit: 07/09/2020 02:02 am by xyv »

Offline Vettedrmr

  • Full Member
  • ****
  • Posts: 1540
  • Hot Springs, AR
  • Liked: 2117
  • Likes Given: 3262
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #16 on: 07/09/2020 01:50 am »

By 'old', I think you mean aerospace companies where systems engineers rule.  Most engineering companies do have a pecking order based on the product.  For my company, the systems engineer is up there but the real royalty is the sensor designer - focal plane array and detector.  To make a camera, all other disciplines are required and it is the systems engineer that is the technical lead.  For a google, the software engineer is it - pretty much the alpha and the omega.  My experience with companies like Boeing is that their culture is based on flight - the wing designer (aerodynamicist) is the golden child - we used to call them 'metal benders'.  They buy avionics, engines and a whole litany of sub components but own the airplane.  Software just hasn't gotten respect as a discipline to attract the best.  You want a an X37b to re-enter and land... no problem (I know, legacy North American but same kind of a company...) but if you want to attract the best software engineers how are you going to compete with companies where software is the product?

You've pretty much described how it *used* to be.  Aerospace is my database, but I think it was fairly indicative of the engineering industry back then.  Started changing on the F-22 program, and the culture was a lot more integrated on the F-35 program, from the start.
Aviation/space enthusiast, retired control system SW engineer, doesn't know anything!

Online Asteroza

  • Senior Member
  • *****
  • Posts: 2884
  • Liked: 1106
  • Likes Given: 33
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #17 on: 07/09/2020 08:06 am »
So, are we going to actually see these 80 work issues, or is somebody gonna have to pull a FOIA request to claw it out of them?

Offline Vettedrmr

  • Full Member
  • ****
  • Posts: 1540
  • Hot Springs, AR
  • Liked: 2117
  • Likes Given: 3262
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #18 on: 07/09/2020 11:12 am »
So, are we going to actually see these 80 work issues, or is somebody gonna have to pull a FOIA request to claw it out of them?

IMO, Neither.  The summary mentioned that many of the issues involved areas that are ITAR or company proprietary, so I don't think those are available via FOIA.  So I think the areas that were listed are the ones that ARE accessible.  The detail on the individual areas is probably so specific that much of it wouldn't make any sense to us, since the context around it wouldn't be known.

Have a good one,
Mike
Aviation/space enthusiast, retired control system SW engineer, doesn't know anything!

Offline john smith 19

  • Senior Member
  • *****
  • Posts: 10372
  • Everyplaceelse
  • Liked: 2445
  • Likes Given: 13639
Re: Boeing's Starliner (CST-100) - Discussion Thread 5
« Reply #19 on: 07/09/2020 07:10 pm »

In a follow-up comment, she said something to the effect that the developers also sign-off on the code at SpaceX.  So, even if there's a separate QA or integration team involved, the original software developer still has ultimately responsibility.  This approach is one way software can be effectively written and tested.  You can't "throw stuff over a wall" when you're done with development like old school methods, where after development, it's the QA team's responsibility/fault.

Modern software practices (at least in the good companies I've worked in or with) are such that developers play a role in development, testing, release/rollback, and ultimate retirement of code.  Developers are involved in the full life-cycle of software, not just the front-end work.
AFAIK this was SOP for the staff at IBM Federal Systems. The people who wrote the Shuttle software.

Part of why they were the model of a CMM 5 development process.
MCT ITS BFR SS. The worlds first Methane fueled FFSC engined CFRP SS structure A380 sized aerospaceplane tail sitter capable of Earth & Mars atmospheric flight.First flight to Mars by end of 2022 TBC. T&C apply. Trust nothing. Run your own #s "Extraordinary claims require extraordinary proof" R. Simberg."Competitve" means cheaper ¬cheap SCramjet proposed 1956. First +ve thrust 2004. US R&D spend to date > $10Bn. #deployed designs. Zero.

Tags:
 

Advertisement NovaTech
Advertisement Northrop Grumman
Advertisement
Advertisement Margaritaville Beach Resort South Padre Island
Advertisement Brady Kenniston
Advertisement NextSpaceflight
Advertisement Nathan Barker Photography
1