Boeing's Starliner (CST-100) - Discussion Thread 5

[gongora]

New Discussion Thread for Boeing's Starliner/CST-100.

Thread 1:
https://forum.nasaspaceflight.com/index.php?topic=22125.0

Thread 2:
https://forum.nasaspaceflight.com/index.php?topic=32438.0

Thread 3:
https://forum.nasaspaceflight.com/index.php?topic=39244.0

Thread 4:
https://forum.nasaspaceflight.com/index.php?topic=47916.0

News articles:
https://www.nasaspaceflight.com/tag/cst-100/
https://www.nasaspaceflight.com/tag/starliner/

L2 Master Thread:
https://forum.nasaspaceflight.com/index.php?topic=29664.0

--

Civil and respectful at all times and remain on topic please.

[gongora]

The discussion about lifeboat scenarios for ISS is over in this thread.  As it's not specific to a particular crew vehicle, any further discussion should move to the Commercial Crew discussion threads.

[abaddon]

July 6, 2020
MEDIA ADVISORY M20-076

NASA Provides Update on Commercial Crew Program, Close Call Review of Boeing’s Orbital Flight Test

NASA will host a media teleconference at 2:30 p.m. EDT Tuesday, July 7, to discuss the outcome of its High Visibility Close Call review of the December 2019 uncrewed Orbital Flight Test of Boeing’s Starliner spacecraft.

Tuesday is a good sign as bad news tends to be announced on Fridays to bury it in the dead space of the news cycle over the weekend.  Hopefully we'll get some good info and updated dates.

[leovinus]

Eric Berger asks a good question "Can you tell a bit more where NASA oversight dropped the ball on the software overview for Starliner?"

Answer by Steve: None
Answer by Kathy: Nasa perceived less risk on un-crewed flight (and implies that more oversight would come later on the crewed flight)

The real question is still "Why did Boeing not do end-to-end question of all HW/SW in the first place?"

[leovinus]

Eric Berger asks a good question "Can you tell a bit more where NASA oversight dropped the ball on the software overview for Starliner?"

Answer by Steve: None
Answer by Kathy: Nasa perceived less risk on un-crewed flight (and implies that more oversight would come later on the crewed flight)

The real question is still "Why did Boeing not do end-to-end question of all HW/SW in the first place?"

... and why did NASA missed that even though it is in the Shuttle best-practice recommendations ?

[Lars-J]

... and why did NASA missed that even though it is in the Shuttle best-practice recommendations ?

That answer is obvious: Because they trusted Boeing.

[woods170]

... and why did NASA missed that even though it is in the Shuttle best-practice recommendations ?

That answer is obvious: Because they trusted Boeing.

NASA forgot about the Russian proverb “Trust but verify”.

[Vettedrmr]

I blew it and didn't get on-line until about 10 minutes left in the Q&A, but the summary from futurespacetourist was damning both for Boeing skipping stuff it regularly does on its aeronautics work, and NASA for, as you say, just trusting Boeing.  But the board's recommendations were almost verbatim what we had to do on every fighter program I worked on (F-16, F-22, F-35), and the plans had to be in place before we ever approved a single line of code.  Way back in the 80s, all the way until I retired in '13.

For example, we introduced an auto-coder for a portion of a safety-critical control system on the F-35 program, and the JSF Program Office required us to fully characterize the coder.  We limited the constructs that could be used, subjected the ones we did use to the full peer review, unit testing, software integration testing, etc. processes that we did for hand-coded software.  That was done BEFORE any of that software was approved for formal testing (much less actual flight).

And the notion of assessing the code to ensure all logical paths that can be tested ARE tested?  Are you kidding?  100% test coverage (documenting those paths that are impossible to take, and why they can't be exercised) is another one of those I did my entire career.

Not saying they actually did this, but it's almost like Boeing approached OFT-1 as an Alpha test, assuming any bugs wouldn't be very bad.

Good news is that OFT wasn't lost, data was gathered, and the problems were bad enough to wake Boeing and NASA up.

Now to go back and listen to the front end of the news conference.

[EDIT] Listened to the conference, question: WHERE WAS BOEING?

Have a good one,
Mike

[leovinus]

A quote from the update thread from today:

NASA and Boeing Complete Orbital Flight Test Reviews

[snip]

* Develop a best practices document for use by future programs that implement the shared accountability model used in NASA’s Commercial Crew Program.

[snip]

https://www.nasa.gov/feature/nasa-and-boeing-complete-orbital-flight-test-reviews

That is a good recommendation of course but the non-use of existing best practice documents certainly seems a contributing factor to the issues with OFT-1 Starliner.

As an example, there is a Shuttle "best (software) practices" document named "The Legacy of Space Shuttle Flight Software", NTRS report 20110014946,
discussed in the previous thread with, e.g., end-to-end testing recommendations, and it seems like Boeing and NASA forgot about it.

... and why did NASA missed that even though it is in the Shuttle best-practice recommendations ?

That answer is obvious: Because they trusted Boeing.

NASA forgot about the Russian proverb “Trust but verify”.

Agreed. Based on Woods' observation, today's recommendation above should be augmented with something like

.. and NASA will verify.

[xyv]

Lueders: SpX has robust way of doing this. After software is done, they go back and ask engineers if it's operating the way they expected. Strong systems engineering approach. To be sure owners of systems know how they operate. Was big learning experience for us. This was a gift

Well this is refreshing.  It appears that NASA is catching on to modern software development.  When have we ever heard NASA lauding Space X for a "strong systems engineering approach"...let alone when contrasting them to Boeing.  I must say I am pleased...Boeing can recover or not... but NASA I think, is still critical and they appear to be learning.

[loekf]

What the ...

https://twitter.com/spcplcyonline/status/1280580812312895488"

This is what in the real world is called the V-model. Implementation is verified against tests from the implementer. Also never test modules in splendid isolation, usually things pop up during integration. Test your own stuff. It doesn't make sense to blindly implement code or hardware, then throw it across the fence.

Makes me wonder how Boeing manages their other product ranges.
(Vaguely remember they outsourced the 737 Max to a bunch of inexperienced Indian SW engineers. Actually I've seen this in other companies before, but you only get good quality if people understand what they are doing)

[mn]

After software is done, they go back and ask engineers if it's operating the way they expected

So how do you NOT do this?

How in the world is this considered revolutionary?

I must be not understanding this at all.

[lonestriker]

What the ...

https://twitter.com/spcplcyonline/status/1280580812312895488"

This is what in the real world is called the V-model. Implementation is verified against tests from the implementer. Also never test modules in splendid isolation, usually things pop up during integration. Test your own stuff. It doesn't make sense to blindly implement code or hardware, then throw it across the fence.

Makes me wonder how Boeing manages their other product ranges.
(Vaguely remember they outsourced the 737 Max to a bunch of inexperienced Indian SW engineers. Actually I've seen this in other companies before, but you only get good quality if people understand what they are doing)

In a follow-up comment, she said something to the effect that the developers also sign-off on the code at SpaceX.  So, even if there's a separate QA or integration team involved, the original software developer still has ultimately responsibility.  This approach is one way software can be effectively written and tested.  You can't "throw stuff over a wall" when you're done with development like old school methods, where after development, it's the QA team's responsibility/fault.

Modern software practices (at least in the good companies I've worked in or with) are such that developers play a role in development, testing, release/rollback, and ultimate retirement of code.  Developers are involved in the full life-cycle of software, not just the front-end work.

[abaddon]

Modern software practices (at least in the good companies I've worked in or with) are such that developers play a role in development, testing, release/rollback, and ultimate retirement of code.  Developers are involved in the full life-cycle of software, not just the front-end work.

This is still a (somewhat) new concept and involves not only the work being done but the moving away from specialized roles (e.g. dev-qa) to more generalized positions that are expected to perform all of these jobs, as you note.  This has impacts on hiring as it becomes harder to find developers who are interested/willing to specialize in qa and live operations type roles.  Due to market dominance of players (e.g. Google) pushing this paradigm I expect it will become defacto standard sooner than later even at the older guard companies that have been resistant to change.

[Vettedrmr]

After software is done, they go back and ask engineers if it's operating the way they expected

So how do you NOT do this?

How in the world is this considered revolutionary?

I must be not understanding this at all.

There are a few reasons. 

1.) large complex systems have to have the s/w work broken down far enough to get stuff done that the individual developers are spending all their time writing, debugging, and doing low level testing, and unless they just do it on their own initiative, the team doesn't have the time to teach overall system functionality.

2.) This is the tougher one.  In big, old, companies there is a pecking order of engineers.  System engineers are at the top of the heap, then electrical, then software, then testing, then QA. We had to tear that hierarchy down more than one time during my career.  And every time we did the overall process improved, lines of communication REALLY improved, and we got better at meeting our milestones.

Not saying that's the case at Boeing, just my experiences.

Have a good one,
Mike

[xyv]

2.) This is the tougher one.  In big, old, companies there is a pecking order of engineers.  System engineers are at the top of the heap, then electrical, then software, then testing, then QA. We had to tear that hierarchy down more than one time during my career.  And every time we did the overall process improved, lines of communication REALLY improved, and we got better at meeting our milestones.

By 'old', I think you mean aerospace companies where systems engineers rule.  Most engineering companies do have a pecking order based on the product.  For my company, the systems engineer is up there but the real royalty is the sensor designer - focal plane array and detector.  To make a camera, all other disciplines are required and it is the systems engineer that is the technical lead.  For a google, the software engineer is it - pretty much the alpha and the omega.  My experience with companies like Boeing is that their culture is based on flight - the wing designer (aerodynamicist) is the golden child - we used to call them 'metal benders'.  They buy avionics, engines and a whole litany of sub components but own the airplane.  Software just hasn't gotten respect as a discipline to attract the best.  You want an X37b to re-enter and land... no problem (I know, legacy North American but same kind of a company...) but if you want to attract the best software engineers how are you going to compete with companies where software is the product?

[Vettedrmr]

By 'old', I think you mean aerospace companies where systems engineers rule.  Most engineering companies do have a pecking order based on the product.  For my company, the systems engineer is up there but the real royalty is the sensor designer - focal plane array and detector.  To make a camera, all other disciplines are required and it is the systems engineer that is the technical lead.  For a google, the software engineer is it - pretty much the alpha and the omega.  My experience with companies like Boeing is that their culture is based on flight - the wing designer (aerodynamicist) is the golden child - we used to call them 'metal benders'.  They buy avionics, engines and a whole litany of sub components but own the airplane.  Software just hasn't gotten respect as a discipline to attract the best.  You want a an X37b to re-enter and land... no problem (I know, legacy North American but same kind of a company...) but if you want to attract the best software engineers how are you going to compete with companies where software is the product?

You've pretty much described how it *used* to be.  Aerospace is my database, but I think it was fairly indicative of the engineering industry back then.  Started changing on the F-22 program, and the culture was a lot more integrated on the F-35 program, from the start.

[Asteroza]

So, are we going to actually see these 80 work issues, or is somebody gonna have to pull a FOIA request to claw it out of them?

[Vettedrmr]

So, are we going to actually see these 80 work issues, or is somebody gonna have to pull a FOIA request to claw it out of them?

IMO, Neither.  The summary mentioned that many of the issues involved areas that are ITAR or company proprietary, so I don't think those are available via FOIA.  So I think the areas that were listed are the ones that ARE accessible.  The detail on the individual areas is probably so specific that much of it wouldn't make any sense to us, since the context around it wouldn't be known.

Have a good one,
Mike

[john smith 19]

In a follow-up comment, she said something to the effect that the developers also sign-off on the code at SpaceX.  So, even if there's a separate QA or integration team involved, the original software developer still has ultimately responsibility.  This approach is one way software can be effectively written and tested.  You can't "throw stuff over a wall" when you're done with development like old school methods, where after development, it's the QA team's responsibility/fault.

Modern software practices (at least in the good companies I've worked in or with) are such that developers play a role in development, testing, release/rollback, and ultimate retirement of code.  Developers are involved in the full life-cycle of software, not just the front-end work.

AFAIK this was SOP for the staff at IBM Federal Systems. The people who wrote the Shuttle software.

Part of why they were the model of a CMM 5 development process.