Commercial Crew - Discussion Thread 3

[FutureSpaceTourist]

Wow

https://twitter.com/rdanglephoto/status/1208890495176495105

Dragon x Starliner. Space is for everyone. #SpaceX #Falcon9 #Dragon #ULALaunch #AtlasV #Starliner

[su27k]

Henry Vanderbilt made an excellent point on nasawatch:

There's been a lot of focus on Boeing's cultural problems over the last couple days, but I think we may be missing a more fundamental issue here.

I think it's fair to say that the NASA-supervised development process in Commercial Crew has intentionally applied large amounts of time and money and manpower and paperwork to the goal of getting every nitpicking detail right the first time. It's the established NASA way.

As opposed to the old "Right Stuff" era model of fly early, see what breaks, fix it and fly again, rinse & repeat - historically, producing far more rapid progress at far lower cost (along with occasional spectacular blowups.)

So, tell me again what we gained here for spending those extra years and billions on attempting to make Commercial Crew first-flight perfect, rather than continuing the older faster looser development model from COTS (as some of us vehemently argued for)?

I think the recent issues (not just OFT, but the parachute issue in pad abort too) shows serious deficiency in NASA's oversight/insight process on Commercial Crew, at least on Boeing side. Boeing's problems are fairly simple to catch, so why didn't NASA catch them?

The stated reason for NASA using FAR in CCtCAP, which led to schedule delays and higher cost (had to be covered by providers themselves), is that they needed more assurance on crewed missions, but this latest string of anomalies shows this line of reasoning is broken. It seems to me we'd be better of if they use SAA instead and trade more paperwork for more flight testing.

[gongora]

Henry Vanderbilt made an excellent point on nasawatch:

There's been a lot of focus on Boeing's cultural problems over the last couple days, but I think we may be missing a more fundamental issue here.

I think it's fair to say that the NASA-supervised development process in Commercial Crew has intentionally applied large amounts of time and money and manpower and paperwork to the goal of getting every nitpicking detail right the first time. It's the established NASA way.

As opposed to the old "Right Stuff" era model of fly early, see what breaks, fix it and fly again, rinse & repeat - historically, producing far more rapid progress at far lower cost (along with occasional spectacular blowups.)

So, tell me again what we gained here for spending those extra years and billions on attempting to make Commercial Crew first-flight perfect, rather than continuing the older faster looser development model from COTS (as some of us vehemently argued for)?

I think the recent issues (not just OFT, but the parachute issue in pad abort too) shows serious deficiency in NASA's oversight/insight process on Commercial Crew, at least on Boeing side. Boeing's problems are fairly simple to catch, so why didn't NASA catch them?

The stated reason for NASA using FAR in CCtCAP, which led to schedule delays and higher cost (had to be covered by providers themselves), is that they needed more assurance on crewed missions, but this latest string of anomalies shows this line of reasoning is broken. It seems to me we'd be better of if they use SAA instead and trade more paperwork for more flight testing.

The old "Right Stuff" model was not low cost.  It was fast because the government was throwing lots of money at it, the vehicles were simpler, safety standards were looser.  Commercial Crew followed the same contracting structure as the cargo program, SAA's followed by FAR contracts.

edit:  also, Boeing's problems have not been "simple to catch" by the customer.  It's Boeing personnel that should have caught them.

[woods170]

Henry Vanderbilt made an excellent point on nasawatch:

There's been a lot of focus on Boeing's cultural problems over the last couple days, but I think we may be missing a more fundamental issue here.

I think it's fair to say that the NASA-supervised development process in Commercial Crew has intentionally applied large amounts of time and money and manpower and paperwork to the goal of getting every nitpicking detail right the first time. It's the established NASA way.

As opposed to the old "Right Stuff" era model of fly early, see what breaks, fix it and fly again, rinse & repeat - historically, producing far more rapid progress at far lower cost (along with occasional spectacular blowups.)

So, tell me again what we gained here for spending those extra years and billions on attempting to make Commercial Crew first-flight perfect, rather than continuing the older faster looser development model from COTS (as some of us vehemently argued for)?

I think the recent issues (not just OFT, but the parachute issue in pad abort too) shows serious deficiency in NASA's oversight/insight process on Commercial Crew, at least on Boeing side. Boeing's problems are fairly simple to catch, so why didn't NASA catch them?

The stated reason for NASA using FAR in CCtCAP, which led to schedule delays and higher cost (had to be covered by providers themselves), is that they needed more assurance on crewed missions, but this latest string of anomalies shows this line of reasoning is broken. It seems to me we'd be better of if they use SAA instead and trade more paperwork for more flight testing.

The old "Right Stuff" model was not low cost.  It was fast because the government was throwing lots of money at it, the vehicles were simpler, safety standards were looser.  Commercial Crew followed the same contracting structure as the cargo program, SAA's followed by FAR contracts.

edit:  also, Boeing's problems have not been "simple to catch" by the customer.  It's Boeing personnel that should have caught them.

Emphasis mine.

Not correct. COTS was all SAA and contained all demonstration missions of Dragon and Cygnus. Only the operational CRS is FAR based.
CCP on the other hand was all SAA up to and including CCiCAP, but no demonstration missions. All CCP demo missions are part of FAR based CCtCAP.
And that is a major difference.

[gongora]

The FAR contracts were awarded before the demo missions flew, so I'm not seeing that big of a difference.

[woods170]

The FAR contracts were awarded before the demo missions flew, so I'm not seeing that big of a difference.

The difference is the demo missions being performed under SAA for COTS, which provided a very substantial amount of flexibility. The CCP demo missions are under FAR contracting, which is much more intrusive in nature.

[MakeItSo]

The after landing news conference left me baffled.  A reporter finally asked an intelligent question & to his credit Jim Chilton answered.   The clock was off by 11 hours!!! Now as a professional software engineer I'm just aghast that their autonomous navigation program didn't check any sensors (GPS for altitude, inertial measurement,  etc) before blindly firing it's attitude control thrusters? When something occurs such as orbital insertion burn, a success or failure flag isn't set that can be checked? 

Sorry there is much bigger concern here of the software than that it just got the wrong time from the Atlas.  This is supposed to be an autonomous system.  You can't simply rely on one thing, the clock, as input.  Can you imagine a self driving car relying only on the dashboard clock?  Much of the Boeing software design staff and software QA team should be fired.  At least their management should be.

And where were all the NASA civil servants when they did the design reviews?  Obviously they didn't have the experience to ask the correct questions. 

And none of this was caught in any of the simulations!  So are we to trust the simulation of the in flight abort?  NASA should definitely insist on a IFA test.  Especially with it occurring while the solid rocket boosters are still firing.

[woods170]

The after landing news conference left me baffled.  A reporter finally asked an intelligent question & to his credit Jim Chilton answered.   The clock was off by 11 hours!!! Now as a professional software engineer I'm just aghast that their autonomous navigation program didn't check any sensors (GPS for altitude, inertial measurement,  etc) before blindly firing it's attitude control thrusters? When something occurs such as orbital insertion burn, a success or failure flag isn't set that can be checked? 

Sorry there is much bigger concern here of the software than that it just got the wrong time from the Atlas.  This is supposed to be an autonomous system.  You can't simply rely on one thing, the clock, as input.  Can you imagine a self driving car relying only on the dashboard clock?  Much of the Boeing software design staff and software QA team should be fired.  At least their management should be.

And where were all the NASA civil servants when they did the design reviews?  Obviously they didn't have the experience to ask the correct questions. 

And none of this was caught in any of the simulations!  So are we to trust the simulation of the in flight abort?  NASA should definitely insist on a IFA test.  Especially with it occurring while the solid rocket boosters are still firing.

Welcome to the forum.

Hate to disappoint you but none of what you think should happen will happen.

[Steven Pietrobon]

Much of the Boeing software design staff and software QA team should be fired.  At least their management should be.

That is absolutely the wrong thing to do and could lead to a worse outcome since the new team could make even more mistakes. The people who wrote and managed the software will have learned a lot more than if the mission had gone perfectly, and will be stronger for it.

[docmordrid]

The after landing news conference left me baffled.  A reporter finally asked an intelligent question & to his credit Jim Chilton answered.   The clock was off by 11 hours!!! Now as a professional software engineer I'm just aghast that their autonomous navigation program didn't check any sensors (GPS for altitude, inertial measurement,  etc) before blindly firing it's attitude control thrusters? When something occurs such as orbital insertion burn, a success or failure flag isn't set that can be checked? 
>

Sounds like they were preparing to power up Atlas V at T-11:20.  Did Starliner read power up instead of launch time?

SpaceFlightNow...

The Atlas 5 countdown typically lasts nearly seven hours for a satellite launch. For Starliner’s Orbital Flight Test, , which will head to the International Space Station on an unpiloted shakedown mission, the countdown will run 11 hours, 20 minutes.

The countdown is set to commence at 7:16 p.m. EST Thursday (0016 GMT Friday), when teams in the Atlas Spaceflight Operations Center at Cape Canaveral will begin procedures to power up the launch vehicle and conduct propulsion and guidance, navigation and control system checkouts.

[savantu]

Much of the Boeing software design staff and software QA team should be fired.  At least their management should be.

That is absolutely the wrong thing to do and could lead to a worse outcome since the new team could make even more mistakes. The people who wrote and managed the software will have learned a lot more than if the mission had gone perfectly, and will be stronger for it.

It doesn't work like that in the real world. There is a saying about the old dog and new tricks and it also applies to us, humans. There is a point where you need to reshuffle the team so questions like " Why " are flying again and critical thinking is employed. The Boeing team SW design and implementation is so poor, that just getting them a new boss will not repair the architecture& implementation errors. Their sufficiency and boredom/lack of involvement lead to this failure. Not management.

[woods170]

Much of the Boeing software design staff and software QA team should be fired.  At least their management should be.

That is absolutely the wrong thing to do and could lead to a worse outcome since the new team could make even more mistakes. The people who wrote and managed the software will have learned a lot more than if the mission had gone perfectly, and will be stronger for it.

I disagree with you here Steven. Teams don't automatically learn from their mistakes. I've seen that happen numerous times over the past two decades. The Boeing SW team is NOT going to automatically learn from this mishap. Nor will they be automatically stronger because of the mishap.
It will require significant effort and willingness from both the team members and their management to learn the lessons that need to be learned. Only then can the team become stronger.

[LastWyzard]

Much of the Boeing software design staff and software QA team should be fired.  At least their management should be.

That is absolutely the wrong thing to do and could lead to a worse outcome since the new team could make even more mistakes. The people who wrote and managed the software will have learned a lot more than if the mission had gone perfectly, and will be stronger for it.

I disagree with you here Steven. Teams don't automatically learn from their mistakes. I've seen that happen numerous times over the past two decades. The Boeing SW team is NOT going to automatically learn from this mishap. Nor will they be automatically stronger because of the mishap.
It will require significant effort and willingness from both the team members and their management to learn the lessons that need to be learned. Only then can the team become stronger.

It seems like this might require an organizational change.  Several years ago we had a big push to outsource SW with little success, the results were poor to say the least.  The SW developers never really seemed to "own" the final product.  The solution was to embed SW teams into the product development groups.  The SW developers became much more familiar with the final product and understood how the SW was to be integrated into the product and how it was to be used.  Also, in an open office environment it was much easier to turn to a product developer and ask for clarification on a spec issue.  It was more expensive compared to outsourcing but the quality improvement was dramatic.

[Eric Hedman]

Much of the Boeing software design staff and software QA team should be fired.  At least their management should be.

That is absolutely the wrong thing to do and could lead to a worse outcome since the new team could make even more mistakes. The people who wrote and managed the software will have learned a lot more than if the mission had gone perfectly, and will be stronger for it.

We still don't know enough details on what went wrong.  In addition to resolving this bug, we don't know if the fundamental design of the software is any good.  If this was a one off problem, replacing the staff is a mistake.  If the software is unsound, wholesale replacement of people is called for.  In my close to forty years of working in software development, I haven't seen incompetent people or teams learn much from mistakes.  They usually go from one struggling project to the next.

[lonestriker]

Much of the Boeing software design staff and software QA team should be fired.  At least their management should be.

That is absolutely the wrong thing to do and could lead to a worse outcome since the new team could make even more mistakes. The people who wrote and managed the software will have learned a lot more than if the mission had gone perfectly, and will be stronger for it.

I disagree with you here Steven. Teams don't automatically learn from their mistakes. I've seen that happen numerous times over the past two decades. The Boeing SW team is NOT going to automatically learn from this mishap. Nor will they be automatically stronger because of the mishap.
It will require significant effort and willingness from both the team members and their management to learn the lessons that need to be learned. Only then can the team become stronger.

I'll have to agree with woods here. Having been a software engineer for more than 25 years (jeez, I've never actually put that down on paper before, so I'm officially older than dirt), from an outside observer, Boeing's development environment, development methodologies, unit/integration testing, code review, and QA certification would have to be abysmally insufficient to allow such a bug to get all the way through to launch.  Whether it's old school waterfall design or more agile continuous integration/continuous deployment, any process with any sort of rigor should catch these types of "bugs" (the quotes are because it's not a bug in software but in their software development practices.)  The issue isn't that a single bug got through.  The issue is that a single bug should never have the ability to cause such a catastrophic failure.  They were lucky that the manifestation of the bug was relatively benign (just wasting propellant) and not something more disastrous with crew on board.

The Boeing software engineers may be incapable of fixing the underlying issues.  That sort of wide-scale change requires a fundamental redesign of their approach to developing software.

[JonathanD]

I think it's premature to turn this into the Spanish Inquisition, at least until the root cause is definitively identified, which in this case *should* be relatively straight-forward given they have the spacecraft and all the logs.  Those working in this program are real people who have proudly dedicated their careers to this work, and I think it is inappropriate to infer they are incompetent or unqualified until we have more information (assuming it is shared).  An investigation should be patient, professional, and thorough, identifying root cause as well as any other problem points and also exonerating those who did their job perfectly and were not involved.

[Coastal Ron]

Sure, this problem happened in a software system, not a hardware one. But that doesn't mean it was bad software. My guess is that the system design is lacking redundancy.

Here we have this big vehicle that is filled with computers, and the only place they can get the mission time information from is the launch vehicle? Really? Who thought that system design up?

It seems like NASA has always been known for having backups on backups, with no single points of failure allowed except in unique circumstances, and they have insisted on the same from their crew providers. Look at how much time and effort they have put into parachutes!

Here we have the failure of a mission due to a (we presume) single wrong clock - WITH NO BACKUP!

That to me is not a software bug, it's a design failure. And while it would be easy to fire the people who did the detailed work, chances are they were following management guidance for the design of the vehicle, so I suggest we all focus our attention on Boeing management, not the workers.

[Mondagun]

Sure, this problem happened in a software system, not a hardware one. But that doesn't mean it was bad software. My guess is that the system design is lacking redundancy.

Here we have this big vehicle that is filled with computers, and the only place they can get the mission time information from is the launch vehicle? Really? Who thought that system design up?

It seems like NASA has always been known for having backups on backups, with no single points of failure allowed except in unique circumstances, and they have insisted on the same from their crew providers. Look at how much time and effort they have put into parachutes!

Here we have the failure of a mission due to a (we presume) single wrong clock - WITH NO BACKUP!

That to me is not a software bug, it's a design failure. And while it would be easy to fire the people who did the detailed work, chances are they were following management guidance for the design of the vehicle, so I suggest we all focus our attention on Boeing management, not the workers.

Exactly this. To quote one of Akin's 'laws' of spacecraft design:
To design a spacecraft right takes an infinite amount of effort. This is why it's a good idea to design them to operate when some things are wrong.

[Ike17055]

it requires no particular level of genius to predict that development might take longer than publicly acknowledged.  Dream Chaser was even further behind conceptually than the others, and would likely have had an even more difficult evolution. They hadn't even picked the engines and fuel yet at the time of contract awards...and the unknowns would likely be plentiful.  THe approach of going with a capsule centered strategy seemed to offer the best chance of a reasonably timely development cycle.  Nobody would be flying yet if we did a downselect to three.

...or maybe SNC wins a contract, but loses a vehicle in an accident and can't fund to replace it.  (They were planning on building only two vehicles IIRC).  Or more likely, they have similar issues and have to deal with them but push through to eventual success.

I know SNC is a favorite sacred cow of some folks around here, and I would have liked to see them have a shot at the brass ring myself, but imaging SNC sails smoothly through all of the obstacles the chosen providers have had to deal with is fanciful in my mind.  It's far more likely they too would have had problems along the way.  I agree that a downselect to three would have been a mistake.

I'd prefer to at this point look forward to their cargo vehicle and see how that goes, and hope the two selected Commercial Crew providers can work through their issues and safely return astros to space using US vehicles.

"Every vehicle will have its development issues" which is why always I believed in spreading the risk especially in the new industry of commercial spaceflight...I stated years ago with the down-select to two that we would end up buying more seats on Soyuz and sometimes I just hate being proven right...

[Ike17055]

achievable through other means. you cannot lose the base experience inputs without new consequences.  THAT is "the real world." what you are advocating is the "Soviet approach".

Much of the Boeing software design staff and software QA team should be fired.  At least their management should be.

That is absolutely the wrong thing to do and could lead to a worse outcome since the new team could make even more mistakes. The people who wrote and managed the software will have learned a lot more than if the mission had gone perfectly, and will be stronger for it.

It doesn't work like that in the real world. There is a saying about the old dog and new tricks and it also applies to us, humans. There is a point where you need to reshuffle the team so questions like " Why " are flying again and critical thinking is employed. The Boeing team SW design and implementation is so poor, that just getting them a new boss will not repair the architecture& implementation errors. Their sufficiency and boredom/lack of involvement lead to this failure. Not management.