Boeing's Starliner (CST-100) - Discussion Thread 6

[leovinus]

Waterfall does not guarantee deficient testing. Neither does it guarantee sufficient testing.
In fact, Boeing still uses waterfall for Starliner development efforts. But they improved their integrated end-to-end test strategy by no longer cutting that test up in temporal blocks. The test now runs the full 3 days.

The reason they initially cut it up in temporal blocks, was to save money. Instead, that decision cost them money: ~ $685M.

This has happened before.  Mars Polar Lander crashed when the leg extension (performed while high up) bumped the "weight on legs" switch.  The software saw this, thought it was on the ground, and cut the engines leading to the crash.

The error was that the software should have ignored the transient case as legs opened (This was a known effect at the system level, but did not get translated into the software specifications).

But why was it not caught during testing of the flight hardware?  There were a number of reasons, but as I understand it, the hardware-in-the-loop simulations were broken into two phases - entry through legs out, and legs out through landing.  The landing simulation started with the legs out, and so never saw the transient.

So, a lesson not learned by Boeing. Cutting up tests in temporal blocks is not necessarily a bad thing. But it can become a bad thing when the split is made at the exact point of a phase transition.

More than two years ago, we had a similar discussion here and to quote from that:

As you know, in 2011, Nasa published the report The Legacy of Space Shuttle Flight Software, NTRS, 20110014946 with experience of 30 years of development, tests, improvements, on the Shuttle's Primary Avionics Software System (PASS) which lead to list of "Lessons learned". The software process was at CMM-5, i.e., the highest and best, most reproducible. We discussed the report and Lessons Learned in more detail in an earlier post.

Therefore, it is simply incomprehensible that one of Nasa's partners and contractors (Boeing) was allowed to build a new human spaceflight system (Starliner) without fully embracing those "Lessons learned" even though it included the end-to-end, hardware/software, system testing recommendation at least twice in Section IX "Lessons Learned", (c)(h), p33-35. In addition, the 2011 Nasa report was around the same time as the start of the Starliner development between IIRC between 2010 and 2012, i.e., hot from the press, and ready to used.

The question on why Boeing did not embrace "Lessons learned" from the best comparable human spaceflight system available (Shuttle, i.e. PASS) will one day make a very good article on this site after some investigative sleuthing and interviewing of the involved people. We can speculate that the list of 61 issues will relate to this question as well.

In the meantime, let's hope Boeing has learned it's lesson and we will get a safe and effective Starliner.

[meekGee]

This thread is going to be a riot

[SoftwareDude]

I think an FOIA request on NASA's 80 recommendations to Boeing is in order.

[Jim]

I think an FOIA request on NASA's 80 recommendations to Boeing is in order.

it will be denied.  Propriety information.

[meekGee]

I think an FOIA request on NASA's 80 recommendations to Boeing is in order.

it will be denied.  Propriety information.

Probably true, but it really doesn't matter anymore.

[SoftwareDude]

Does anyone know where to direct the FOIA for recommendations to Boeing's request? HQ, which is in/near Washington DC, Cape Kennedy, one of the others? It looks like the FOIA might be accepted because all NASA opinions are fair game.

[Jim]

Does anyone know where to direct the FOIA for recommendations to Boeing's request? HQ, which is in/near Washington DC, Cape Kennedy, one of the others? It looks like the FOIA might be accepted because all NASA opinions are fair game.

No, because propriety information is exempt from FOIA.  And NASA recommendations would reveal such information.

[SoftwareDude]

Does anyone know where to direct the FOIA for recommendations to Boeing's request? HQ, which is in/near Washington DC, Cape Kennedy, one of the others? It looks like the FOIA might be accepted because all NASA opinions are fair game.

No, because propriety information is exempt from FOIA.  And NASA recommendations would reveal such information.

It doesn't hurt to try.

[woods170]

Waterfall does not guarantee deficient testing. Neither does it guarantee sufficient testing.
In fact, Boeing still uses waterfall for Starliner development efforts. But they improved their integrated end-to-end test strategy by no longer cutting that test up in temporal blocks. The test now runs the full 3 days.

The reason they initially cut it up in temporal blocks, was to save money. Instead, that decision cost them money: ~ $685M.

This has happened before.  Mars Polar Lander crashed when the leg extension (performed while high up) bumped the "weight on legs" switch.  The software saw this, thought it was on the ground, and cut the engines leading to the crash.

The error was that the software should have ignored the transient case as legs opened (This was a known effect at the system level, but did not get translated into the software specifications).

But why was it not caught during testing of the flight hardware?  There were a number of reasons, but as I understand it, the hardware-in-the-loop simulations were broken into two phases - entry through legs out, and legs out through landing.  The landing simulation started with the legs out, and so never saw the transient.

So, a lesson not learned by Boeing. Cutting up tests in temporal blocks is not necessarily a bad thing. But it can become a bad thing when the split is made at the exact point of a phase transition.

It's important to take the state from the previous block and use it to initialize the next one.

Which often requires adding some additional stuff to the test setup. That adds cost and we all know that Boeing was cutting cost left and right during Starliner development. It caused OFT-1 to go the way it went.

Ultimately, in fixing the problem, Boeing chose to implement the better of the two fixes: to NOT cut up the test in temporal blocks, instead running it in one long go.

[Lee Jay]

Waterfall does not guarantee deficient testing. Neither does it guarantee sufficient testing.
In fact, Boeing still uses waterfall for Starliner development efforts. But they improved their integrated end-to-end test strategy by no longer cutting that test up in temporal blocks. The test now runs the full 3 days.

The reason they initially cut it up in temporal blocks, was to save money. Instead, that decision cost them money: ~ $685M.

This has happened before.  Mars Polar Lander crashed when the leg extension (performed while high up) bumped the "weight on legs" switch.  The software saw this, thought it was on the ground, and cut the engines leading to the crash.

The error was that the software should have ignored the transient case as legs opened (This was a known effect at the system level, but did not get translated into the software specifications).

But why was it not caught during testing of the flight hardware?  There were a number of reasons, but as I understand it, the hardware-in-the-loop simulations were broken into two phases - entry through legs out, and legs out through landing.  The landing simulation started with the legs out, and so never saw the transient.

So, a lesson not learned by Boeing. Cutting up tests in temporal blocks is not necessarily a bad thing. But it can become a bad thing when the split is made at the exact point of a phase transition.

It's important to take the state from the previous block and use it to initialize the next one.

Which often requires adding some additional stuff to the test setup. That adds cost and we all know that Boeing was cutting cost left and right during Starliner development. It caused OFT-1 to go the way it went.

Ultimately, in fixing the problem, Boeing chose to implement the better of the two fixes: to NOT cut up the test in temporal blocks, instead running it in one long go.

You seem to know a lot about this.  What do you think about the approach of using accelerated time to do tasks such as this?

[woods170]

It's important to take the state from the previous block and use it to initialize the next one.

Which often requires adding some additional stuff to the test setup. That adds cost and we all know that Boeing was cutting cost left and right during Starliner development. It caused OFT-1 to go the way it went.

Ultimately, in fixing the problem, Boeing chose to implement the better of the two fixes: to NOT cut up the test in temporal blocks, instead running it in one long go.

You seem to know a lot about this.  What do you think about the approach of using accelerated time to do tasks such as this?

Accelerated time can work just fine for "dead" phases in the timeline. But that also is subject to assumptions of what exactly are "dead" phases.

We've tried it several times in a bunch of end-to-end tests for a process that ran 4 days in a single go. The first 2 times we ran with accelerated time in "dead" phases, we got bitten in the behind real hard. Some of the phases we considered to be "dead", were not actually "dead". Accelerated time caused 15 sequential processes to partly start running parallel, which badly screwed up the net results of our test.
On the third attempt we finally had all (truly) "dead" phases identified correctly. We now run that test (which is 4 days in real time) in 2 days and 17 minutes. Fortunately it is for a process that runs only 4 times each year. Regression testing therefore consumes just a little over 8 days of our precious time annually.

On a personal note:
This coming september I will no longer have to worry about the test I described above. My assignment at my current client will be wrapping up then. My next assignment will reunite me with one of my favourite subjects: ETL testing. And for once the client is not some d*mn financial institution.

[AJW]

So long as we're beating a dead horse, I ran Test Automation for a Silicon Valley company.   Over four decades on the Dev side I had seen first-hand how badly testing was done at organizations like Microsoft and I wanted to approach it and integrate it completely into the development cycle.  The framework I built started at midnight every night by rebuilding the entire test automation code base and then began running tests across multiple platforms, cloud services, mobile devices and custom hardware we developed.   By 9am, my dashboard would report on over 17,000 tests run on development, production and staging services.

I can't imagine not running end-to-end tests.   I get it, simulating months of coast time seems like a waste, but somewhere in an organization there will be systems doing longevity tests, so put them to use.   Of course you can independently speed up sectional tests by pre-setting values, but you can't tell me that somewhere in the years of delays, there wasn't time or systems available to run full-length mission tests.  I had lunch yesterday with a good friend who was brought in to fix the Hubble mirror flaw.   Hubble sat in storage for four years with that flaw.  Another case of years of opportunity, and inadequate testing.

Since this thread is going over lots of previous ground, I don't recall ever hearing what the resolution was for the communications issue that was blamed on cell towers.

[Jim]

  Hubble sat in storage for four years with that flaw.  Another case of years of opportunity, and inadequate testing.

Not possible to test in the spacecraft configuration.  Testing had to be done on the mirror.  Don't even know if testing was feasible in the telescope configuration.

[Eric Hedman]

I can't imagine not running end-to-end tests.   I get it, simulating months of coast time seems like a waste, but somewhere in an organization there will be systems doing longevity tests, so put them to use.   Of course you can independently speed up sectional tests by pre-setting values, but you can't tell me that somewhere in the years of delays, there wasn't time or systems available to run full-length mission tests.  I had lunch yesterday with a good friend who was brought in to fix the Hubble mirror flaw.   Hubble sat in storage for four years with that flaw.  Another case of years of opportunity, and inadequate testing.

The ironic thing is the Hubble backup mirror, which is now in the Smithsonian collection, did not have the manufacturing flaw.

https://airandspace.si.edu/collection-objects/hubble-space-telescope-backup-mirror/nasm_A20010288000

https://www.nytimes.com/1990/07/18/us/hubble-has-backup-mirror-unused.html

[Surfdaddy]

So long as we're beating a dead horse, I ran Test Automation for a Silicon Valley company.   Over four decades on the Dev side I had seen first-hand how badly testing was done at organizations like Microsoft and I wanted to approach it and integrate it completely into the development cycle.  The framework I built started at midnight every night by rebuilding the entire test automation code base and then began running tests across multiple platforms, cloud services, mobile devices and custom hardware we developed.   By 9am, my dashboard would report on over 17,000 tests run on development, production and staging services.

I can't imagine not running end-to-end tests.   I get it, simulating months of coast time seems like a waste, but somewhere in an organization there will be systems doing longevity tests, so put them to use.   Of course you can independently speed up sectional tests by pre-setting values, but you can't tell me that somewhere in the years of delays, there wasn't time or systems available to run full-length mission tests.  I had lunch yesterday with a good friend who was brought in to fix the Hubble mirror flaw.   Hubble sat in storage for four years with that flaw.  Another case of years of opportunity, and inadequate testing.

Since this thread is going over lots of previous ground, I don't recall ever hearing what the resolution was for the communications issue that was blamed on cell towers.

I used to work with a guy who had a favorite saying: "Why is there never enough time to do it right, but enough time to do it over?".

[Steven Pietrobon]

I used to work with a guy who had a favorite saying: "Why is there never enough time to do it right, but enough time to do it over?".

That's an easy one. The first part is because we have a choice and we're too stupid or greedy to make the right one! The second part is because we have no choice. :-)

[edzieba]

  Hubble sat in storage for four years with that flaw.  Another case of years of opportunity, and inadequate testing.

Not possible to test in the spacecraft configuration.  Testing had to be done on the mirror.  Don't even know if testing was feasible in the telescope configuration.

Not for the Perkin-Elmer mirror design, the Kodak/Itek design could be tested when assembled:

Wollensak said that the Perkin-Elmer design did not allow for the primary mirror to be tested after it was placed into the assembled telescope. He said that the instrument was designed to work in zero gravity and that the gravity of Earth caused the glass to sag slightly, which would have changed the focus.

Kodak and Itek, however, said Wollensak, had developed a way to prevent the sag and thus test the mirrors as an assembled unit.

[woods170]

  Hubble sat in storage for four years with that flaw.  Another case of years of opportunity, and inadequate testing.

Not possible to test in the spacecraft configuration.  Testing had to be done on the mirror.  Don't even know if testing was feasible in the telescope configuration.

The additional four years would have made no difference. Two existing refractive null correctors had already demonstrated the fault in the HST primary mirror BEFORE the mirror was built into the optical assembly. However, the new Reflective Null Corrector that Perkin-Elmer had constructed specially for Hubble, showed NO error.

Perkin-Elmer willingly chose to ignore the results from the existing refractive null correctors: they believed that only their new Reflective Null Corrector was telling "the truth".

However, after STS-31 the fault in Hubble's primary mirror was traced all the way back to a design error in Perkin-Elmer's new Reflective Null Corrector.
The results from the older refractive null correctors turned out to have been correct all the time.

Summarizing: the fault in Hubble's primary mirror was known AND confirmed at an early stage. But Perkin-Elmer believed those measurement results themselves to be faulty. Any additional testing in the "storage" years would not have changed that attitude.

But I digress.

[libra]

I can't imagine not running end-to-end tests.   I get it, simulating months of coast time seems like a waste, but somewhere in an organization there will be systems doing longevity tests, so put them to use.   Of course you can independently speed up sectional tests by pre-setting values, but you can't tell me that somewhere in the years of delays, there wasn't time or systems available to run full-length mission tests.  I had lunch yesterday with a good friend who was brought in to fix the Hubble mirror flaw.   Hubble sat in storage for four years with that flaw.  Another case of years of opportunity, and inadequate testing.

The ironic thing is the Hubble backup mirror, which is now in the Smithsonian collection, did not have the manufacturing flaw.

https://airandspace.si.edu/collection-objects/hubble-space-telescope-backup-mirror/nasm_A20010288000

https://www.nytimes.com/1990/07/18/us/hubble-has-backup-mirror-unused.html

The other ironic thing is that the Shuttle and Hubble were expressedly sold and designed, together and hand-in-hand (1972-1987 approximately), for the Shuttle to bring back Hubble to Earth solid ground for refurbishment, updates or... repairs "if something ever went wrong".  NASA had been burned by two failures of OAO early telescopes, in 1966 and 1970 (at launch).

Yet when in spring 1990 did the fecal matter hit the rotating machine in a truly spectacular way - surely the Shuttle was put to good use to repair Hubble (no question about that), except it was found to be easier, and thus happened, IN SPACE.

"Hubble Perkin Elmer mirror is flawed !"
...
"Let's use the Shuttle to bring it back to Earth, since we have the Kodak backup mirror that is not flawed"
...
"Well no, easier to put correcting lenses on Hubble flawed P.E mirror."
...
"Using the Shuttle to do that on the ground ?"
...
"No, in space."

[AJW]

P-E didn't have the experience that Kodak had having built over 70 mirrors including those for the KH-11s, but P-E submitted the lower bid wanting to develop the skillset.   P-E's equipment had a measuring point with a protective cap to prevent it from damage, and the measurements were made to the cap instead of to the measuring point itself leading to a 2mm error. (I don't think this was ever reported)  In the end, Kodak's mirror team were brought in to figure out P-E's problem.  One benefit was that during the delays, there had been so much advancement in digital camera technology, that the replacement optics package was really an enhancement.  At one point P-E must have realized that they were in over their heads since they subcontracted Kodak to build a backup mirror, which didn't have the flaw and is the one now on display.  P-E never ran end-to-end tests or even compared the two mirrors and instead relied on their machine that built the mirror to test the mirror.  So another example of failure due to lack of end-to-end and independent testing and why the engineers doing the testing should be as qualified and empowered as those doing the development.