-
#960
by
GWH
on 08 Feb, 2020 05:59
-
Should they refly?
Not the right question. Identify and remedy the testing flaws, THEN figure out if there is still enough risk to justify another OFT.
A little risk is ok (I am a SpaceX fan after all).
The bigger problem here is arrogance. Boeing has way too much faith in their process, and assumed they can go straight from abort to OFT to crew. All assuming things will be perfect. They have bragged about how close the crewed test flight will fly to the OFT mission. While there have been other factors affecting delays, it seems like SpaceX took a path that had them testing more sooner. To me that is the attitude of a company that expects problems and looks to test to find them. (Disclaimer: the exploded Dragon still was not a "good test").
Boeing seems to be carrying on with this attitude that the sheer magnificence of their teams will ace every test, so they put it off until they think everything perfect and low and behold they find all sorts of problems.
I think a detailed review will find if that is truly the case and how much they have let arrogance cloud their test regimes vs working to break things to find the flaws.
-
#961
by
Rebel44
on 08 Feb, 2020 08:01
-
Should they refly?
Not the right question. Identify and remedy the testing flaws, THEN figure out if there is still enough risk to justify another OFT.
A little risk is ok (I am a SpaceX fan after all).
The bigger problem here is arrogance. Boeing has way too much faith in their process, and assumed they can go straight from abort to OFT to crew. All assuming things will be perfect. They have bragged about how close the crewed test flight will fly to the OFT mission. While there have been other factors affecting delays, it seems like SpaceX took a path that had them testing more sooner. To me that is the attitude of a company that expects problems and looks to test to find them. (Disclaimer: the exploded Dragon still was not a "good test").
Boeing seems to be carrying on with this attitude that the sheer magnificence of their teams will ace every test, so they put it off until they think everything perfect and low and behold they find all sorts of problems.
I think a detailed review will find if that is truly the case and how much they have let arrogance cloud their test regimes vs working to break things to find the flaws.
IMO, given the extent of problems during the OTF, another uncrewed test flight to ISS is necessary in order to regain sufficient confidence to fly with a crew.
Even in industries that are far less demanding, it is normal that if the product badly fails a test, manufacturer has to redo the test and pass before that product can be declared fit for purpose and handed over to customers.
-
#962
by
FutureSpaceTourist
on 08 Feb, 2020 08:37
-
Reflecting on this, the key and shocking point to me is just how basic the critical failures during OFT were. They were not subtle, complex, or unexpected conditions but fundamental normal parts of any flight. Of course such failures could only get to flight due to systemic failures throughout Boeing’s processes (and, arguably, NASA’s too).
I’m pleased that they are at least now taking the correct actions, seeking to understand the process and cultural failures, fixing them and fully re-verifying the software.
An OFT2 cannot prove that they have fully fixed things. One flight cannot possibly exercise the full flight envelope, the abnormal / error conditions that the software has to cope with but which you don’t expect to see on any given flight (such as various hardware failures, more extreme environmental conditions, non-fatal launch vehicle issues etc etc).
But would I have an OFT2 flight? Yes, I would. The failures (both wider process and specific software issues) have been so bad that a clean flight is needed to rebuild some confidence and trust. With process failures this bad we have to assume that more bugs will be found that are currently unknown. Fixing bugs is not without risk of introducing unintended consequences.
So my opinion is that now is not the time to pass on a significant opportunity (OFT2) to learn more, on the assumption that because of the bad failures they’ll make sure it’s right the second time.
-
#963
by
cebri
on 08 Feb, 2020 10:29
-
Reflecting on this, the key and shocking point to me is just how basic the critical failures during OFT were. They were not subtle, complex, or unexpected conditions but fundamental normal parts of any flight. Of course such failures could only get to flight due to systemic failures throughout Boeing’s processes (and, arguably, NASA’s too).
I’m pleased that they are at least now taking the correct actions, seeking to understand the process and cultural failures, fixing them and fully re-verifying the software.
An OFT2 cannot prove that they have fully fixed things. One flight cannot possibly exercise the full flight envelope, the abnormal / error conditions that the software has to cope with but which you don’t expect to see on any given flight (such as various hardware failures, more extreme environmental conditions, non-fatal launch vehicle issues etc etc).
But would I have an OFT2 flight? Yes, I would. The failures (both wider process and specific software issues) have been so bad that a clean flight is needed to rebuild some confidence and trust. With process failures this bad we have to assume that more bugs will be found that are currently unknown. Fixing bugs is not without risk of introducing unintended consequences.
So my opinion is that now is not the time to pass on a significant opportunity (OFT2) to learn more, on the assumption that because of the bad failures they’ll make sure it’s right the second time.
+1000. I get you may have some issues with interdependent software that it may manifest if a series of unlikely conditions are met. But to have the wrong firing map coded and not noticing it, IMO reveals a very flawed process there.
-
#964
by
HVM
on 08 Feb, 2020 10:30
-
Starliner used its CM thrusters during pad abort test, why didn't mapping error show up then?
-
#965
by
cebri
on 08 Feb, 2020 10:39
-
Starliner used its CM thrusters during pad abort test, why didn't mapping error show up then?
The failure is very strange. It should have shown up in the simulations as well. Maybe one of the flawed processes is change/version control. When you have different development branches you need to identify how different software elements interact between each other to make sure one change wont affect the rest of the software in an unwanted way. Also may (pure speculation) reveal a very flawed testing process if the latest software wasn't tested for all flight regimes before being deployed.
-
#966
by
Vettedrmr
on 08 Feb, 2020 11:36
-
I've been mulling on this now, and thought of something that I really haven't seen much of in these threads:
Thank God for ASAP. As much as we gritch about them picking over past things with SpaceX that we (the public) believe are long solved, they put this info out to the public that NASA and Boeing haven't. And, to be fair, both NASA and SpaceX were very reticent to release info after DM-1's pad explosion. But ASAP released a bombshell, and has put both Boeing and NASA in the position of having to deal with these issues.
I was with Steven P. for the longest time after OFT, that the 2nd flight could be crewed with additional test points for what didn't get tested during OFT. Now, I think a reflight is necessary. In fact, ISS management is probably going to be VERY cautious about letting Starliner anywhere near ISS, until such time as the ability to approach and retreat from ISS has been demonstrated. What would happen *if* Starliner is making its initial approach, is commanded to stop, and instead fires thrusters the wrong way? Say it can't happen? Based on OFT's issues, how can anyone say that?
You can say it by going through the code, with NASA heavily involved in the review process, by coming up with a flight test profile that demonstrates everything Starliner has to do while a safe distance away from ISS. Then you'll have the confidence to allow approach and docking.
And I'd do it unmanned. Yes, if OFT was manned the crew might have saved the docking portion of the mission. But there's a good chance the thruster mapping issue wouldn't have been found, with potential catastrophic results. I think OFT-2 is needed from a crew risk POV.
It sucks. I want both Starliner and Crew Dragon servicing the ISS. At least now we have one supplier that's about ready.
Have a good one,
Mike
-
#967
by
Gary
on 08 Feb, 2020 11:53
-
I've just read the news article on the additional anomalies and while I'm not surprised by the anomalies themselves, after all, test flights are there to find these issues, I'm more appalled by the cavalier attitude of some of the staff.
to say " the second issue “wasn’t an anomaly because we found it and fixed it” totally misses the point, it wasn't supposed to happen, that's an anomaly. Sure, you found and fixed it but you only found it because the anomaly happened.
This comes across to me in the same manner as "Foam hitting the vehicle isn't an anomaly because we check for it" or "MCAS crashing the planes into the ground isn't an anomaly because we designed it that way".
Yeah, I'm being a bit cruel to Boeing here but come on, be open with the issues.
I voted yes to another test flight, not because I think that the craft specifically needs one but because Boeing needs practice in being more honest and not trying to weasel out of issues because that never ends well.
-
#968
by
DistantTemple
on 08 Feb, 2020 13:23
-
For an astute, in depth, critical analysis of Starliner's performance, a party is discussing it on this thread:
https://forum.nasaspaceflight.com/index.php?topic=42585.msg2043698#msg2043698This has ideas that relate to :
In fact, ISS management is probably going to be VERY cautious about letting Starliner anywhere near ISS, until such time as the ability to approach and retreat from ISS has been demonstrated. What would happen *if* Starliner is making its initial approach, is commanded to stop, and instead fires thrusters the wrong way? Say it can't happen? Based on OFT's issues, how can anyone say that?
Risk to the ISS is why Dragon2 had such stringent protocols.... and is the reason for keep-out boundaries etc... It is stunning to even imagine a spacecraft having permission to get close to the ISS when there is the slightest question of risky software, or thrusters incorrectly firing!
It (might seem to be) a surprising situation, where SpaceX is now the more reliable, and safer provider, including better quality control, than Boeing!
One outcome could be that SX is asked to bring forward or provide additional Dragon2 missions.
Another is that Boeing could actually bring in SX software development management to help them with oversight and quality control, particularly, how to run that department in a way to give good access and oversight to NASA!!! SpaceX could make their experience at this a saleable asset!!!!
Edit: added quote and comment on ISS
-
#969
by
Gary
on 08 Feb, 2020 13:34
-
Another is that Boeing could actually bring in SX software development management to help them with oversight and quality control, particularly, how to run that department in a way to give good access and oversight to NASA!!! SpaceX could make their experience at this a saleable asset!!!!
Edit: added quote and comment on ISS
What happened to the team who did the QA checking on Shuttle software? I know that in Feynmans addendum to the Rogers commission challenger report he singled out that team for doing very high quality work.
-
#970
by
woods170
on 08 Feb, 2020 13:35
-
“Mr. Loverro continued, saying Boeing and NASA did not have to disclose the second issue to the media or the U.S. taxpayers because “We fixed it. You wouldn’t want us talking about something that didn’t happen.”
Besides the outright absurdity of this, why is the NASA guy saying “we fixed it”?
Absurd it is.
Loverro stated that Boeing and NASA didn't have to disclose the second issue because "they fixed it".
It is exactly this horrific attitude towards safety AND transparancy that was chastised by both the Rogers Commission and CAIB.
What we have here is a NASA official that has clearly NOT learned the deeper lessons behind both the Challenger and Columbia tragedies.
-
#971
by
woods170
on 08 Feb, 2020 13:49
-
We are all talking about NASA making Boeing redo the test and whatnot....but with everything that has come to light today....I think the ISS partners may have some say so too at this point.
Don't the ISS partners have to sign off on anything even getting close to the ISS?
There is information about this in L2:
https://forum.nasaspaceflight.com/index.php?topic=29664.msg2041859#msg2041859
-
#972
by
erioladastra
on 08 Feb, 2020 13:55
-
I once visited a place that wrote the code that runs inside jet engine controllers for commercial airliners. IIRC, they told me that their staff of 750 could write around 4,000 lines of code a week, including all the necessary verification and testing. Commercial code is written faster than that by a factor of 1,000 or so. Human-safety-critical code requires a heck of a lot of scrutiny and testing.
Assuming that same staff could review/re-test 8000 lines a week, that's 125 weeks. I still think about a year, but it's for sure not a 2-3 month task for a team of 10-20.
Code review and typical code testing, while important, are not the final/best solution. Typically software code testing is module A was to set X to 1 when Y said Z. Does why say Z? So X=1? Yes, done. Peer/code review will probably give you the same answer. What you really need to do is integrated system testing in as flight like environment as you can possibly get to see how it works. This is what the flight control team does and in the case of OFT clearly it saved Boeing's bacon. Twice.
-
#973
by
TorenAltair
on 08 Feb, 2020 14:00
-
I don‘t expect another OFT. I assume a go-ahead („all bugs fixed“) from NASA in April for a CFT launch on May 20.
-
#974
by
SoftwareDude
on 08 Feb, 2020 14:06
-
“Mr. Loverro continued, saying Boeing and NASA did not have to disclose the second issue to the media or the U.S. taxpayers because “We fixed it. You wouldn’t want us talking about something that didn’t happen.”
Besides the outright absurdity of this, why is the NASA guy saying “we fixed it”?
Absurd it is.
Loverro stated that Boeing and NASA didn't have to disclose the second issue because "they fixed it".
It is exactly this horrific attitude towards safety AND transparancy that was chastised by both the Rogers Commission and CAIB.
What we have here is a NASA official that has clearly NOT learned the deeper lessons behind both the Challenger and Columbia tragedies.
The real problem is that NASA knows Boeing's lobbyists are responsible for so much of NASA's funding. Therefore, NASA has to keep Boeing in the running at all costs.
-
#975
by
erioladastra
on 08 Feb, 2020 14:44
-
Starliner used its CM thrusters during pad abort test, why didn't mapping error show up then?
The flaw was in the orbital disposal burn of the
SM system, nothing to do with CM RCS. Therefore it was never even in play to detect during PAT.
-
#976
by
erioladastra
on 08 Feb, 2020 14:48
-
Also: Did i hear it correct that Starliner was effectively jammed by CELLPHONE TOWERS?
I heard it was causing noise from the ground, other's please confirm...
That is essentially what I heard.
Does anyone on NSF has any insight into other (any other) instance of TDRS interference or known failure modes?
The question about using/not using S band was not answered in the presser. There's a known issue with TRDS and s-band over the Indian ocean (at least that's what I can tell by googling).
The issue was not at TDRS. It has been a growing problem for other vehicles as well.
-
#977
by
erioladastra
on 08 Feb, 2020 14:51
-
Every night, deploy all software to a simulated test-bed and undergo automated full mission tests.
That would not buy you a whole lot. Models, are notoriously inaccurate. First they are usually written by the people who wrote the code so they tend to prove a tautology (code is supposed to do this, model fed code to that). yes, you can have independent people develop the models. Better, not perfect. Second, you run a sim and you hit a failure. Was it a flight software code issue or a model/sim problem - you now have to devote resources to running that down. Not saying you can't or shouldn't do this, but there is some breakover where you are spending more for diminishing returns. it is part of the process that should have occurred.
-
#978
by
HVM
on 08 Feb, 2020 14:55
-
Starliner used its CM thrusters during pad abort test, why didn't mapping error show up then?
The flaw was in the orbital disposal burn of the SM system, nothing to do with CM RCS. Therefore it was never even in play to detect during PAT.
Both test fights used service module thrusters continuously, why they would have different mapping for disposal?
-
#979
by
erioladastra
on 08 Feb, 2020 14:59
-
Given the nature of the second major software issue it clearly points to a systemic issue.
Good to hear that NASA is now looking into how Boeing does its software development and verification process.
Absolutely.
But why didn't NASA notice that their testing procedures wouldn't have caught this? With all the in-depth oversight NASA has had with both commercial crew providers, why did nobody at NASA ask if they were doing an all-up full-mission simulation? That should have caught all of these bugs.
Note that OFT was your full-up mission simulation. Even if you want to throw infinite time or money at something like this you can neve truly do a full-up sim on the ground from end-to-end. Somewhere along the line you have to make mods (e.g., well we can't blow the pyros because...well that should be obvious) or you don't have a real-size ISS in front of you. However, you should do as much as you can and as high a fidelity as you can do to mitigate as much risk as possible. And to be clear, I am NOT trying to defend Boeing at all, just noting in general. Clearly they skimped on the full-up integrated testing (which is kind of ironic - commercial space was to be leaner, cut down on overheard, take some risks and the company that may have done that to a fault was the big behemoth classic company, though from what I hear there is still a lot of concerns with their competitor).