-
#980
by
erioladastra
on 08 Feb, 2020 15:00
-
Earlier NASA said that clearly their oversight of Boeing had been insufficient and that they intended to address that going forward.
Ouch. IMO Kathy Lueders will have some explaining to do. Not good for her and the CCP office.
Possibly but it may be wider than that - is the issue that NASA (wrongly) assumed that didn’t need that level of oversight and so didn’t provide the resources needed? Kathy did say that the things they did look at were fine, while acknowledging that they could only look at so much.
It shouldn't have taken much in the way of NASA resources to catch the fact that Boeing's test plan would miss such blatant and easily-found bugs. A single days of meetings by one good software person should have flagged the problem.
Also, I'd fault every level of Boeing's management for this.
is it possible that the Boeing plan met industry and NASA standards???
-
#981
by
erioladastra
on 08 Feb, 2020 15:02
-
Is there a standard differentiation between "lines of code" and "scripts"?
If Starliner "operated ~66% of the scripts correctly" then it operated about a third of them incorrectly!
How is a third of their "scripts" being operated incorrectly not a horrible thing?
Is it OK if 90% of their scripts are operated correctly?
I doubt it.
Scripts are presumably higher-level exercises of sequences of functions e.g. a script that would point Starliner from heading A to heading B by executing a series of thruster firings. (I made that up).
They're just saying that the OFT exercised 66% of their scripts and 33% weren't exercised, possibly due to abbreviated mission duration or because they're contingency based (wouldn't execute on a nominal mission).
Correct. Or that some were done manually from the ground.
-
#982
by
erioladastra
on 08 Feb, 2020 15:06
-
Where was Chris Ferguson in this? Wouldn't he have been in the simulators going through each mission procedure, event & working through potential anomaly recovery checklists? I mean orbital insertion, coms & SM jettison seem like fairly important mission milestones.
I suspect he was but he would have been working with the CFT software, a simulated coms system and doing a SM jettison using a model that might have the same error in it or not have the fidelity needed.
-
#983
by
erioladastra
on 08 Feb, 2020 15:13
-
Starliner used its CM thrusters during pad abort test, why didn't mapping error show up then?
The flaw was in the orbital disposal burn of the SM system, nothing to do with CM RCS. Therefore it was never even in play to detect during PAT.
Both test fights used service module thrusters continuously, why they would have different mapping for disposal?
I don't think it was a mis-mapping on the vehicle but in the sequence of which jets to fire when and for how long.
-
#984
by
kevinof
on 08 Feb, 2020 15:18
-
According to Garrett Reisman, the SpaceX nickname for the CST-100 is the "POS 100"? He was on the Joe Rogan channel doing an interview.
So I suspect that Nasa will delay BOTH Startliner and Spacex while they go through a complete review of both. That way they claim they are even handed, appease their masters and have both available at the same time.
-
#985
by
SoftwareDude
on 08 Feb, 2020 15:29
-
Given the nature of the second major software issue it clearly points to a systemic issue.
Good to hear that NASA is now looking into how Boeing does its software development and verification process.
Absolutely.
But why didn't NASA notice that their testing procedures wouldn't have caught this? With all the in-depth oversight NASA has had with both commercial crew providers, why did nobody at NASA ask if they were doing an all-up full-mission simulation? That should have caught all of these bugs.
Note that OFT was your full-up mission simulation. Even if you want to throw infinite time or money at something like this you can neve truly do a full-up sim on the ground from end-to-end. Somewhere along the line you have to make mods (e.g., well we can't blow the pyros because...well that should be obvious) or you don't have a real-size ISS in front of you. However, you should do as much as you can and as high a fidelity as you can do to mitigate as much risk as possible. And to be clear, I am NOT trying to defend Boeing at all, just noting in general. Clearly they skimped on the full-up integrated testing (which is kind of ironic - commercial space was to be leaner, cut down on overheard, take some risks and the company that may have done that to a fault was the big behemoth classic company, though from what I hear there is still a lot of concerns with their competitor).
If you watch the SpaceX factory tour done several years ago, there is an interview with a software guy running a fullup simulation.
-
#986
by
ZachS09
on 08 Feb, 2020 15:31
-
According to Garrett Reisman, the SpaceX nickname for the CST-100 is the "POS 100"? He was on the Joe Rogan channel doing an interview.
So I suspect that Nasa will delay BOTH Startliner and Spacex while they go through a complete review of both. That way they claim they are even handed, appease their masters and have both available at the same time.
Why delay SpaceX if their recent tests were successful (DM-1 and IFA)? I’m not counting the first static fire that failed. SpaceX didn’t have any inflight problems FAIK.
-
#987
by
kevinof
on 08 Feb, 2020 15:33
-
One word - Politics.
According to Garrett Reisman, the SpaceX nickname for the CST-100 is the "POS 100"? He was on the Joe Rogan channel doing an interview.
So I suspect that Nasa will delay BOTH Startliner and Spacex while they go through a complete review of both. That way they claim they are even handed, appease their masters and have both available at the same time.
Why delay SpaceX if their recent tests were successful (DM-1 and IFA)? I’m not counting the first static fire that failed. SpaceX didn’t have any inflight problems FAIK.
-
#988
by
DigitalMan
on 08 Feb, 2020 15:34
-
One word - Politics.
According to Garrett Reisman, the SpaceX nickname for the CST-100 is the "POS 100"? He was on the Joe Rogan channel doing an interview.
So I suspect that Nasa will delay BOTH Startliner and Spacex while they go through a complete review of both. That way they claim they are even handed, appease their masters and have both available at the same time.
Why delay SpaceX if their recent tests were successful (DM-1 and IFA)? I’m not counting the first static fire that failed. SpaceX didn’t have any inflight problems FAIK.
Thankfully, NASA pointed out that SpaceX already had their review.
-
#989
by
demorcef
on 08 Feb, 2020 15:46
-
Boeing has a real problem with software and holding people's lives in their hands. I am so glad that they got called out on the carpet for this one. Absolutely disgusting how Boeing's legendary engineering skills are being completely undermined by a toxic corporate management culture. The 737max debacle was caused by the same damn disease of shareholder capitalism where the most important thing is always the share price. Boeing's stock can't go down fast or far enough for me. SpaceX isn't perfect but damn if I would trust Boeing anymore to do anything right.
-
#990
by
Gary
on 08 Feb, 2020 15:51
-
-
#991
by
rcoppola
on 08 Feb, 2020 15:56
-
Arrogance, hubris, incompetence, mismanagement.
Failure is often a lack of imagination. No, not to succeed but to fail. To imagine all the ways you're creation can fail is as important as all the ways your creation should and could succeed. But there is an honesty, a truth, a humility that needs to be pervasive for a management culture to value that.
Remember what Jim B said when he was in front of SpaceX and discussing the Abort Fuel System failure. He said he was confident that they'd solve for it because Elon, many months before it happened actually mentioned the complexity of that system as a potential area for concern. Real intelligence isn't being right but knowing you're wrong.
Boeing management seems not to posses the humility and competence needed to properly imagine, embrace and plan for failure. They fear it. Deny it. Obfuscate when it intrudes on their self constructed and perpetuated reality. It is my very strong belief, nay, desire, to see them redo OFT. If for no other reason than to be forced to account for their arrogance and ineptitude in the only way they seem to understand....$$.
-
#992
by
mulp
on 08 Feb, 2020 16:03
-
The problem with two parts colliding just after separation was Elon's 3rd? big error back with Falcon. My guess he asked during reviews "they aren't going to crash together after separation, right? You triple checked?"
Seriously, I don't follow Boeing, ULA, et al like I do SpaceX, so is Boeing running through anything close to launching on a regular basis like SpaceX?
Does Boeing have anything close to the experience of launching cargo to ISS every several months?
SpaceX built on its cargo launch process and while there were issues noted, the launch, docking, etc, landing went to plan.
For Boeing, it launched, and nothing went to plan.
I question the change of plan to extend the first manned flight for either, but SpaceX has done something similar for cargo, and once for Dragon 2.
But has Boeing done anything approaching that just once without crew?
This seems like carrying paying customers on the next flight after the first test plane crashed on its maiden flight.
As a test engineer, I have recommended more testing/verification on multiple big releases of computer systems, and was happy not be the one making costly decisions - no lives were at stake. I definitely do not want to be the Boeing manager on this. I think he should cost Boeing shareholders $400 million.
-
#993
by
abaddon
on 08 Feb, 2020 16:07
-
Starliner used its CM thrusters during pad abort test, why didn't mapping error show up then?
It's a good question. My guess is the abort thrusters are a very specific cluster that are only used in an abort scenario and are not part of the "mapping" that is used to enact more regular and complex/nuanced operations. Just a guess though.
-
#994
by
abaddon
on 08 Feb, 2020 16:15
-
According to Garrett Reisman, the SpaceX nickname for the CST-100 is the "POS 100"? He was on the Joe Rogan channel doing an interview.
That's disappointing if true, that's unprofessional (and not very creative).
-
#995
by
rockets4life97
on 08 Feb, 2020 16:18
-
Since SpaceX got brought into the thread, I'll remind everyone that Elon has previously talked about how Tesla isn't a car company, but a software company. I expect the same applies to SpaceX. All of which to say, I expect SpaceX's software practices to be industry standard for the software industry.
-
#996
by
Gary
on 08 Feb, 2020 16:36
-
For Boeing, it launched, and nothing went to plan.
Boeing didn't launch, they hitched a ride on a ULA Altlas V so you could say that ULA delivered on their part of the deal but Boeing blew it.
-
#997
by
groknull
on 08 Feb, 2020 16:52
-
Starliner used its CM thrusters during pad abort test, why didn't mapping error show up then?
It's a good question. My guess is the abort thrusters are a very specific cluster that are only used in an abort scenario and are not part of the "mapping" that is used to enact more regular and complex/nuanced operations. Just a guess though.
When two spacecraft (e.g. Command Module and Service Module) are together as one unit, each thruster has a specific orientation relative to the center of mass of the combined unit. There will be a map from desired rotations and translations to individual thruster firings.
When the two modules separate, each becomes its own spacecraft, with different vehicle mass, center of mass location, and rotational moments. Each thruster now has a different orientation relative to the new center of mass, so the mappings from desired rotations and translations to thruster firing will be different. Thruster firing durations will also be shorter, due to reduced mass and moments.
If the SM thruster map doesn't get changed to independent spacecraft mode after separation, thruster firings will be longer than necessary, resulting in more aggressive/abrupt/larger rotations and translations than desired.
Additionally, control inputs (mapped to thruster firings) won't get you to where you want to be. Translation requests can invoke rotations, and simple, around the CoM rotations can include significant translations. (Ask any 3D graphics person who has screwed up homogeneous coordinates.)
Edit:
To address the pad abort question...
Service Module thruster firings may not have been tested post-separation. They may simply have been shut off entirely post-separation because SM trajectory wasn't considered important. (That is a guess.) The important thing on pad abort is to safely return the CM occupants to the ground. Remapping the CM thrusters correctly post-separation would be important to control the attitude of the CM for drogue chute deployment.
On the other hand, if post-separation orientation is all aerodynamic, then thruster remapping would not have been tested at all during the pad abort test.
-
#998
by
freddo411
on 08 Feb, 2020 17:03
-
Every night, deploy all software to a simulated test-bed and undergo automated full mission tests.
That would not buy you a whole lot. Models, are notoriously inaccurate. First they are usually written by the people who wrote the code so they tend to prove a tautology (code is supposed to do this, model fed code to that). yes, you can have independent people develop the models. Better, not perfect. Second, you run a sim and you hit a failure. Was it a flight software code issue or a model/sim problem - you now have to devote resources to running that down. Not saying you can't or shouldn't do this, but there is some breakover where you are spending more for diminishing returns. it is part of the process that should have occurred.
Your points are well taken, however, while automated testing definitely requires an investment of time and effort, its value is enormous.
A couple of the benefits:
* Discovers regression bugs
* Exposes errors earlier
* Teaches the programmers a lot about the system and the software
* Exposes not only bugs, but also performance errors, configurations difficulties, and potentially strange edge cases
* Engages programmers and PMs into the process of OPERATING the system, giving them a real stake in quality
-
#999
by
freddo411
on 08 Feb, 2020 17:06
-
Starliner used its CM thrusters during pad abort test, why didn't mapping error show up then?
The flaw was in the orbital disposal burn of the SM system, nothing to do with CM RCS. Therefore it was never even in play to detect during PAT.
Both test fights used service module thrusters continuously, why they would have different mapping for disposal?
After separation, the SM has a very different center of mass, so firing thruster $X give you a different torque.