Boeing and SpaceX have signed separate CCtCap agreements.
Are you sure? I was under the impression that all 3 parties signed one contract.
Boeing and SpaceX have signed separate CCtCap agreements.
Are you sure? I was under the impression that all 3 parties signed one contract.
They have separate contracts.
My concern is this:
On the Pad Abort, Boeing’s procedures let a human error go through that should have been caught.
Now, on the OFT, It appears the whole problem was triggered by Starliner grabbing the wrong MET and running the mission on that time. Once again, the question comes up of how did it get past them. Somebody should have asked the question of what happens if MET is wrong and built in procedures to deal with that.
This leaves two possibilities: either they did build in failure tolerance and it failed, or they did not build in failure tolerance.
At this point, I’d actually be less concerned with a multiple systems failure where both the primary and backup systems failed than the alternative.
If they didn’t have backup systems then that’s two failures in two successive tests because something that should have been caught wasn’t because nobody was checking.
That’s a huge problem when dealing with a NASA’s supposed culture of safety.
There is precedence for modifying contracts to adjust to changing circumstances. I seem to remember that SpaceX was initially contracted for 3 dragon demo flights but the last 2 were combined into a single flight.
Let me check...
Initially, the objectives of the C2+ mission were to have been accomplished by two separate missions; Dragon C2 would have carried out a fly-by of the ISS, practiced rendezvous maneuvers and communications with the station, before returning to Earth. A second mission, C3, would have been the first mission to berth with the station. In July 2011, NASA gave tentative approval to combine the objectives of the two missions. In December 2011, NASA formally approved the merger of the COTS 2 and 3 missions into the Dragon C2+ flight. There were several launch delays, the last one occurring on 19 May 2012, due to a launch abort during the last second before liftoff.
https://en.wikipedia.org/wiki/Dragon_C2%2BYep...
There is precedence for modifying contracts to adjust to changing circumstances.
Of course contracts can be modified. No one is arguing that they can't.
The point everyone is making about looking at the contract is that the current contract REQUIRES that Boeing do an un-crewed ISS docking. And Boeing agreed to that - and is no doubt being paid for that too.
Both NASA and Boeing have reasons to change the un-crewed ISS docking requirement:
- For Boeing, for business reasons, they don't want to incur the additional costs that come with re-flying this mission.
- For NASA, they want a redundant crew transportation capability as quickly as possible.
So there are mutually beneficial reasons for modifying the existing contract, both related to cost and time. unfortunately there is no way to know if "quality" will be sacrificed...
Did a little digging in the Atlas V User's Guide for some info on software interfaces between LV and spacecraft
https://www.ulalaunch.com/docs/default-source/rockets/atlasvusersguide2010.pdfand found that, as of 2010 anyway, there really aren't any. But I did find an interesting quote on Spaceflight Now from last month, regarding the unique aspects of the Atlas V used for Starliner:
“Electrically, one of the unique things about this mission is that the launch vehicle and spacecraft are going to be talking to each other,” Weiss said in a recent interview. “We normally don’t have that. They will be sharing data throughout (the) flight.”
The ULA launch team probably stay's busy with Atlas-V launches.
Is the Boeing launch team new? It's been ten years since the shuttle launched.
Unfortunately software errors like this are not unheard of. The control software developed by Northrup for the guided missile destroyer USS McCain left the ship in situations were the ship was uncontrollable costing 10 lives when it crashed in the Singapore Straits in 2017.
https://features.propublica.org/navy-uss-mccain-crash/navy-installed-touch-screen-steering-ten-sailors-paid-with-their-lives/From the article:
'The NTSB put it plainly: “The design of the John S McCain’s touch-screen steering and thrust control system,” the board found, “increased the likelihood of the operator errors that led to the collision.”
The Navy investigators, for their part, determined that the system’s “known vulnerabilities” and risks had not been “clearly communicated to the operators on ships with these systems.”'Fortunately this error with Starliner was found with no one onboard. Hopefully this problem can be fixed quicker and better than the problem on the USS McCain. The real world always exposes in my experience problems that were not apparent in using simulation no matter how much you do. Management doesn't have an easy task in determining if testing software used in critical situations is sufficiently rigorous and complete.
I'm wondering if a decent methodology would be to develop a simulation of each hardware component right along with the software designed to control it. So you'd be unit testing and integrating your sim in parallel with unit testing and integrating the actual iron.
The buzzword du jour for what you are referencing is "digital twin." However, the emulators that I've dealt with don't allow mocks of direct address reading, only bus and protocol mocks.
I have worked on ASIL D/ISO 26262 software in automotive applications. The main testing environment consisted of software-in-the-loop (SIL) and hardware-in-the-loop (HIL) tools. SIL runs the program against a software simulation either from a quick build on your own computer or on a real controller on your desk. HIL offers much more fidelity and essentially consists of all the hardware in car laid out across a room with all the real controllers running real software. You'd plug the controller you were testing in at the appropriate place and could then see the real hardware as the program runs, like watching the throttle move when running a WLTP cycle. In terms of simulation quality, you get real physical simulation of everything down to stuff like valve open/close latency. In terms of test frequency, I would sometimes spend multiple hours per day in a HIL and test small changes. It's a perfectly normal debugging tool that you are expected to use liberally. Ironically enough, one of my first tasks in the simulator was to track down a timing issue that could prevent the controller from taking the proper actions.
The teams also work slightly differently form other software projects. There is a single, tight group of about six people responsible for the whole software. They know everything that's going on and define the architecture in terms of signals (pretty much global variables) and modules (compact pieces of code that act upon and modify those variables). Although it all compiles down to C at some point, most of the development work is actually done in a graphical environment that handles metadata like units on the signals. I also wrote static analysis tools that could visualize the data flow and autonomously reposition modules while keeping all constraints in terms of read/write sequencing and access to operating system function provably equal. Every change is requested and orchestrated by the central team. In the case of small changes, they might change the code themselves, anything past a few lines is specified and given to a programmer. A copy of the high level specifications is given to a second team in a separate building that develops a corresponding "level 2" component. This component is run after all the modules, but before any actions are committed, and performs all sorts of bounds and sanity checking. In the case of automotive controllers, this is the only code required to run on a replicated processor core as that should be enough to cover most hardware failures.
While this methodology is somewhat antiquated by todays standards, it results in plenty safe software. I can't imagine how the MET error could have gotten past a simple test even with a simulated rocket that just replays a launch recording. All I can come up with is that they tried to emulate the rocket somehow which lead to them testing against their own, wrong, preconception about how the signaling should function.
Emphasis mine.
In other words: under your scenario Boeing tested not against the real-deal Atlas V EDS but against a stub.
Let's just hope that is NOT what actually went wrong because that would make the MET issue even more ridiculous than it already is.
There is precedence for modifying contracts to adjust to changing circumstances.
Of course contracts can be modified. No one is arguing that they can't.
The point everyone is making about looking at the contract is that the current contract REQUIRES that Boeing do an un-crewed ISS docking. And Boeing agreed to that - and is no doubt being paid for that too.
Both NASA and Boeing have reasons to change the un-crewed ISS docking requirement:
- For Boeing, for business reasons, they don't want to incur the additional costs that come with re-flying this mission.
- For NASA, they want a redundant crew transportation capability as quickly as possible.
So there are mutually beneficial reasons for modifying the existing contract, both related to cost and time. unfortunately there is no way to know if "quality" will be sacrificed...
I've been waiting for someone to touch on this because it gives me a chance to share some of the info I got from various sources over the last couple of days. Prime thing being that:
- There is NOT going to be a re-do of OFT. Every indicator I have, from both NASA and industry sources, says that NASA is going to sign off on the OFT milestone, despite the fact that it didn't dock with ISS. One source mentioned that it would have required an IFA hotfire-type mishap (in other words: a catastrophic failure) for NASA to demand a re-do of OFT.
- Legal agreements between Boeing and NASA are going to be altered to reflect this. In other words: requirements regarding ISS approach, emergency retreat, re-approach, docking, docked operations and undocking are going to be removed from the OFT contractual requirments, by MUTUAL consent. All those elements are going to be deferred to CFT.
- CFT is going to be delayed, but not by much (at least, that is the expectation). Bridenstine announced that NASA is going to have an additional good hard look into how things are run in the Boeing Starliner program. My sources confirmed that this is indeed going to happen but they also indicated that any serious findings would likely not delay CFT by all that much.
- The MET issue is being downplayed by both NASA and Boeing folks. I keep hearing: "It's a simple fix". Which strongly suggests that a thorough review of the architecture of the Starliner software suite is likely not going to happen. It also suggests that a major review of how Boeing has its software tested is not going to happen either.
Personally I feel that is a MAJOR mistake (but heck, I'm biased because I work in the IT industry as a software QA engineer).
There is precedence for modifying contracts to adjust to changing circumstances.
Of course contracts can be modified. No one is arguing that they can't.
The point everyone is making about looking at the contract is that the current contract REQUIRES that Boeing do an un-crewed ISS docking. And Boeing agreed to that - and is no doubt being paid for that too.
Both NASA and Boeing have reasons to change the un-crewed ISS docking requirement:
- For Boeing, for business reasons, they don't want to incur the additional costs that come with re-flying this mission.
- For NASA, they want a redundant crew transportation capability as quickly as possible.
So there are mutually beneficial reasons for modifying the existing contract, both related to cost and time. unfortunately there is no way to know if "quality" will be sacrificed...
I've been waiting for someone to touch on this because it gives me a chance to share some of the info I got from various sources over the last couple of days. Prime thing being that:
- There is NOT going to be a re-do of OFT. Every indicator I have, from both NASA and industry sources, says that NASA is going to sign off on the OFT milestone, despite the fact that it didn't dock with ISS. One source mentioned that it would have required an IFA hotfire-type mishap (in other words: a catastrophic failure) for NASA to demand a re-do of OFT.
- Legal agreements between Boeing and NASA are going to be altered to reflect this. In other words: requirements regarding ISS approach, emergency retreat, re-approach, docking, docked operations and undocking are going to be removed from the OFT contractual requirments, by MUTUAL consent. All those elements are going to be deferred to CFT.
- CFT is going to be delayed, but not by much (at least, that is the expectation). Bridenstine announced that NASA is going to have an additional good hard look into how things are run in the Boeing Starliner program. My sources confirmed that this is indeed going to happen but they also indicated that any serious findings would likely not delay CFT by all that much.
- The MET issue is being downplayed by both NASA and Boeing folks. I keep hearing: "It's a simple fix". Which strongly suggests that a thorough review of the architecture of the Starliner software suite is likely not going to happen. It also suggests that a major review of how Boeing has its software tested is not going to happen either.
Personally I feel that is a MAJOR mistake (but heck, I'm biased because I work in the IT industry as a software QA engineer).
Agree 100%. In my old role as a software Test Architect, I would want someone very senior to sign this off.
Good post Woods.
Doesn't inspire me with confidence. I run a s/w cloud business and have to deal with my devs and their assumptions (good and bad) on a daily basis. From where I'm sitting the Boeing (not talking about the 737 Max, just the Starliner) software needs a hard look at as there are too many assumptions going on to fill me with any confidence. If this was my s/w stack I'd do a top down review of every system and ensure all teams where fully synced, especially on critical systems. This is just one bug but how many more are there in there?
Yet Nasa appears to be comfortable with pushing on regardless.
https://twitter.com/thegrabster/status/1208487317658308609There is precedence for modifying contracts to adjust to changing circumstances.
Of course contracts can be modified. No one is arguing that they can't.
The point everyone is making about looking at the contract is that the current contract REQUIRES that Boeing do an un-crewed ISS docking. And Boeing agreed to that - and is no doubt being paid for that too.
Both NASA and Boeing have reasons to change the un-crewed ISS docking requirement:
- For Boeing, for business reasons, they don't want to incur the additional costs that come with re-flying this mission.
- For NASA, they want a redundant crew transportation capability as quickly as possible.
So there are mutually beneficial reasons for modifying the existing contract, both related to cost and time. unfortunately there is no way to know if "quality" will be sacrificed...
I've been waiting for someone to touch on this because it gives me a chance to share some of the info I got from various sources over the last couple of days. Prime thing being that:
- There is NOT going to be a re-do of OFT. Every indicator I have, from both NASA and industry sources, says that NASA is going to sign off on the OFT milestone, despite the fact that it didn't dock with ISS. One source mentioned that it would have required an IFA hotfire-type mishap (in other words: a catastrophic failure) for NASA to demand a re-do of OFT.
- Legal agreements between Boeing and NASA are going to be altered to reflect this. In other words: requirements regarding ISS approach, emergency retreat, re-approach, docking, docked operations and undocking are going to be removed from the OFT contractual requirments, by MUTUAL consent. All those elements are going to be deferred to CFT.
- CFT is going to be delayed, but not by much (at least, that is the expectation). Bridenstine announced that NASA is going to have an additional good hard look into how things are run in the Boeing Starliner program. My sources confirmed that this is indeed going to happen but they also indicated that any serious findings would likely not delay CFT by all that much.
- The MET issue is being downplayed by both NASA and Boeing folks. I keep hearing: "It's a simple fix". Which strongly suggests that a thorough review of the architecture of the Starliner software suite is likely not going to happen. It also suggests that a major review of how Boeing has its software tested is not going to happen either.
Personally I feel that is a MAJOR mistake (but heck, I'm biased because I work in the IT industry as a software QA engineer).
An interesting question is, how can the a wrong mission clock affect attitude control that badly.
We heard "wrong deadbands because the craft thought it was doing a burn" but that doesn't explain half of it. In the few displays that we had on the screens, the vehicle was shown to be at 90 degrees off the angle it should have been, firing the thrusters like crazy.
Now the thing is, for an orbit correction burn, the most common orientations are pro-grade and retro-grade. But those aren't trivial orientations, since they are defined relative to a rorating reference frame. Where "prograde" is rotates with one revolution per orbit.
In a spacecraft, attitude is measured using IMUs and a few other sensors such as magnetometers, startrackers, GPS, etc.
But on the lowest level, basically you can always calculate the orientation of the craft relative to the ecliptic (aka relative to the galaxy/ global reference frame. Not the sun, not earth, but the universe as a whole) by simply integrating over all rotations reported by IMUs.
Startrackers also can confirm that orientation, and fix any drifts accumulated, but there are several, redundant space grade IMUs which have very little drift and support integrating over a few hours. The startrackers will confirm and correct this universal frame orientation, by looking at distant stars, which are in the same universal reference frame. (Let's ignore relativistic frame dragging for now, Earth isn't heavy enough for that to have an effect over only a few orbits)
Problem is, that won't tell you where "prograde" is, because "prograde" depends on the position within the orbit, and which side of the earth you are currently on. There are no sensors for that. You cannot measure "down" when in space.
(Well, in theory you could using radar, cameras, computer vision, try to detect the horizon, etc... but Starliner doesn't need to, because:)
Actually, there are. There's GPS, and there's magnetometers.
The magnetometers can tell the orientation of the frame relative to the earth magnetic field, but unfortunately one needs to know the location relative to the earth to know what the field looks like, before the orientation can be calculated, since it is not symmetric (And to make things worse, the earth rotates itself underneath the orbit)
Then of course GPS can provide the location relative to earth (along with a perfect time source) but GPS depends on radio reception and might not be available on all orbits and throughout all regimes (not during ascend in the fairing, not during reentry, ...)
But if the craft has a decent model of the earth, and knows which orbit it is in, it can calculate where it should be - without GPS - using the orbital parameters - and the clock.
Now assume your clock is of by 11 hours, that means, it is off by approximately 7.3 orbits. So basically its 1/3 of an orbit phase shifted to where it thinks it is, and the earth is also rotated by roughly 165° underneath it relative to the assumption (resulting in a drastically shifted geomagnetic field)
so the magnetic field estimate for the location will be off by a lot. And then the idea where prograde is in relation to the universal reference frame is also off.
It puts the attitude estimation into a situation where it has completely wrong assumptions about the world. Startrackers and IMU agree, but then the magnetometers heavily disagree... then again, the earth magnetic field can fluctuate and get distorted by solar wind and other effects, so it would likely just discard the magnetometers. But all GPS receivers telling it a position that doesn't match the one calculated by mission clock? And a magnetic mismatch? That should make the system go into some sort of fallback.
There's 3 thinkable causes for that mismatch.
1. GPS failure/Jamming/Fake Signals
2. Assumed Orbital parameters are wrong
3. Mission time is wrong
funnily enough, by integrating over the received GPS signal, a correct orbit (regardless of mission time reference) can be calculated. A really robust system would do this calculation whenever GPS is available, because then this "measured orbital hypothesis" is available for cross-checking.
Whenever 2 independent systems don't agree on the state of the system as a whole, it's good to have a 3rd hypothesis (preferably with an uncertainty estimate) to fix this.
it's pretty obvious on board Starliner - despite having the sensors available - the software did no such intelligent fault-checking and recovery. it also did not go into a passive failsafe and wait for commands.
instead it burnt through 1/4 of RCS propellant, overheated the RCS thrusters and generally went into what in software is called "undefined behavior"
and that tells anyone with only rudimentary knowledge of control systems, that Starliner most likely cannot be fixed with a single line of code to give it a correct mission time. That would remove the event that triggered the problem/bug, But the problem seems to run deeper.
Also a wrong MET is only one of two kinds this issue can be triggered. If Starliner for some reason "forgets" its orbital parameters or gets them wrong - for example due to a power fluctuation or reboot - it could end up in the same situation again and spew all its RCS like crazy.
I've been waiting for someone to touch on this because it gives me a chance to share some of the info I got from various sources over the last couple of days. Prime thing being that:
- There is NOT going to be a re-do of OFT. Every indicator I have, from both NASA and industry sources, says that NASA is going to sign off on the OFT milestone, despite the fact that it didn't dock with ISS. One source mentioned that it would have required an IFA hotfire-type mishap (in other words: a catastrophic failure) for NASA to demand a re-do of OFT.
- Legal agreements between Boeing and NASA are going to be altered to reflect this. In other words: requirements regarding ISS approach, emergency retreat, re-approach, docking, docked operations and undocking are going to be removed from the OFT contractual requirments, by MUTUAL consent. All those elements are going to be deferred to CFT.
- CFT is going to be delayed, but not by much (at least, that is the expectation). Bridenstine announced that NASA is going to have an additional good hard look into how things are run in the Boeing Starliner program. My sources confirmed that this is indeed going to happen but they also indicated that any serious findings would likely not delay CFT by all that much.
- The MET issue is being downplayed by both NASA and Boeing folks. I keep hearing: "It's a simple fix". Which strongly suggests that a thorough review of the architecture of the Starliner software suite is likely not going to happen. It also suggests that a major review of how Boeing has its software tested is not going to happen either.
Personally I feel that is a MAJOR mistake (but heck, I'm biased because I work in the IT industry as a software QA engineer).
This does not fill me with confidence. If I were a NASA astronaut I would now be having misgivings about flying this spacecraft because it lends the impression that both Boeing and NASA are more concerned with optics than they are with safety. NASA is going to sign off on the OFT as being successfully completed? Really? I know, as Jim has stated repeatedly, that there is no relationship between Starliner and Max, but that only means that we now have 2 completely separate human systems that have suffered avionics problems and Boeing poo-pooed the seriousness of the issue on both unrelated systems. All that does is call into question the competence of Boeing Corp itself. Because of Boeing arrogance, 346 people died on the Max in 2 separate crashes because the avionics left the aircraft uncontrollable, and the flight avionics are still not fixed to this day. Flight avionics on the Starliner left the spacecraft out of control as well. Are people going to die on the Starliner for the same arrogance? Let us hope not. If I were an astronaut assigned to Starliner, I'd put on a good public face but I'd be more than just a little concerned, and I'd be just plain angry because NASA appears to be playing along with it, bowing before the Boeing god. Not to put too fine a point on this, but if it were SpaceX instead of Boeing, I guarantee you that NASA would be all over them like stink on crap. Thank God that Boeing is not also building the Orion spacecraft.
Chuck - You messed up the quoting. That post was from Woods, not NICP.