-
#660 by Vettedrmr on 24 Jun, 2024 01:01
-
...And yes, I know they brought 4 of them back on line, but that's just using software to work around a hardware problem.
Or it was a software problem, i.e., limits too conservative based on the conditions of the previous flight which were different than the one this flight encountered. Things like that happen when you move from a sample of 1 to 2 to 3.
First, it's not a sample size of 1, 2, and 3. There are 28 thrusters per SM (IDK how many are actually used during the flight up to the ISS).
It's not impossible that the limits were just too conservative, BUT best I understand these are the same type of thrusters as used in the previous 2 flights, which gave them a lot of data. Additionally, the other thrusters on this SM *didn't* trip that same limit, so I have a bit of a raised eyebrow at it only being a software issue. -
#661 by SoftwareDude on 24 Jun, 2024 01:50
-
There are a few stories like this floating around and I don't know how true they are, but it is all we have https://www.apr.org/news/2024-06-22/nasa-delays-landing-of-the-problem-filled-starliner-capsule-until-july....And yes, I know they brought 4 of them back on line, but that's just using software to work around a hardware problem.
Or it was a software problem, i.e., limits too conservative based on the conditions of the previous flight which were different than the one this flight encountered. Things like that happen when you move from a sample of 1 to 2 to 3.
Anyway, it sounds like overheating reduces the output of the thrusters, causing them to go offline.QuoteA test firing of the jets on the vehicle showed they’re overheating, possibly due to the frequency of the thrusts, or “burns,” that are needed to return the astronauts safely to the Earth.
A similar article says one of the thrusters was working at only 8% and has been permanently taken offline. This raises the question of whether heat damage permanently damaged one thruster but made others okay after they cooled down. -
#662 by DanClemmensen on 24 Jun, 2024 02:01
-
Are these the same valves that failed during the 2021 launch attempt for OFT-2?This brings up the other question: Are these thrusters different from those on the OFT-2? Was there a problem with them then? It seems like OFT-2 would have seen this problem.
I think this was your question in your latest update. IDK if they're the same or not; I *assume* they're the same, since I've not read anything about them being different. So you question is a good one: why are these failing so much? And yes, I know they brought 4 of them back on line, but that's just using software to work around a hardware problem.
Whatever it is, IMO Boeing has their work cut out for convincing NASA to bring their astronauts on Starliner.
https://en.wikipedia.org/wiki/Boeing_Starliner_Spacecraft_2#Analysis_of_inoperable_valves
In that situation, 13 of the 24 valves failed, and they brought nine of them back alive by cycling them. IIRC they got a few of them back by commanding them to cycle, and a few more by physically moving them by hand after returning the spacecraft to the VIF. Later analysis showed that the valves were corroded, and the root cause was moisture, with a probable but unproven moisture source being the Florida air. The different behavior was apparently due to differing levels of corrosion. The actual valve movement failure was caused by the corrosion products, not directly by corrosion of the valves themselves.
But now on CFT, we saw three different performance levels (assuming 24 valves): 19 valves function normally, four valves failed to function but functioned after being cycled, and one valve refused to function even after being commanded to cycle.
To my mind as an uninformed outside armchair analyst, I see a pattern: something, possibly Florida air, introduced moisture that caused varying amounts of corrosion. Different valves corroded to a lesser or greater extent along a continuum. The total moisture load was smaller during CFT because of the mitigations Boeing added after the OFT-2 scrub and/or the smaller total amount of time the spacecraft was exposed to the environment. So, on CFT, we have 19 wiht not enouhg corrosion to affect function, 4 corroded but recovered, and one corroded to much for recovery by cycling.
But what is the actual mechanism of corrosion, and did the corrosion stop after the spacecraft reach vacuum? even if the corrosion stopped, will the valves all continue to function, or will the corrosion products drift around and cause problems?
This is only speculation and is not intended as FUD. Boeing and NASA probably considered and rejected this hypothesis within five minutes of the first report of a CFT valve failure. -
#663 by abaddon on 24 Jun, 2024 02:08
-
A similar article says one of the thrusters was working at only 8% and has been permanently taken offline.
That information was directly from the most recent NASA/Boeing presser. -
#664 by meekGee on 24 Jun, 2024 05:04
-
Are these the same valves that failed during the 2021 launch attempt for OFT-2?This brings up the other question: Are these thrusters different from those on the OFT-2? Was there a problem with them then? It seems like OFT-2 would have seen this problem.
I think this was your question in your latest update. IDK if they're the same or not; I *assume* they're the same, since I've not read anything about them being different. So you question is a good one: why are these failing so much? And yes, I know they brought 4 of them back on line, but that's just using software to work around a hardware problem.
Whatever it is, IMO Boeing has their work cut out for convincing NASA to bring their astronauts on Starliner.
https://en.wikipedia.org/wiki/Boeing_Starliner_Spacecraft_2#Analysis_of_inoperable_valves
In that situation, 13 of the 24 valves failed, and they brought nine of them back alive by cycling them. IIRC they got a few of them back by commanding them to cycle, and a few more by physically moving them by hand after returning the spacecraft to the VIF. Later analysis showed that the valves were corroded, and the root cause was moisture, with a probable but unproven moisture source being the Florida air. The different behavior was apparently due to differing levels of corrosion. The actual valve movement failure was caused by the corrosion products, not directly by corrosion of the valves themselves.
But now on CFT, we saw three different performance levels (assuming 24 valves): 19 valves function normally, four valves failed to function but functioned after being cycled, and one valve refused to function even after being commanded to cycle.
To my mind as an uninformed outside armchair analyst, I see a pattern: something, possibly Florida air, introduced moisture that caused varying amounts of corrosion. Different valves corroded to a lesser or greater extent along a continuum. The total moisture load was smaller during CFT because of the mitigations Boeing added after the OFT-2 scrub and/or the smaller total amount of time the spacecraft was exposed to the environment. So, on CFT, we have 19 wiht not enouhg corrosion to affect function, 4 corroded but recovered, and one corroded to much for recovery by cycling.
But what is the actual mechanism of corrosion, and did the corrosion stop after the spacecraft reach vacuum? even if the corrosion stopped, will the valves all continue to function, or will the corrosion products drift around and cause problems?
This is only speculation and is not intended as FUD. Boeing and NASA probably considered and rejected this hypothesis within five minutes of the first report of a CFT valve failure.
Are you sure you're not mixing up two failures?
IIRC:
The "5 failed but 4 were coaxed back" I thought was thrusters overheating / exceeding parameters
The valve problem was just a single failure.
This is out of memory - I may be wrong. -
#665 by Vettedrmr on 24 Jun, 2024 10:11
-
Are you sure you're not mixing up two failures?
IIRC:
The "5 failed but 4 were coaxed back" I thought was thrusters overheating / exceeding parameters
The valve problem was just a single failure.
This is out of memory - I may be wrong.
I believe you're correct. -
#666 by laszlo on 24 Jun, 2024 11:24
-
...And yes, I know they brought 4 of them back on line, but that's just using software to work around a hardware problem.
Or it was a software problem, i.e., limits too conservative based on the conditions of the previous flight which were different than the one this flight encountered. Things like that happen when you move from a sample of 1 to 2 to 3.
First, it's not a sample size of 1, 2, and 3. There are 28 thrusters per SM (IDK how many are actually used during the flight up to the ISS).
It's not impossible that the limits were just too conservative, BUT best I understand these are the same type of thrusters as used in the previous 2 flights, which gave them a lot of data. Additionally, the other thrusters on this SM *didn't* trip that same limit, so I have a bit of a raised eyebrow at it only being a software issue.
I was talking about the number of flights, not number of thrusters. It could well be a hardware problem, but until all the data is in and analyzed we can't say for sure which it is. It could also be a process problem with the manufacturing. or something no one has thought of yet. My point was that it's too early to say that it was just using software to work around a hardware problem. I think that we're basically in agreement, -
#667 by Vettedrmr on 24 Jun, 2024 11:29
-
I think that we're basically in agreement.
Also, all I meant by "using software to fix a hardware problem" was that they adjusted the software limits to get those thrusters back on-line, and to not think everything was hokey-dokey (a technical term) with the hardware. -
#668 by DanClemmensen on 24 Jun, 2024 14:37
-
I am definitely not sure. If there are separate problems, the OFT-2 question relates to only the single-valve failure, not the 5-thruster overheating problem.Are these the same valves that failed during the 2021 launch attempt for OFT-2?
https://en.wikipedia.org/wiki/Boeing_Starliner_Spacecraft_2#Analysis_of_inoperable_valves
Are you sure you're not mixing up two failures?
IIRC:
The "5 failed but 4 were coaxed back" I thought was thrusters overheating / exceeding parameters
The valve problem was just a single failure.
This is out of memory - I may be wrong.
If you re correct, then we do not yet have any indication that the proximate cause of the 5-thruster anomaly has anything to do with any valve problem.
In a way, that's worse. It means NASA/Boeing must analyze and make certification decisions about two separate problems. -
#669 by bstrong on 24 Jun, 2024 16:35
-
I think multiple folks are misremembering things. Jeff Foust's article summarizes the latest briefing well:
https://spacenews.com/starliner-iss-stay-extended-to-complete-thruster-and-helium-leak-testing/
Notably:
1) The RCS issue during docking also happened on OFT-2Quote“Coming out of that, we feel very confident in the thrusters and the team is just making sure to go look at the thrusters in detail across the whole flight,” he said. That includes comparing their performance to that on an uncrewed test flight in May 2022, called OFT-2, where two thrusters malfunctioned during approach but were restored before the end of the mission.
2) There were also some helium leaks on OFT-2 (I don't remember any previous mention of this)QuoteThree of the larger leaks likely have similar causes, he noted, while two smaller ones may instead be similar to leaks seen on the OFT-2 mission.
3) Nappi is no longer calling the thruster issues a software calibration problem (like he did in the first briefing after docking), and Stitch is talking about overheatingQuoteStich said engineers are studying what caused the thrusters to go offline during the approach, which may be linked to heavy use. “We have some theories about what’s going on inside the thruster where the thruster gets very hot,” he said, such as high temperatures preventing proper mixing of fuel and oxidizer.
Personally, I don't give credence to anything Nappi says at this point, especially when Stitch is remaining silent while he explains away problems.
-
#670 by Athelstane on 24 Jun, 2024 16:37
-
Via Payload this morning, an interesting back-and-forth:Quote
[ NASA’s commercial crew program manager Steve] Stich said the new high-level review is “similar to what was done ahead of NASA’s SpaceX Demo-2 return after two months on orbit.” One of the lead engineers on SpaceX’s Crew Dragon program, Abhi Tripathi, suggested that circumstances were different.
“The Demo-2 duration was primarily driven by a very specific (one-off) set of crew rotation and geopolitical considerations at that specific point in time,” Tripathi wrote in a social media post. “The demo mission durations were originally conceived to support the provider’s certification data packages. In the case of Starliner, NASA seems intent on collecting additional data for operational certification determination based on how the mission unfolds in real time.”
Link to Payload daily digest: https://pyld.omeclk.com/portal/public/ViewCommInBrowser.jsp?Sv4%2BeOSSucwiV%2BSifRJiNeUHzeOgHiti%2FnkjsnxDrJ1gRLQy0xuIBP0nEgA1zCr3TdZ0Px81yb4%3DA
Link to Tripathi's X post: https://x.com/SpaceAbhi/status/1804337983358931190 -
#671 by DanClemmensen on 24 Jun, 2024 16:51
-
I think multiple folks are misremembering things. Jeff Foust's article summarizes the latest briefing well:
If those theories are valid, then they have a design problem. I hope it's a minor design problem, but they would need to redesign, re-simulate, and possibly create some applicable tests to run on the ground. What is a reasonable estimate on the time this would take? I hope they can do valid tests on the ground, even though the problem was observed in zero g in a vacuum.
https://spacenews.com/starliner-iss-stay-extended-to-complete-thruster-and-helium-leak-testing/QuoteStich said engineers are studying what caused the thrusters to go offline during the approach, which may be linked to heavy use. “We have some theories about what’s going on inside the thruster where the thruster gets very hot,” he said, such as high temperatures preventing proper mixing of fuel and oxidizer.
-
#672 by bstrong on 24 Jun, 2024 17:13
-
The other implication is that the backup deorbit plan using RCS thrusters could be problematic if those thrusters can't stand up to heavy use. And with the other valve and helium issues, the likelihood of an OMAC failure in adjacent doghouses may not seem as remote as it did before launch.
-
#673 by SoftwareDude on 24 Jun, 2024 17:30
-
The other implication is that the backup deorbit plan using RCS thrusters could be problematic if those thrusters can't stand up to heavy use. And with the other valve and helium issues, the likelihood of an OMAC failure in adjacent doghouses may not seem as remote as it did before launch.
I remember they lost one axis of rotation when approaching the station needed to approach the ISS but restored it by bringing a thruster back online. I wish I had a transcript. -
#674 by DanClemmensen on 24 Jun, 2024 20:31
-
In this very unlikely scenario, which two Crew-9 crew members would get bumped? the crew is one Russian and three Americans. I'm guessing the Russian will fly (contractual requirement), so two Americans must stay on the ground. Since Dragon 2 has successfully flown uncrewed to the ISS eleven times, I don't think Dragon flight qualification is absolutely necessary, and the American crew member could technically be any one of Cardman, Hague, or Wilson.Can SpaceX send up a capsule with spacesuits to fit the two astronauts and rescue the two?
In the very unlikely event of a rescue using Crew Dragon, that Dragon will launch with two empty seats. It will have plenty of room for those two suits. SpaceX has four Crew Dragons, and the rescue could be deferred. The simplest would be to fly Crew-9 (already scheduled for August) with only two crew.
They are launching rockets every couple of days. Don't know if they have capsules already designated for specific crews or could build a capsule quickly enough to not delay any future launches.
Ideally, Suni and Butch will come home on Starliner with no problems and the Crew-9 scenario will be moot. -
#675 by AmigaClone on 24 Jun, 2024 22:42
-
In this very unlikely scenario, which two Crew-9 crew members would get bumped? the crew is one Russian and three Americans. I'm guessing the Russian will fly (contractual requirement), so two Americans must stay on the ground. Since Dragon 2 has successfully flown uncrewed to the ISS eleven times, I don't think Dragon flight qualification is absolutely necessary, and the American crew member could technically be any one of Cardman, Hague, or Wilson.Can SpaceX send up a capsule with spacesuits to fit the two astronauts and rescue the two?
In the very unlikely event of a rescue using Crew Dragon, that Dragon will launch with two empty seats. It will have plenty of room for those two suits. SpaceX has four Crew Dragons, and the rescue could be deferred. The simplest would be to fly Crew-9 (already scheduled for August) with only two crew.
They are launching rockets every couple of days. Don't know if they have capsules already designated for specific crews or could build a capsule quickly enough to not delay any future launches.
Ideally, Suni and Butch will come home on Starliner with no problems and the Crew-9 scenario will be moot.
Would it not be possible for Suni and Butch to return as the 'fifth astronaut' on Crew 8 and the next Axion mission as was considered at one point for Rubio after the coolent leak on Soyuz MS22? -
#676 by DanClemmensen on 25 Jun, 2024 01:35
-
No. I think that idea was idle speculation by some of us armchair analysts, considering options for an emergency situation. The problem is that Crew Dragon does not have provision for a fifth astronaut: no couch and no suit umbilical port. The emergency crew return would have required some Apollo-13-style bootleg engineering, including a kludge to connect Rubio's Soyuz to the Crew dragon and an improvised fifth seat build from random soft material (dirty laundry) plus the seat shell from the Soyuz.
In this very unlikely scenario, which two Crew-9 crew members would get bumped? the crew is one Russian and three Americans. I'm guessing the Russian will fly (contractual requirement), so two Americans must stay on the ground. Since Dragon 2 has successfully flown uncrewed to the ISS eleven times, I don't think Dragon flight qualification is absolutely necessary, and the American crew member could technically be any one of Cardman, Hague, or Wilson.Can SpaceX send up a capsule with spacesuits to fit the two astronauts and rescue the two?
In the very unlikely event of a rescue using Crew Dragon, that Dragon will launch with two empty seats. It will have plenty of room for those two suits. SpaceX has four Crew Dragons, and the rescue could be deferred. The simplest would be to fly Crew-9 (already scheduled for August) with only two crew.
They are launching rockets every couple of days. Don't know if they have capsules already designated for specific crews or could build a capsule quickly enough to not delay any future launches.
Ideally, Suni and Butch will come home on Starliner with no problems and the Crew-9 scenario will be moot.
Would it not be possible for Suni and Butch to return as the 'fifth astronaut' on Crew 8 and the next Axion mission as was considered at one point for Rubio after the coolent leak on Soyuz MS22?
For the CFT, NASA/Boeing already have declared that the Starliner spacecraft is usable in an emergency. The ongoing analysis seems to be about whether or not it is to be used if there is no emergency. -
#677 by rst on 25 Jun, 2024 11:42
-
For the CFT, NASA/Boeing already have declared that the Starliner spacecraft is usable in an emergency. The ongoing analysis seems to be about whether or not it is to be used if there is no emergency.
True -- but AIUI, the kind of "emergency" they're talking about is one that threatens to render the station itself uninhabitable. Under those circumstances, Butch and Suni would pretty clearly have better odds on Starliner than they would trying to remain (and it's not like any of the other capsules there have a spare seat!). But NASA saying so, while being kinda quiet otherwise, is ... less than a full vote of confidence in the spacecraft generally. -
#678 by DanClemmensen on 25 Jun, 2024 15:37
-
Quiet is good, I think. They have the time, so they are doing the best analysis they can. While the denizens of this forum would like some insight into NASA's process and interim preliminary findings, they would almost certainly result in even more manic/depressive clickbait headlines in the general press: disaster predicted on even-numbered days and re-evaluations and reassurance on odd-numbered days.For the CFT, NASA/Boeing already have declared that the Starliner spacecraft is usable in an emergency. The ongoing analysis seems to be about whether or not it is to be used if there is no emergency.
True -- but AIUI, the kind of "emergency" they're talking about is one that threatens to render the station itself uninhabitable. Under those circumstances, Butch and Suni would pretty clearly have better odds on Starliner than they would trying to remain (and it's not like any of the other capsules there have a spare seat!). But NASA saying so, while being kinda quiet otherwise, is ... less than a full vote of confidence in the spacecraft generally.
The Crew-8 capsule probably has the capability to land two extra astronauts using last-ditch heroic Apollo-13-style kludge engineering. Crew dragon was originally designed to carry seven, and the space for the extra three seats is more or less still there behind the four available seats. Uncomfortable and dangerous, but less so than a launch abort. NASA has almost certainly decided that Starliner is safer than any such kludge, and there are multiple alternatives if there is no critical emergency on ISS.
I still hope the Starliner can be approved for the return and I suspect (without any inside info) that this will happen. -
#679 by Vettedrmr on 25 Jun, 2024 16:34
-
Quiet is good, I think. They have the time, so they are doing the best analysis they can.
There's a saying in the defense industry when it comes to investigations like this: "No news is, no news." NASA/Boeing made a good call to stop pushing the departure date by a day or two and just say that it's going to be later than July 2nd. As you mention, there's no reason for urgency for their return (other than the 45 day limit on the vehicle), and plenty of reasons to do the best job they can of analyzing the data.QuoteI still hope the Starliner can be approved for the return and I suspect (without any inside info) that this will happen.
Ditto. The rank and file at Boeing need some good news.
