-
#1140
by
SoftwareDude
on 16 Feb, 2020 20:47
-
There are lots of things that can go wrong. Maybe its interference. Maybe it's something else. However, Boeing seems to know what goes wrong as soon as it happens. Before anyone can investigate a problem, someone has a narrative. With the 737 Max, on the day after the second crash, Boeing announced that software fixes were already available. We know how that went.
This is a sign of a poorly run organization. Here is what I think we know so far. If you have something to add please do.
1. The OMT did not fire
2. The Attitude Maneuvering thrusters used more fuel than it should have given the OMT did not fire
2a. Some of the thrusters overheated and failed because of excessive firing. (Thank you PM3)
3. One thruster did not fire at all
4. There was a valve mapping problem fixed in flight, whatever that means
5. There was a communication outage
-
#1141
by
PM3
on 16 Feb, 2020 20:55
-
2a. Some of the thrusters overheated and failed because of excessive firing.
-
#1142
by
Pueo
on 16 Feb, 2020 22:06
-
I agree - this story about cell tower interference makes no sense if the comms issue happened shortly after the launch when Starliner was over the Atlantic.
Not that it should matter, but I believe the interference is claimed to have occurred while over Europe, not the Atlantic.
Starliner separation from the Centaur was south of Greenland, over the open north Atlantic, according to this map.
The Centaur MECO was well off-shore of Nova Scotia.
The last several posts agree on the logical point that this cell tower interference "explanation" lacks at least some critical detail, and doesn't yet make any sense.
Watching the replay of the launch the telemetry seems to be updating reliably until about 1 min before the scheduled orbital insertion maneuver where it starts to freeze for seconds at a time. Reportedly the absence of the burn is what alerted ground control to the MET error and the issues with uplinking the commands came in response to that. Therefore antenna issue must have been occurring while Starliner was squarely over Saudi Arabia.
Tinfoil hat I wonder if Starliner was being lit up by any BMD radar systems while it was flying over the Middle East. It was flying slow enough to deorbit so it might not have been just classified as a satellite and ignored. I know that the Aegis AN/SPY-1 at least runs S-band.
-
#1143
by
jpo234
on 17 Feb, 2020 13:00
-
Tinfoil hat I wonder if Starliner was being lit up by any BMD radar systems while it was flying over the Middle East. It was flying slow enough to deorbit so it might not have been just classified as a satellite and ignored. I know that the Aegis AN/SPY-1 at least runs S-band.
BMD = Ballistic Missile Defense
-
#1144
by
Rebel44
on 17 Feb, 2020 15:01
-
I agree - this story about cell tower interference makes no sense if the comms issue happened shortly after the launch when Starliner was over the Atlantic.
Not that it should matter, but I believe the interference is claimed to have occurred while over Europe, not the Atlantic.
Starliner separation from the Centaur was south of Greenland, over the open north Atlantic, according to this map.
The Centaur MECO was well off-shore of Nova Scotia.
The last several posts agree on the logical point that this cell tower interference "explanation" lacks at least some critical detail, and doesn't yet make any sense.
Watching the replay of the launch the telemetry seems to be updating reliably until about 1 min before the scheduled orbital insertion maneuver where it starts to freeze for seconds at a time. Reportedly the absence of the burn is what alerted ground control to the MET error and the issues with uplinking the commands came in response to that. Therefore antenna issue must have been occurring while Starliner was squarely over Saudi Arabia.
https://www.youtube.com/watch?v=PIDytLL734A?t=5815
Tinfoil hat I wonder if Starliner was being lit up by any BMD radar systems while it was flying over the Middle East. It was flying slow enough to deorbit so it might not have been just classified as a satellite and ignored. I know that the Aegis AN/SPY-1 at least runs S-band.
The power output of air defense radars is so much higher that there would be no chance to mistake it for extra noisy cell phone towers. Frequencies used by missile defense radars (like radar used for THAAD) are usually in the X band. AN/SPY-1 is using S-band, but it should be easy to differentiate that someone just hit your receiver with ~6MW radar.
Warships in the area should also be well aware of planned space launches.
-
#1145
by
abaddon
on 17 Feb, 2020 15:12
-
IIRC the interference issue was initially reported as a "antenna hardware issue" but that was almost immediately changed to "radio interference". The likely answer is the hardware is too sensitive and will need to be fixed to be more robust.
-
#1146
by
Barley
on 17 Feb, 2020 19:36
-
TDRS satellites are also in a *higher* orbit than Starliner, which makes it a little odd that ground interference could have a significant effect. It may be that the spacecraft antenna design was not directional enough that some ground interference could be captured.
IIRC the narrative was that the craft had poor attitude control and was not aiming the primary antennae at TRDS. They were trying to communicate with a secondary, presumably omni-directional, antennae when the cell interference occurred. With an omni-directional antennae it does not matter which direction the interference is coming from.
-
#1147
by
MaxTeranous
on 17 Feb, 2020 21:32
-
The bigger issue is that it’s not exactly an unknown unknown. Mobile phones have been a thing for a long time now, very much an expected rf hazard, and not designing/testing the capsule to cope with it is another black mark.
-
#1148
by
thirtyone
on 17 Feb, 2020 21:40
-
TDRS satellites are also in a *higher* orbit than Starliner, which makes it a little odd that ground interference could have a significant effect. It may be that the spacecraft antenna design was not directional enough that some ground interference could be captured.
IIRC the narrative was that the craft had poor attitude control and was not aiming the primary antennae at TRDS. They were trying to communicate with a secondary, presumably omni-directional, antennae when the cell interference occurred. With an omni-directional antennae it does not matter which direction the interference is coming from.
Honestly, if that were the case, I'd personally feel NASA should not have included "antenna interference" as a problem. It's just another symptom of the MET (maybe thruster?) issues. The most sane way to deal with ground interference, apart from somehow forcing the rest of the world to comply with your spectrum requirements, is to use more directional antennas, and if you don't know where to point, I don't think there's a whole lot more you could do with your antenna design to mitigate the issue. Using a second directional antenna instead of an omni would have just resulted in two incorrectly pointed antennas.
Either it was a backup omni antenna, in which case I would not blame the antenna system but whatever was pointing the primary antenna, or it wasn't, and the phased array performance was insufficient, in which case it would be reasonable to blame the antenna system.
Another interesting note - Boeing also designed the latest three TDRS satellites. They *really* ought to be quite familiar with the design requirements for communicating with those birds and avoiding ground interference, so I don't know what's going on there.
-
#1149
by
A12
on 17 Feb, 2020 21:53
-
The power output of air defense radars is so much higher that there would be no chance to mistake it for extra noisy cell phone towers. Frequencies used by missile defense radars (like radar used for THAAD) are usually in the X band. AN/SPY-1 is using S-band, but it should be easy to differentiate that someone just hit your receiver with ~6MW radar.
Warships in the area should also be well aware of planned space launches.
If the receiver onboard has a wide enough bandwidth, the high power air defense radar could have desensitized it completely via rf input stages saturation.
-
#1150
by
Chasm
on 17 Feb, 2020 21:55
-
The bigger issue is that it’s not exactly an unknown unknown. Mobile phones have been a thing for a long time now, very much an expected rf hazard, and not designing/testing the capsule to cope with it is another black mark.
What I would like to know on how much legacy that part of the design has. In the specs, design or implementation.
There was a lot of stress on legacy in virtually any presentation. With the flight test we learned that at least in the instance of the mission clock there seems to be a lot more than a sane amount of faithfully emulating the behavior of an old electro(?)mechanical legacy systems.
In 2020 it would not surprise me if the Starliner comm system is in part or whole OTS, from a time before cell or even mobile phones were a thing.
-
#1151
by
PM3
on 17 Feb, 2020 22:09
-
Cell towers are just a speculation that was briefly mentioned at a the recent news conference. The Boeing guy who accidentially mentioned this speculation probably regrets it, given all the ongoing speculation now that is based on his speculation. They actually didn't know at that time what caused the interference, and there is no later information on this issue.
-
#1152
by
ChrisWilson68
on 19 Feb, 2020 03:48
-
Could there have been a possibility that ULA would have noticed the wrongly-set timer during the L-7 minute GO/NO GO poll and scrub the attempt to investigate it?
Why would ULA have insight into their payload state machine or be bothered about it at all? That is on the spacecraft owner to do. All ULA needs is a GO decision from the spacecraft side to proceed with the launch.
That’s kind of the point I was making. I meant to say I was wondering what if the spacecraft side noticed the timer issue and relayed it to ULA during the poll.
I agree Zach, if you go back right after the occurrence I posted the the same question a couple of months back (I know this is a long thread) what were they looking at on the consoles during the final GO/NO GO polls... Something was either: overlooked, over-ruled, incompetent and covered-up on both sides (Boeing and NASA)...
https://www.nasa.gov/feature/how-the-mission-is-controlled-inside-nasa-and-boeing-joint-operations
Before launch, there was no wrongly-set timer on the spacecraft. So it couldn't have been noticed on someone's data console because the problem hadn't happened yet.
The MET was supposed to be set to zero at time of spacecraft separation from the rocket, after it was in orbit. After separation, if the MET happened to be one of the pieces of data transmitted by the spacecraft, it might show up on someone's console and someone might notice. But before that, it wouldn't really matter what the MET was set to since it was expected to be zeroed at separation.
Having the MET set to eleven hours just before launch was probably normal and expected, because it was just a garbage value at that point. It wouldn't be set to the correct value until separation, and nothing would use it before separation either.
-
#1153
by
SoftwareDude
on 19 Feb, 2020 07:18
-
Could there have been a possibility that ULA would have noticed the wrongly-set timer during the L-7 minute GO/NO GO poll and scrub the attempt to investigate it?
Why would ULA have insight into their payload state machine or be bothered about it at all? That is on the spacecraft owner to do. All ULA needs is a GO decision from the spacecraft side to proceed with the launch.
That’s kind of the point I was making. I meant to say I was wondering what if the spacecraft side noticed the timer issue and relayed it to ULA during the poll.
I agree Zach, if you go back right after the occurrence I posted the the same question a couple of months back (I know this is a long thread) what were they looking at on the consoles during the final GO/NO GO polls... Something was either: overlooked, over-ruled, incompetent and covered-up on both sides (Boeing and NASA)...
https://www.nasa.gov/feature/how-the-mission-is-controlled-inside-nasa-and-boeing-joint-operations
Before launch, there was no wrongly-set timer on the spacecraft. So it couldn't have been noticed on someone's data console because the problem hadn't happened yet.
The MET was supposed to be set to zero at time of spacecraft separation from the rocket, after it was in orbit. After separation, if the MET happened to be one of the pieces of data transmitted by the spacecraft, it might show up on someone's console and someone might notice. But before that, it wouldn't really matter what the MET was set to since it was expected to be zeroed at separation.
Having the MET set to eleven hours just before launch was probably normal and expected, because it was just a garbage value at that point. It wouldn't be set to the correct value until separation, and nothing would use it before separation either.
This has been bothering me too but I have a theory. The Boeing folks would like to have all the mission data correlated across. all systems including the Atlas, Centaur, and Starliner itself. So each record contains a timestamp that makes sense across all elements of the flight; completely synchronized down to the tick. I know how the AP-101 shuttle computers could do that. The AP-101 can have an external time that is loaded into it from an external source, even another AP-101. That's the only computer that I know of that can do that. ULA currently provides support for the HAL/S compiler. So are these guys all using AP-101s? God help us, but that is my working theory. Boeing uses the AP-101 (System 4/PI) on some of its F-15 fighters, E3-Sentry aircraft, and Harpoon anti-ship missile.
SpaceX evidently designed and built its own flight computers using three non-hardened intel CPUs each with voting. There are three of these flight computers also voting. SpaceX flight computers run Linux.
-
#1154
by
Rocket Science
on 19 Feb, 2020 23:53
-
Could there have been a possibility that ULA would have noticed the wrongly-set timer during the L-7 minute GO/NO GO poll and scrub the attempt to investigate it?
Why would ULA have insight into their payload state machine or be bothered about it at all? That is on the spacecraft owner to do. All ULA needs is a GO decision from the spacecraft side to proceed with the launch.
That’s kind of the point I was making. I meant to say I was wondering what if the spacecraft side noticed the timer issue and relayed it to ULA during the poll.
I agree Zach, if you go back right after the occurrence I posted the the same question a couple of months back (I know this is a long thread) what were they looking at on the consoles during the final GO/NO GO polls... Something was either: overlooked, over-ruled, incompetent and covered-up on both sides (Boeing and NASA)...
https://www.nasa.gov/feature/how-the-mission-is-controlled-inside-nasa-and-boeing-joint-operations
Before launch, there was no wrongly-set timer on the spacecraft. So it couldn't have been noticed on someone's data console because the problem hadn't happened yet.
The MET was supposed to be set to zero at time of spacecraft separation from the rocket, after it was in orbit. After separation, if the MET happened to be one of the pieces of data transmitted by the spacecraft, it might show up on someone's console and someone might notice. But before that, it wouldn't really matter what the MET was set to since it was expected to be zeroed at separation.
Having the MET set to eleven hours just before launch was probably normal and expected, because it was just a garbage value at that point. It wouldn't be set to the correct value until separation, and nothing would use it before separation either.
Since Bridenstine said it was an "automation" error, then they should have a "spacecraft timer" readout during status check for polling. Just sayin'...
-
#1155
by
erioladastra
on 20 Feb, 2020 00:36
-
Could there have been a possibility that ULA would have noticed the wrongly-set timer during the L-7 minute GO/NO GO poll and scrub the attempt to investigate it?
Why would ULA have insight into their payload state machine or be bothered about it at all? That is on the spacecraft owner to do. All ULA needs is a GO decision from the spacecraft side to proceed with the launch.
That’s kind of the point I was making. I meant to say I was wondering what if the spacecraft side noticed the timer issue and relayed it to ULA during the poll.
I agree Zach, if you go back right after the occurrence I posted the the same question a couple of months back (I know this is a long thread) what were they looking at on the consoles during the final GO/NO GO polls... Something was either: overlooked, over-ruled, incompetent and covered-up on both sides (Boeing and NASA)...
https://www.nasa.gov/feature/how-the-mission-is-controlled-inside-nasa-and-boeing-joint-operations
Before launch, there was no wrongly-set timer on the spacecraft. So it couldn't have been noticed on someone's data console because the problem hadn't happened yet.
The MET was supposed to be set to zero at time of spacecraft separation from the rocket, after it was in orbit. After separation, if the MET happened to be one of the pieces of data transmitted by the spacecraft, it might show up on someone's console and someone might notice. But before that, it wouldn't really matter what the MET was set to since it was expected to be zeroed at separation.
Having the MET set to eleven hours just before launch was probably normal and expected, because it was just a garbage value at that point. It wouldn't be set to the correct value until separation, and nothing would use it before separation either.
MET is set at ignition, not spacecraft separation.
-
#1156
by
erioladastra
on 20 Feb, 2020 00:37
-
The bigger issue is that it’s not exactly an unknown unknown. Mobile phones have been a thing for a long time now, very much an expected rf hazard, and not designing/testing the capsule to cope with it is another black mark.
What I would like to know on how much legacy that part of the design has. In the specs, design or implementation.
There was a lot of stress on legacy in virtually any presentation. With the flight test we learned that at least in the instance of the mission clock there seems to be a lot more than a sane amount of faithfully emulating the behavior of an old electro(?)mechanical legacy systems.
In 2020 it would not surprise me if the Starliner comm system is in part or whole OTS, from a time before cell or even mobile phones were a thing.
No it is not that old. First, remember that you might be viewing TDRS with the earth limb in the field of view. Second, not all countries follow US regulations on where their cell tower emissions go. Third, other spacecraft, including ISS, are starting to run into RFI due to cell towers.
-
#1157
by
erioladastra
on 20 Feb, 2020 00:44
-
TDRS satellites are also in a *higher* orbit than Starliner, which makes it a little odd that ground interference could have a significant effect. It may be that the spacecraft antenna design was not directional enough that some ground interference could be captured.
IIRC the narrative was that the craft had poor attitude control and was not aiming the primary antennae at TRDS. They were trying to communicate with a secondary, presumably omni-directional, antennae when the cell interference occurred. With an omni-directional antennae it does not matter which direction the interference is coming from.
As I have stated elsewhere in this thread, "poor attitude" is incorrect. There is no primary antenna. How software chose antennae was likely a contributing factor.
-
#1158
by
erioladastra
on 20 Feb, 2020 00:45
-
Could there have been a possibility that ULA would have noticed the wrongly-set timer during the L-7 minute GO/NO GO poll and scrub the attempt to investigate it?
Why would ULA have insight into their payload state machine or be bothered about it at all? That is on the spacecraft owner to do. All ULA needs is a GO decision from the spacecraft side to proceed with the launch.
That’s kind of the point I was making. I meant to say I was wondering what if the spacecraft side noticed the timer issue and relayed it to ULA during the poll.
I agree Zach, if you go back right after the occurrence I posted the the same question a couple of months back (I know this is a long thread) what were they looking at on the consoles during the final GO/NO GO polls... Something was either: overlooked, over-ruled, incompetent and covered-up on both sides (Boeing and NASA)...
https://www.nasa.gov/feature/how-the-mission-is-controlled-inside-nasa-and-boeing-joint-operations
Before launch, there was no wrongly-set timer on the spacecraft. So it couldn't have been noticed on someone's data console because the problem hadn't happened yet.
The MET was supposed to be set to zero at time of spacecraft separation from the rocket, after it was in orbit. After separation, if the MET happened to be one of the pieces of data transmitted by the spacecraft, it might show up on someone's console and someone might notice. But before that, it wouldn't really matter what the MET was set to since it was expected to be zeroed at separation.
Having the MET set to eleven hours just before launch was probably normal and expected, because it was just a garbage value at that point. It wouldn't be set to the correct value until separation, and nothing would use it before separation either.
This has been bothering me too but I have a theory. The Boeing folks would like to have all the mission data correlated across. all systems including the Atlas, Centaur, and Starliner itself. So each record contains a timestamp that makes sense across all elements of the flight; completely synchronized down to the tick. I know how the AP-101 shuttle computers could do that. The AP-101 can have an external time that is loaded into it from an external source, even another AP-101. That's the only computer that I know of that can do that. ULA currently provides support for the HAL/S compiler. So are these guys all using AP-101s? God help us, but that is my working theory. Boeing uses the AP-101 (System 4/PI) on some of its F-15 fighters, E3-Sentry aircraft, and Harpoon anti-ship missile.
SpaceX evidently designed and built its own flight computers using three non-hardened intel CPUs each with voting. There are three of these flight computers also voting. SpaceX flight computers run Linux.
There are no AP-101s on Starliner.
-
#1159
by
SoftwareDude
on 20 Feb, 2020 02:31
-
There are no AP-101s on Starliner.
Do you know what they are using?