Proton-M Failure Reaction and Discussion Thread - July 2, 2013

[Prober]

"According to the info leaks on NK forum, a whole assembly of angular velocity sensors and attitude angle transducers (apparently in the roll channel) was simply installed upside down on this unhappy rocket, and that is what was clearly seen in the wreck pieces... How that was possible to do and escaped quality checks is yet to be seen. Rumors are that the guy who was doing the assembling has only worked for several months on the plant. "

poor design then.....should be set to only install one way the proper one.  I would not place blame on the new worker. Management and training should be helping him.

[Targeteer]

"According to the info leaks on NK forum, a whole assembly of angular velocity sensors and attitude angle transducers (apparently in the roll channel) was simply installed upside down on this unhappy rocket, and that is what was clearly seen in the wreck pieces... How that was possible to do and escaped quality checks is yet to be seen. Rumors are that the guy who was doing the assembling has only worked for several months on the plant. "

poor design then.....should be set to only install one way the proper one.  I would not place blame on the new worker. Management and training should be helping him.

A crash investigation following an F-15 crash in Germany revealed identical flight control rods (pitch and roll) were re-installed after maintenance in reverse by experienced and fully qualified personnel.  The pilot pulled the stick to climb out and rolled unexpectedly on take-off, crashing.  A post service functional check was not accomplished but signed off as complete in service logs.  One of the maintainers committed suicide following the investigation.

A functional check of the guidance elements should have detected the installation error--if it was performed, or required...

[Kabloona]

From russianspaceweb.com:

"By July 9, it is transpired that investigators sifting through the wreckage of the doomed rocket had found critical angular velocity sensors, DUS, installed upside down. Each of those sensors had an arrow that was suppose to point toward the top of the vehicle, however multiple sensors on the failed rocket were pointing downward instead. As a result, the flight control system was receiving wrong information about the position of the rocket and tried to "correct" it, causing the vehicle to swing wildly and, ultimately, crash. The paper trail led to a young technician responsible for the wrong assembly of the hardware, but also raised serious issues of quality control at the Proton's manufacturing plant, at the rocket's testing facility and at the assembly building in Baikonur. It appeared that no visual control of the faulty installation had been conducted, while electrical checks had not detected the problem since all circuits had been working correctly."

[Ben the Space Brit]

They'll probably have to strip down every Proton on the line and in the warehouses now to ensure that their G/N sensors control mechanism feedback sensors are installed correctly.  That could mean a lengthy stand-down for the type.

[Prober]

They'll probably have to strip down every Proton on the line and in the warehouses now to ensure that their G/N sensors control mechanism feedback sensors are installed correctly.  That could mean a lengthy stand-down for the type.

yes, side mount assembly and new trainees need re-enforcement.  Can be many other hidden problems from the last few months.

[Crispy]

Orientation-sensitive parts should be shaped/mounted such that upside-down installation is impossible, IMO.

[asmi]

They'll probably have to strip down every Proton on the line and in the warehouses now to ensure that their G/N sensors control mechanism feedback sensors are installed correctly.  That could mean a lengthy stand-down for the type.

You don't need to re-check every Proton to return it to flight. If that is indeed the root cause, just make sure the one you launch is OK.

[zaitcev]

How that was possible to do and escaped quality checks is yet to be seen.

Apparently only the integrity of electrical connections was checked against wires broken, shorting, or swapped around. In this case wires were correct but the sensor installed backwards into its fixture. Interestingly enough, the wire bundle "keys" the sensors because the connector is on one end of the cylinder. So you'd think it were fail-safe. But if you offset the whole harness, you can install all the sensors backwards (although to be sure the pictures at NK were from a different application, not specifically Proton).

[Remes]

The sensors are really designed in a symmetrical shape and can theoretically be installed in the wrong way: http://www.zavod-zvezda.ru/produkciya.html

Those two mention the Proton:

Small-sized float gyro unit - TSE99-6SA intended for use as a sensor gauge angular velocity.
Mounted on board the launch vehicle (LV), "Proton-M".
Weight (g) 300
Dimensions (mm) Ǿ50h93, 5
Current Transfer Ratio torque sensor (A / K / hr)
4.2 x10-3
100x10-3
3 520h10
Power consumption (W) <= 1.2
Systematic component of the drift independent of overload (deg / hr) <= 4.2
The random component of the drift (deg / hr) <= 0.025
Operational life, (h) 50000

 
TSE99-1EA
TSE99-1EA

Float gyroscopic device TSE99-1EA designed for use in complex control units (PSC), in gyro-stabilized platforms as an indicator of small angular velocities and azimuthal orientation.
Successfully used on spacecraft, as launch vehicles (LV): "Proton-M", "Zenit-2SL», «Zenit-2SLB", and the boosters, "Fregat", "DM-SL», «DM 03. "

Key Features:
Characteristics Value
Weight (g) 510
Dimensions (mm) Ǿ64h95
Power Consumption (W) <= 4.3
The transfer coefficient current moment sensor (mA / ° C / hour) 170h10-3
Systematic component of the drift independent of overload (deg / hr) <= 0.65
The random component of the drift (deg / hr) <= 0.005
The random component of the drift in the azimuthal startup (deg / hr) 0.0010
Resource work (hours) 150000

So, does any one know, whether this is the IMU itself? Taking the angular velocity and getting the angles of the rocket by integration? Or is there another imu which delivers the angles, and this angular velocity sensor are taken for control purposes, as deriving the angles doesn't deliver a good signal?

A functional check of the guidance elements should have detected the installation error--if it was performed, or required...

Guess it's hard to rotate a rocket. One could have tested the sensors when the rocket was errected from the horizontal to the vertical. At least for the sensors whose measurement axis is perpendicular to the errection plane.

[Prober]

http://www.russianspaceweb.com/proton_glonass49.html#culprit
 
"The paper trail led to a young technician responsible for the wrong assembly of the hardware, but also raised serious issues of quality control at the Proton's manufacturing plant, at the rocket's testing facility and at the assembly building in Baikonur. It appeared that no visual control of the faulty installation had been conducted, while electrical checks had not detected the problem since all circuits had been working correctly."
 
not a good way to launch a rocket of any type and this one had the toxic nature (should be extra careful)

[Jim]

A functional check of the guidance elements should have detected the installation error--if it was performed, or required...

Guess it's hard to rotate a rocket. One could have tested the sensors when the rocket was errected from the horizontal to the vertical. At least for the sensors whose measurement axis is perpendicular to the errection plane.

US rockets use to have a test (guidance torque test or something similar) to prevent a miswired IMU, where the rocket would go west vs east. (looking to see if this is still done)

[Stan Black]

Pictures of the roll out on June 8th
http://novosti-kosmonavtiki.ru/forum/messages/forum12/topic12414/message1081962/#message1081962

Why the Proton launcher rolls out from MIK-92A-50 fairing first ??

Hope they realise it is back-to-front when they get it to the launch site!

I really should not have said that… sorry Khrunichev.

[kevin-rf]

Wow, one of the media outlets actually quotes the factual information on what might of gone wrong!

http://www.theblaze.com/stories/2013/07/10/the-reported-simple-reason-why-that-russian-rocket-may-have-crashed-and-the-conspiracy-around-it/

Though at the end they include a conspiracy theory about the crash... If you thought Foxnews was right wing, you'll love the Blaze!

[Kabloona]

A functional check of the guidance elements should have detected the installation error--if it was performed, or required...

Guess it's hard to rotate a rocket. One could have tested the sensors when the rocket was errected from the horizontal to the vertical. At least for the sensors whose measurement axis is perpendicular to the errection plane.

US rockets use to have a test (guidance torque test or something similar) to prevent a miswired IMU, where the rocket would go west vs east. (looking to see if this is still done)

On Transfer Orbit Stage, our IMU was totally integrated, i.e. all gyros were inside the box and thus there were no external sensors that could be mis-installed. Before first flight, we ran a 24-hr end-to-end IMU/TVC check to verify that, as the earth turned and the IMU sensed the rotation, the TVC vectored in the right direction. Fortunately it did.

However, if I read the report correctly, Proton has individual angular *rate* sensors that are external to the IMU, and some of those were installed upside down. An end-to-end system test would not have caught that error, because a *rate* sensor will give the same zero reading regardless of orientation if the vehicle is sitting still.

Anyway, the simple solution is for QC to look and verify each rate sensor arrow is pointing up. 

[ugordan]

Meet the QC guy

Well, at least he definitely got fired 

I'm guessing you didn't see Caddyshack.

[edkyle99]

However, if I read the report correctly, Proton has individual angular *rate* sensors that are external to the IMU, and some of those were installed upside down. An end-to-end system test would not have caught that error, because a *rate* sensor will give the same zero reading regardless of orientation if the vehicle is sitting still.

Would these be like "rate gyros"?

 - Ed Kyle

[Kabloona]

Would these be like "rate gyros"?

 - Ed Kyle

That's how I read it, but I don't claim firsthand knowledge of the sensors in question.

[simonbp]

Anyway, the simple solution is for QC to look and verify each rate sensor arrow is pointing up. 

Which has been the same all the recent Russian failures. The design would work fine if only it were actually built right... 

[owais.usmani]

How come Proton QC issues always emerge on Russian federal missions, and never on ILS missions?

[woods170]

How come Proton QC issues always emerge on Russian federal missions, and never on ILS missions?

Coincidence. But that will undoubtly be questioned by the more conspiracy-oriented forum members here.