Author Topic: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread  (Read 941280 times)

Offline arnezami

  • Full Member
  • **
  • Posts: 282
  • Liked: 262
  • Likes Given: 345
Hi guys,

Nice to see its starting to work for you too. :)

I'm going to explain a bit more how this all  works and how you can use it. Because I feel there is a need for it.

Attached are 4 illustrations representing 4 scenarios. I think these are going to be useful in understanding how it all works and what you can do with this tool.

1. All is perfect.

In the first illustration you see that all is perfect. The top part represents the bits from the stream. You can see that there are "end markers" in the stream. These basicly tell the decoder that the data for a block has ended. (btw I'm not entirely sure how this actually works, but this I think is close to it). At each end-marker the decoder happily switches to the next block of 16x16 pixels. Note that the length of the data for each 16x16 block is not the same.

The lower part represents the 16x16 pixel blocks. They have identifying numbers. And as you can see they have the colors (representing its content) of the correct bits-stream-parts. So far so good.

2. Defect in bitstream

In the second illustration you see there is a defect in the bitstream (yellow). A bitsteam error can simply mean the content of a 16x16 pixel block doesn't seem quite right (without affecting other blocks too much). However lets assume that this time its worse: the defective bits are now interpreted by the decoder as an end-marker! Ouch. This is big trouble because firstly block 2 is broken off too early and will contain some nasty data (yellow in lower part) which can severely affect all blocks to its right and bottom). But secondly block 3 starts with completely wrong data! It is now on the complete wrong path since all bits are interpreted wrongly and the chance that some of these bits are (wrongly) interpreted as end-markers increases too. This is actually happening in this illustration/scenario. And at some point it even reads a sequence of bits that don't make sense at all. The decoder stops in such a situation. This leaves blocks 5-9 grey. This is the reason only the top parts of the I-frames were visible and the lower parts were grey.

In order to fix this you could try to restore the bitstream itself. But this is very hard to do and extremely time consuming (fixing compressed data is no joke!). The worse is: you don't know when the bad bits stop. We need to know where the good bits start again and try to avoid being dependent on having to do tedious and time consuming work in order to make any progress.

3. Reassigning the bitpointer twice

The problem now becomes: how do you "skip" a block? This seems impossible to do. There is no nice index of the bitstream (as fas as I know).

The first thing to do is make sure is not to halt the decoder when it encounters an error. This way the decoder sometimes rediscovers actual end-marks. And when it does it usually will keep finding more valid end-marks. Of course these will be read by the wrong blocks (aka you see them appearing in the wrong position in the picture). But still useful.

So we now know some valid end-marks (which also mark the start of block-data of course). In order to make use of them we should on-the-fly reassign the bitstream pointer of a block to such an end/start-marker. But first we need to make sure that block 2 doesn't read the bad bits.

So we do two reassignments:

1) Just before the decoder starts with decoding block 2 we reassign its bitpointer to (for example) a block in the past. In this case we set its bitpointer to the bits of block 1 (see third illustration). This is mainly to prevent that really bad visual effects come into our pixels (maybe there is a way to simply "do nothing" with a block, which may be better. I don't know, maybe others do).

2) Just before block 3 starts we assign its bitstream to the end-mark of block 2 (aka start-mark of block 3. Btw  we assume we found/guessed this somehow. This prevents us from reading the bad bits. The bad data is "skipped".

In the lower part of illustration 3 you can see how the content will look like if we do the above. Only one block is "broken" (contains data from a different block that is) and all the other blocks get their own intended data.

4. Reassigning the bitpointer once

A useful technique I discovered was that if you know the bitposition of certain good block-data (using a combination of your eyes which make out the x,y coordinate and the log which contains the corresponding bitpointer at that time) you can simply set the bitpointer of block 0 to that bitposition. I do this is in illustration 4. Then you can easily see how much good data there is and where something is wrong again. If you do this randomly (assigning block 0 to random bitpositions) you can easely detect good data parts in the stream. Which is what we need. So with this you can create a "data-map" of good and bad data. And that will eventually allow you to carefully "skip" the bad data.

Hopefully that helps. My fingers hurt :P

Regards and have a good day,

arnezami

[edit] grammar
« Last Edit: 05/05/2014 05:06 AM by arnezami »

Offline dorkmo

  • Full Member
  • ****
  • Posts: 682
  • Liked: 308
  • Likes Given: 815
awesome stuff.

to comment on the content of what can be seen so far,
it looks to me like the legs both came out a little bit. then only the left leg progressed, then stopped. then the right leg extended fully. then finally the left leg extended fully.

i wonder if thats typical of whats been seen in testing

Are you serious? I don't see how you could have possibly seen such detail from any version of the video so far. SpaceX has the actual flight data of all these systems and they say the legs deployed correctly as hoped. I'm gonna go with that.

watching the coerced video, it looks like the entire deployment occured durring the clock time of 00:06 to 00:10. theres only a sliver of blocks in a handful of frames and the image jumps around from left to right a lot.

also it looks like around 15 seconds you stop seeing any signs of flame

and to me it looks like maybe at 17 you can see a splash up of water from impact or a leaning/settling of the rocket to the side, which is up in the video

Offline mlindner

  • Software Engineer
  • Full Member
  • ****
  • Posts: 1953
  • Space Capitalist
  • Silicon Valley, CA -- previously in Ann Arbor, MI
  • Liked: 545
  • Likes Given: 221
<snip>

arnezami I've been talking to michaelni a lot today in learning how to do this process and I think your concept of "end markers" is off. Every block has a pre-defined size which is determined by its format. When a block is mis-interpreted then the size automatically becomes wrong and the block "eats" extra bits from the next block or misses bits from its block shoving them into the next block. Occasionally these errors cancel each other out (because there is a small finite number of block types and sizes) and the streaming is "resynced" to the proper block ordering so you get a string of correct blocks. You often don't see this resync because after its resynced the blocks are still referencing all the previous corrupted blocks. This is why after you fix one block suddenly lots more blocks become readable IMO.

If michaelni could comment though that would probably clear more things up.

Edit: The key difference that matters from your explanation is that changing the start position of a block can _increase_ or _decrease_ how far away the stop position is for that block because of how the block is interpreted.
« Last Edit: 05/05/2014 05:51 AM by mlindner »
Internal combustion engine in space. It's just a Bad Idea.TM - Robotbeat

Offline grythumn

  • Full Member
  • **
  • Posts: 210
  • Liked: 120
  • Likes Given: 138
Best image so far of Frame 9.

-Bob


Offline IRobot

  • Full Member
  • ****
  • Posts: 1246
  • Portugal & Germany
  • Liked: 247
  • Likes Given: 225
I wonder if we reach a point from where some data alignment becomes impossible to correctly to determine. In that case, would it be possible to generate a couple of hundred of thousands of combinations, generate a result image for each, put them on a web server and allow the community to evaluate them visually?

Offline mlindner

  • Software Engineer
  • Full Member
  • ****
  • Posts: 1953
  • Space Capitalist
  • Silicon Valley, CA -- previously in Ann Arbor, MI
  • Liked: 545
  • Likes Given: 221
I wonder if we reach a point from where some data alignment becomes impossible to correctly to determine. In that case, would it be possible to generate a couple of hundred of thousands of combinations, generate a result image for each, put them on a web server and allow the community to evaluate them visually?

I've actually been doing exactly that myself. You can throw out some settings because there are errors, but lots are "valid" data wise but are still garbage image-wise. I just have a text editor open with the setting in question, a terminal open so i can hit up arrow key (last command) and enter to run it, and have it open an image viewer with the new image. So i have a feedback time of about 5 seconds per value change. I've been thinking of turning it into a script that others could run.
« Last Edit: 05/05/2014 05:39 AM by mlindner »
Internal combustion engine in space. It's just a Bad Idea.TM - Robotbeat

Offline mlindner

  • Software Engineer
  • Full Member
  • ****
  • Posts: 1953
  • Space Capitalist
  • Silicon Valley, CA -- previously in Ann Arbor, MI
  • Liked: 545
  • Likes Given: 221
Here's all my best iframe8's this far.

Edit: And just a note, I've only been working on the middle section thus far which is why the bottom section stays garbage. Also, the settings used are in the filenames, drop the iframe8- section and change the -'s to :'s.

Edit2: Working on these videos has wanted to make me go find the guy who wrote their rocket video encoder and shake them. Why the hell do they interlace their video and then re-encode the interlacing? It increases the bandwidth for their videos a crap ton by doing that which allows for higher error rate. Further, they didn't add all the stuff built into the codec that anyone streaming video uses to ensure error correction. It's almost like the cameras were slapped on as an afterthought purely for PR. It was #1 on my list if I ever got an internship at SpaceX would be to look into their codebase, look up who did the commits on this code and go bug them about it.
« Last Edit: 05/05/2014 06:15 AM by mlindner »
Internal combustion engine in space. It's just a Bad Idea.TM - Robotbeat

Offline arnezami

  • Full Member
  • **
  • Posts: 282
  • Liked: 262
  • Likes Given: 345
<snip>

arnezami I've been talking to michaelni a lot today in learning how to do this process and I think your concept of "end markers" is off. Every block has a pre-defined size which is determined by its format. When a block is mis-interpreted then the size automatically becomes wrong and the block "eats" extra bits from the next block or misses bits from its block shoving them into the next block. Occasionally these errors cancel each other out (because there is a small finite number of block types and sizes) and the streaming is "resynced" to the proper block ordering so you get a string of correct blocks. You often don't see this resync because after its resynced the blocks are still referencing all the previous corrupted blocks. This is why after you fix one block suddenly lots more blocks become readable IMO.

If michaelni could comment though that would probably clear more things up.

Edit: The key difference that matters from your explanation is that changing the start position of a block can _increase_ or _decrease_ how far away the stop position is for that block because of how the block is interpreted.
Very interesting! If that's true than that would probably help greatly in (semi-)automating this process! Because I had almost given up on that. So I'm very interested in an explanation of how this actually works.

Regards,

arnezami

[edit] When I look at my log though I almost do not see any of the same length of bitsizes per block. So I don't think macroblocks have any sort of fixed sizes.
« Last Edit: 05/05/2014 06:25 AM by arnezami »

Offline mlindner

  • Software Engineer
  • Full Member
  • ****
  • Posts: 1953
  • Space Capitalist
  • Silicon Valley, CA -- previously in Ann Arbor, MI
  • Liked: 545
  • Likes Given: 221
<snip>

arnezami I've been talking to michaelni a lot today in learning how to do this process and I think your concept of "end markers" is off. Every block has a pre-defined size which is determined by its format. When a block is mis-interpreted then the size automatically becomes wrong and the block "eats" extra bits from the next block or misses bits from its block shoving them into the next block. Occasionally these errors cancel each other out (because there is a small finite number of block types and sizes) and the streaming is "resynced" to the proper block ordering so you get a string of correct blocks. You often don't see this resync because after its resynced the blocks are still referencing all the previous corrupted blocks. This is why after you fix one block suddenly lots more blocks become readable IMO.

If michaelni could comment though that would probably clear more things up.

Edit: The key difference that matters from your explanation is that changing the start position of a block can _increase_ or _decrease_ how far away the stop position is for that block because of how the block is interpreted.
Very interesting! If that's true than that would probably help greatly in (semi-)automating this process! Because I had almost given up on that. So I'm very interested in an explanation of how this actually works.

Regards,

arnezami

[edit] When I look at my log though I almost do not see any of the same length of bitsizes per block. So I don't think macroblocks have any sort of fixed sizes.

They're content-based sizes I think. Maybe I'm wrong. When michaelni gets up he can answer.

Just as a note, in my (or michaelni's repo) version of your code, re-compile while uncommenting the "//#define TRACE" variable in libavcodec/get_bits.h. It prints a bunch of info about the contents of every block as it decodes it (run with your debug option on).
« Last Edit: 05/05/2014 06:42 AM by mlindner »
Internal combustion engine in space. It's just a Bad Idea.TM - Robotbeat

Offline mlindner

  • Software Engineer
  • Full Member
  • ****
  • Posts: 1953
  • Space Capitalist
  • Silicon Valley, CA -- previously in Ann Arbor, MI
  • Liked: 545
  • Likes Given: 221
I need some "truth" data. Do we have any camera views from any other video that shows what the lower portion of the image (the rocket itself) looks like at all? I'm trying to figure out something remarkable about it so I can tell one block from another.

I need to order a bunch of puzzle pieces. Each has numbers on the bottom that need to be in order, they are perfectly square, varying flat white and there are either too few pieces or too many pieces and I need to either make some new ones or throw some away.
Internal combustion engine in space. It's just a Bad Idea.TM - Robotbeat

Offline Digitalchromakey

  • Member
  • Posts: 23
  • Phuket
  • Liked: 5
  • Likes Given: 2
<snip>

arnezami I've been talking to michaelni a lot today in learning how to do this process and I think your concept of "end markers" is off. Every block has a pre-defined size which is determined by its format. When a block is mis-interpreted then the size automatically becomes wrong and the block "eats" extra bits from the next block or misses bits from its block shoving them into the next block. Occasionally these errors cancel each other out (because there is a small finite number of block types and sizes) and the streaming is "resynced" to the proper block ordering so you get a string of correct blocks. You often don't see this resync because after its resynced the blocks are still referencing all the previous corrupted blocks. This is why after you fix one block suddenly lots more blocks become readable IMO.

If michaelni could comment though that would probably clear more things up.

Edit: The key difference that matters from your explanation is that changing the start position of a block can _increase_ or _decrease_ how far away the stop position is for that block because of how the block is interpreted.
Very interesting! If that's true than that would probably help greatly in (semi-)automating this process! Because I had almost given up on that. So I'm very interested in an explanation of how this actually works.

Regards,

arnezami

[edit] When I look at my log though I almost do not see any of the same length of bitsizes per block. So I don't think macroblocks have any sort of fixed sizes.
The MPEG 4 spec has the algorithm that allows you to parse the elementary visual bit stream, which can be of varying bit length, however the sequence_end_code x000001B7 is always byte aligned with defined stuffing bit codes preceding the end code depending on the number of bits required.

The sequence start codes are also similarly byte aligned.

Quote
Bits to be stuffed Stuffing Codeword
1 0
2 01
3 011
4 0111
5 01111
6 011111
7 0111111
8 01111111

Offline mlindner

  • Software Engineer
  • Full Member
  • ****
  • Posts: 1953
  • Space Capitalist
  • Silicon Valley, CA -- previously in Ann Arbor, MI
  • Liked: 545
  • Likes Given: 221
<snip>

arnezami I've been talking to michaelni a lot today in learning how to do this process and I think your concept of "end markers" is off. Every block has a pre-defined size which is determined by its format. When a block is mis-interpreted then the size automatically becomes wrong and the block "eats" extra bits from the next block or misses bits from its block shoving them into the next block. Occasionally these errors cancel each other out (because there is a small finite number of block types and sizes) and the streaming is "resynced" to the proper block ordering so you get a string of correct blocks. You often don't see this resync because after its resynced the blocks are still referencing all the previous corrupted blocks. This is why after you fix one block suddenly lots more blocks become readable IMO.

If michaelni could comment though that would probably clear more things up.

Edit: The key difference that matters from your explanation is that changing the start position of a block can _increase_ or _decrease_ how far away the stop position is for that block because of how the block is interpreted.
Very interesting! If that's true than that would probably help greatly in (semi-)automating this process! Because I had almost given up on that. So I'm very interested in an explanation of how this actually works.

Regards,

arnezami

[edit] When I look at my log though I almost do not see any of the same length of bitsizes per block. So I don't think macroblocks have any sort of fixed sizes.
The MPEG 4 spec has the algorithm that allows you to parse the elementary visual bit stream, which can be of varying bit length, however the sequence_end_code x000001B7 is always byte aligned with defined stuffing bit codes preceding the end code depending on the number of bits required.

The sequence start codes are also similarly byte aligned.

Quote
Bits to be stuffed Stuffing Codeword
1 0
2 01
3 011
4 0111
5 01111
6 011111
7 0111111
8 01111111

A link to the spec section in question?
Internal combustion engine in space. It's just a Bad Idea.TM - Robotbeat

Offline Syrinx

  • Member
  • Posts: 30
  • San Carlos, CA, USA
  • Liked: 46
  • Likes Given: 27
I'm anxious to assist any way that I can.  I've got the custom ffmpeg built and installed (from the github link).  I'm a good engineer and programmer but I know next to nothing about MPEG or graphics in general.

I guess I'm confused about what I'm supposed to do now.  What do I give ffmpeg and what should I expect to get out?  Do I need to download the video or iframes from somewhere?  If so, I don't want to repair what you guys have already repaired.  How are you guys staying in sync?  Is there any coordination going on?

Would it be more productive for me to wait until I can assist in a more brute force fashion?  By that I mean using my eyeballs rather than my engineering skills.

Offline mlindner

  • Software Engineer
  • Full Member
  • ****
  • Posts: 1953
  • Space Capitalist
  • Silicon Valley, CA -- previously in Ann Arbor, MI
  • Liked: 545
  • Likes Given: 221
I'm anxious to assist any way that I can.  I've got the custom ffmpeg built and installed (from the github link).  I'm a good engineer and programmer but I know next to nothing about MPEG or graphics in general.

I guess I'm confused about what I'm supposed to do now.  What do I give ffmpeg and what should I expect to get out?  Do I need to download the video or iframes from somewhere?  If so, I don't want to repair what you guys have already repaired.  How are you guys staying in sync?  Is there any coordination going on?

Would it be more productive for me to wait until I can assist in a more brute force fashion?  By that I mean using my eyeballs rather than my engineering skills.

Download the iframes linked previously. Then use the command that was listed to generate .png from that. Then modify the -mmb (the hard part) in various ways to make the image better. Format is listed in previous post as well: X, Y, block_start. Start by using an mmb that one of us mentioned and the given iframe and make sure you get the same image, then start manipulating those values to start teaching yourself what its doing.

./ffmpeg -mmb mmb_stuff_here -debug mb_pos_size -err_detect ignore_err -s:0 704:480 -i path/to/iframe/file/here -f image2 img.png
To see the original iframe just remove the -mmb option.

No coordination other than a couple of PMs between me and arnezami and this forum. I was also talking directly to michaelni on IRC in the #ffmpeg channel on freenode, but that was mostly to ask questions to learn about the format. Just assume that if no one's mentioned their work on a frame then no one has started on it yet. I'm personally stuck on iframe8 right now as I can't figure out the mess at the bottom. It's 3:40 AM here so I'll be heading to bed shortly, but I'll be resuming when I get up. No work or school tomorrow/today (Monday).

If enough people want to coordinate to work on it by tomorrow then I'll probably set up an IRC channel and we can coordinate/teach each other there live.
« Last Edit: 05/05/2014 07:42 AM by mlindner »
Internal combustion engine in space. It's just a Bad Idea.TM - Robotbeat

Offline Syrinx

  • Member
  • Posts: 30
  • San Carlos, CA, USA
  • Liked: 46
  • Likes Given: 27
OK, thanks for the info.

The iframes can be downloaded from reply #89 in this thread.

I downloaded iframe 11, ran this:

ffmpeg -mmb 0:0:10020 -debug mb_pos_size -err_detect ignore_err -i iframes/iframe_11.mpg4-img -f image2 img-11.png

and got the same image that had been posted earlier.

Since there are only 12 iframes, does that mean there are only 12 images in the entire video?

Offline Jarnis

  • Full Member
  • ****
  • Posts: 1119
  • Liked: 567
  • Likes Given: 156
I need some "truth" data. Do we have any camera views from any other video that shows what the lower portion of the image (the rocket itself) looks like at all? I'm trying to figure out something remarkable about it so I can tell one block from another.

I need to order a bunch of puzzle pieces. Each has numbers on the bottom that need to be in order, they are perfectly square, varying flat white and there are either too few pieces or too many pieces and I need to either make some new ones or throw some away.

There are images from the rocketcam in this thread

http://forum.nasaspaceflight.com/index.php?topic=34502.105

However, I think these are actually taken from 2nd stage (higher up in the stack, slightly different angle) and a camera on the 1st stage was used only for this landing footage, so the image is not entirely same. You'd still get some idea as to what the rocket body looks like.

I do agree that it would probably help if SpaceX would release at least a couple of frames from the 1st stage camera from higher up. Reportedly they have good video of the braking burns but chose not to release it, perhaps to keep their secret sauce secret.


Offline Jakusb

  • Full Member
  • ****
  • Posts: 760
  • NL
  • Liked: 542
  • Likes Given: 245
I need some "truth" data. Do we have any camera views from any other video that shows what the lower portion of the image (the rocket itself) looks like at all? I'm trying to figure out something remarkable about it so I can tell one block from another.

I need to order a bunch of puzzle pieces. Each has numbers on the bottom that need to be in order, they are perfectly square, varying flat white and there are either too few pieces or too many pieces and I need to either make some new ones or throw some away.

There are images from the rocketcam in this thread

http://forum.nasaspaceflight.com/index.php?topic=34502.105

However, I think these are actually taken from 2nd stage (higher up in the stack, slightly different angle) and a camera on the 1st stage was used only for this landing footage, so the image is not entirely same. You'd still get some idea as to what the rocket body looks like.

I do agree that it would probably help if SpaceX would release at least a couple of frames from the 1st stage camera from higher up. Reportedly they have good video of the braking burns but chose not to release it, perhaps to keep their secret sauce secret.

Maybe something for Chris to pitch in? :)

Edit/Lar: I PMed Chris. If you want to make sure Chris sees it, PMing him is a better approach than just mentioning his name in a post. He's godlike but not omniescent :)
« Last Edit: 05/05/2014 11:24 AM by Lar »

Offline wronkiew

  • Full Member
  • *
  • Posts: 186
  • 34.502327, -116.971697
  • Liked: 105
  • Likes Given: 121
I think I got a bit more of iframe 8, hard to tell what the correct alignment is.

ffmpeg -mmb 07:14:17233,14:14:57111,08:15:17233,09:15:63546,19:16:17233,21:16:73568,08:21:17233,00:22:111664 -debug mb_pos_size -err_detect ignore_err -s:0 704:480 -i iframe_8.mpg4-img -f image2 img-%03d.png

Offline Digitalchromakey

  • Member
  • Posts: 23
  • Phuket
  • Liked: 5
  • Likes Given: 2
<snip>

arnezami I've been talking to michaelni a lot today in learning how to do this process and I think your concept of "end markers" is off. Every block has a pre-defined size which is determined by its format. When a block is mis-interpreted then the size automatically becomes wrong and the block "eats" extra bits from the next block or misses bits from its block shoving them into the next block. Occasionally these errors cancel each other out (because there is a small finite number of block types and sizes) and the streaming is "resynced" to the proper block ordering so you get a string of correct blocks. You often don't see this resync because after its resynced the blocks are still referencing all the previous corrupted blocks. This is why after you fix one block suddenly lots more blocks become readable IMO.

If michaelni could comment though that would probably clear more things up.

Edit: The key difference that matters from your explanation is that changing the start position of a block can _increase_ or _decrease_ how far away the stop position is for that block because of how the block is interpreted.
Very interesting! If that's true than that would probably help greatly in (semi-)automating this process! Because I had almost given up on that. So I'm very interested in an explanation of how this actually works.

Regards,

arnezami

[edit] When I look at my log though I almost do not see any of the same length of bitsizes per block. So I don't think macroblocks have any sort of fixed sizes.
The MPEG 4 spec has the algorithm that allows you to parse the elementary visual bit stream, which can be of varying bit length, however the sequence_end_code x000001B7 is always byte aligned with defined stuffing bit codes preceding the end code depending on the number of bits required.

The sequence start codes are also similarly byte aligned.

Quote
Bits to be stuffed Stuffing Codeword
1 0
2 01
3 011
4 0111
5 01111
6 011111
7 0111111
8 01111111

A link to the spec section in question?
ISO/IEC 144696-2 Section 6 Video Bitstream Syntax and Semantics.

Offline grythumn

  • Full Member
  • **
  • Posts: 210
  • Liked: 120
  • Likes Given: 138
Some more notes:

The macro blocks are 16x16. Sometimes you want to replace a block that isn't throwing an error because it messes up the image, and that helps work out what number to mess with.

Work sequentially, if possible... changing stuff early in the picture can cause green blocks to reappear in what was junky, but clear, picture.

Some green blocks just won't go away.

All I've gotten from frame 4 is the top of the timestamp, at position 119316:

-Bob

Tags: