Author Topic: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread  (Read 854385 times)

Offline Untribium

  • Member
  • Posts: 32
  • Switzerland
  • Liked: 32
  • Likes Given: 78
Found another bit flip in iframe 52: X:45038:80 relative to iframe52.mpg4-img improves the chroma in the bottom right corner. There are for sure a bunch of other bit flips that are causing the dark spot at bottom right, but it's difficult to find them. I have an idea for potentially automating this search though...maybe on the weekend.

Nice find! X:42196:08 gets rid of some of the dark spot. I'll update the wiki and keep looking for more :)

edit: attached image

edit2: Found another one! X:66234:08 fixes the white gradient thingy at the bottom right. Man, and I just updated the wiki ;)

edit3: And another one...minor color correction using X:50298:08. I should really stop updating the wiki ;)
« Last Edit: 05/22/2014 10:53 PM by Untribium »

Offline saliva_sweet

  • Full Member
  • ****
  • Posts: 508
  • Liked: 380
  • Likes Given: 1252
Some improvements to iframe 72. More sea and a bit of the good stuff. mmb is massive.

Offline Alkan

  • Member
  • Posts: 12
  • Arizona
  • Liked: 13
  • Likes Given: 1
It's awesome that you guys got tweeted by Elon himself!

https://twitter.com/elonmusk/status/469356919979655168

Offline morningdew76

  • Member
  • Posts: 11
  • McLean, VA
  • Liked: 12
  • Likes Given: 7
Found another bit flip in iframe 52: X:45038:80 relative to iframe52.mpg4-img improves the chroma in the bottom right corner. There are for sure a bunch of other bit flips that are causing the dark spot at bottom right, but it's difficult to find them. I have an idea for potentially automating this search though...maybe on the weekend.

Nice find! X:42196:08 gets rid of some of the dark spot. I'll update the wiki and keep looking for more :)

edit: attached image

edit2: Found another one! X:66234:08 fixes the white gradient thingy at the bottom right. Man, and I just updated the wiki ;)

edit3: And another one...minor color correction using X:50298:08. I should really stop updating the wiki ;)

Change 0:29:81738 to 10:28:80196 to get most of row 28.

I've been attempting to recover the top of the numbers with little success.. there are about 20 positions to put 09:28 that result in 10:28:80196, but no places to put 08:28 that result in a correct 10:28.  I haven't tried flipping bits in this section yet (it's about 4k).

Offline Untribium

  • Member
  • Posts: 32
  • Switzerland
  • Liked: 32
  • Likes Given: 78
Sweet! Yeah, the timestamp mmbs are huge :D not really surprising, but a pain to check for flips...I just found a one that fixes some stuff, but is "incompatible" with some I found earlier. Not sure whether it's correct or not...If I ignore (-1) the block it's in, the following blocks look exactly the same, but at the same time, the macroblock itself doesn't look quite right...oh well, I'll have to check it tomorrow :)

Offline SwissCheese

  • Full Member
  • *
  • Posts: 164
  • Liked: 249
  • Likes Given: 80
A little update of the landing sequence. I think I will rather post movies after this update, these gif are getting very big!

The sequence goes from frame 169 to frame 211. The missing frames are 178, 201, 202, 206 and 207.

Edit: somehow I cannot put the gif here, but you can see it there: http://spacexlanding.wikispaces.com/Progress+using+multi+frame+MMBs
« Last Edit: 05/23/2014 12:15 AM by SwissCheese »

Offline SwissCheese

  • Full Member
  • *
  • Posts: 164
  • Liked: 249
  • Likes Given: 80
Found another bit flip in iframe 52: X:45038:80 relative to iframe52.mpg4-img improves the chroma in the bottom right corner. There are for sure a bunch of other bit flips that are causing the dark spot at bottom right, but it's difficult to find them. I have an idea for potentially automating this search though...maybe on the weekend.

Nice find! X:42196:08 gets rid of some of the dark spot. I'll update the wiki and keep looking for more :)

edit: attached image

edit2: Found another one! X:66234:08 fixes the white gradient thingy at the bottom right. Man, and I just updated the wiki ;)

edit3: And another one...minor color correction using X:50298:08. I should really stop updating the wiki ;)

Very nice!

Could you rather put your updated MMB on this page? http://spacexlanding.wikispaces.com/Frames+52-71

Offline Untribium

  • Member
  • Posts: 32
  • Switzerland
  • Liked: 32
  • Likes Given: 78
Found another bit flip in iframe 52: X:45038:80 relative to iframe52.mpg4-img improves the chroma in the bottom right corner. There are for sure a bunch of other bit flips that are causing the dark spot at bottom right, but it's difficult to find them. I have an idea for potentially automating this search though...maybe on the weekend.

Nice find! X:42196:08 gets rid of some of the dark spot. I'll update the wiki and keep looking for more :)

edit: attached image

edit2: Found another one! X:66234:08 fixes the white gradient thingy at the bottom right. Man, and I just updated the wiki ;)

edit3: And another one...minor color correction using X:50298:08. I should really stop updating the wiki ;)

Very nice!

Could you rather put your updated MMB on this page? http://spacexlanding.wikispaces.com/Frames+52-71

Oh, I meant to do that but forgot about it :) the gif looks awesome, nice work! :)

Offline mhenderson

  • Member
  • Posts: 69
  • USA
  • Liked: 101
  • Likes Given: 18
Update to iFrame 92.

On the wiki, user mhenderson is replacing the following -mmb
for iFrame 92 by Untribium

PRIOR MMB
X:54419:00,X:53655:00,0:0:550,14:0:-2:0:0:0:0:12:-6,
2:5:14749,39:7:-1,18:9:27558,25:12:39009,
29:12:-1,17:13:42121,18:13:42526,10:14:-1,
18:14:47626,26:14:-2:0:0:0:0:8:-6,30:14:50837,
1:15:-1,5:15:51900,9:15:-1,21:15:53655,
27:16:61466,42:19:-1,43:19:83153,43:20:-1,
1:21:90585,11:22:-1,17:23:105394,22:24:-1,
23:24:111581,0:27:-1,25:27:125621,28:28:-1,
3:29:140154


With the following -mmb for iFrame 92 by mhenderson
NEW MMB
X:54419:00,X:53655:00,X:140154:0C,0:0:550,
14:0:-2:0:0:0:0:12:-6,25:0:2044,27:0:-2:0:0:0:0:12:-6,
0:1:3127,3:1:-2:0:0:0:0:12:-6,4:1:3405,
5:1:-2:0:0:0:0:12:-6,7:1:-2:0:0:0:0:12:-6,
14:1:-2:0:0:0:0:12:-6,24:1:-2:0:0:0:0:12:-6,25:1:5358,
31:1:5258,42:1:-2:0:0:0:0:12:-6,0:2:-2:0:0:0:0:12:-6,
26:2:-2:0:0:0:0:12:-6,0:3:8701,21:3:-2:0:0:0:0:12:-6,
22:3:-2:0:0:0:0:12:-6,0:4:-2:0:0:0:0:12:-6,
12:4:11713:0:0:0:0:1:0:63,22:4:-2:0:0:0:0:12:-6,
23:4:12333,24:4:-2:0:0:0:0:12:-6,30:4:14749,2:5:14749,
39:7:-1,18:9:27558,25:12:39009,29:12:-1,17:13:42121,
18:13:42526,10:14:-1,18:14:47626,22:14:49012,
23:14:49834,25:14:-2:0:0:0:0:-6:-6,
26:14:-2:0:0:0:0:5:-6,28:14:-2,30:14:50837,
1:15:-1,5:15:51900,9:15:-1,19:15:52863,21:15:53655,
16:16:58104,17:16:58324:-8:-8:-8:-5:0:0,
27:16:61466,42:19:-1,43:19:83153,43:20:-1,1:21:90585,
11:22:-1,17:23:105394,22:24:-1,23:24:111581,0:27:-1,
25:27:125621,0:28:-2:-90:-90:-90:-90:0:0,3:29:140154,
10:29:-2:-90:-90:-90:-90:0:0

These changes bring in more detail in the top five rows
and some detail around the legs. Also clean up the
clock bar at the bottom.

Offline moralec

Thank you for all the updates everyone!

Remember to post your improvements on the wiki every time, in order to keep the latest versions available for the ones that are working in your same frames!

It's amazing to see we are running 24/7: is really a pleasure to wake up in the morning, and be surprise on the updates from the last 6 or 7 hours!

Keep up with the great work! we are almost there!

Offline mhenderson

  • Member
  • Posts: 69
  • USA
  • Liked: 101
  • Likes Given: 18
iFrame 92 image attached

Offline Lee Jay

  • Elite Veteran
  • Global Moderator
  • Senior Member
  • *****
  • Posts: 6652
  • Liked: 936
  • Likes Given: 138
Amazing work, all.

A suggestion, if I might.

Once in a while, when you are producing videos or animations after repair, create a side-by-side with the original frames.  It will drive home the point of the work done to those watching much less casually than the rest of us.

Offline mhenderson

  • Member
  • Posts: 69
  • USA
  • Liked: 101
  • Likes Given: 18
iFrame 92 before and current

Offline meadows.st

  • Full Member
  • *
  • Posts: 151
  • Vancouver BC, Canada
  • Liked: 88
  • Likes Given: 3622
Wow you guys are making awesome progress!!!!

I know we are trying to replace bad mmb with good mmb. (1) So would it be useful to document the XY and POS of good mmb's? I am thinking if we can map out what we know to be good, figuring out what to replace the bad with may be easier.
Welcome to the forum PaulNY!

From what I understand your idea is to catalog good macro blocks in all the frames. Good as in decided by human beings not by some algorithm I assume.

This may be of some use. I'm not sure. (2) We sort of implicitly rely on our vision to detect good vs bad blocks. We also have a nice feature in the online tool that highlights all error-macroblocks in a frame. There are still plenty of non-error-blocks left that look "bad", but we don't know if they are easily repairable or are really bad. (3) We usually judge their "badness" by the destruction they cause to other blocks.

So depending on what you think 1. what the criteria of "bad" is, or even multiple classes of bad and 2. how you think we can deal with those kinds of "badness", your idea might have some value.

Maybe you can try to "fertilize" your ideas a little bit so it becomes more clear if we can use it. :)

Regards,

arnezami

PS. We refer to mmb as "modify macro block". I believe when you say mmb you actually ment to say "macro block".

Unfortunately work has kept me away for a few days, but let me try to explain my idea more.

Lets say I look at I frame 8. MB 1 (0:0) looks right and is blue water. If we can document the position of this MB in the frame, then lets say some MB (x,y) looks bad and we think it's water we could try replacing it with a known value of a good water MB instead of trying to guess. If we can document all the know good and bad MB values in the I frames, then we can then target our repairs more. I feel like when trying to make a repair the majority of the time I spend is guessing MB pos values or the pos to try a invert / flip.

Maybe I am missing something, but if this is where other people spend a lot of time to it would be great if we could create a way to collaborate on this more and optimize the process.

If we can't document known good or bad positions, maybe we could create a DB of recommended or non recommended pos values for each MB to try and optimize the process.  (4) For example if I try to fix / replace MB (X:Y) and try these 400 values that don't work. If I can document that I can save the next guy from trying the same thing.

Added bold/underlined numbers in quoted section to clarify the thoughts I am trying to express. (If this has already been discussed, please let me know and I will remove this post as I am out of my league here and have not had the time to really do my homework and create any proofs of concept yet; I am hoping that these thoughts might trigger a brilliant idea in someone else (even if I am completely off base with this) - I have attempted to follow the thread closely but I may have missed (or forgotten) some posts.)

1) Agreed (as stated before) but the issue is a) the inability to identify a "good" change from a "bad" change and b) the cascading effect of any change.

2) By saying "we ... rely on our vision ... to detect good/bad" I think arnezami means that the ffmpeg-extracted, resulting PNG (or GIF) file is inspected manually after every change to determine the degree of "correctness" of the change. Some changes will make discernible features appear (legs, lines, white-capped waves, the blessed dirt spots, etc) while other changes may remove/alter/"fix" obvious erroneous blocks in the resulting PNG (and thus in what will be the final video).

3) re: Judging "badness" by destruction to other blocks - I understand this to mean that newly resultant cascading failures will become apparent if a "really bad" change is made. I am thinking of a brute-force method (I am sure there are more elegant approaches) initially wherein we could create a reference, fixed integer size per pixel bitmap (or any another format that is a direct representation of each pixel and is essentially "fully" decoded and uncompressed) with a fixed pixel representation so we can compare the resulting file numerically and calculate the degree and number of changes that are created by a changed mb.  We know (reasonably well) that certain colors are not likely to be found in the resulting image and these colors can be detected and compared to a reference. My hypothesis is that we could then script a large number of mmb's and bitflips and numerically compare the resulting bitmaps to narrow down the images that should be inspected by a human.  The analogy I am thinking about is a brute force attack on a cipher (frame by frame) that uses a different key for each "character" and each subsequent "character" is influenced by the key used in each previous "character".  Our advantage (and I think theshadow27 and arnezami and mvpel (and others?) may have already been thinking along these lines) is that we know the ranges of colors that will be highly unlikely and we know large chunks of the clear-text reasonably well.  At the bitmap level, we can numerically calculate a delta from one mmb/bitflip to a reference bitmap.  Obviously the number of calculations increase exponentially due to the cascading errors and even the "known" parts of each image (other than the opaque dirt spots) experience changes from condensation, smoke, etc.

4) From my discussion in 3) above, what I am ultimately thinking about is for each mb(x,y) that looks "bad" and we try a change, that change results in a delta to the original bitmap (i.e. at the discrete pixel level) and a "bad" change would create lots of invalid colors and possibly lots of additional failed cascades and the "degree of goodness/badness" could be described as the number of improbable pixel values (or even the number of changed pixels) in the resulting bitmap. The problem with this approach is that a "good" change may also create a lot of changes until we get down to the final mb that cannot influence any other mb.  I would think that we could use the "prevent inheritance" flag for each mb to then calculate the resulting good/bad value of the change and store this info in a database for each mb (x,y)- obviously if any change results in a ffmpeg error, then the change is (likely?) bad, the second filter would be the comparison to the starting (or reference "best") bitmap.

Added: If this approach is meaningful then we could employ a large(r) army of human eyes to review the resulting candidate images to identify good/better/best/epic-fail images.  If we could script the approach to first target the mb's that result in ffmpeg errors then set the script to attack the remaining "bad but not technically error" mb's and filter out any changes that create new errors or result in improbable colors then hopefully we can significantly reduce the number of human views required and can split up the work to avoid duplication. (obviously this work would require some significant cloud storage (each uncompressed bitmap of 704x520 pixels would result in a >1.5MB filesize) so each frame might require a LOT of storage.
« Last Edit: 05/23/2014 05:44 AM by meadows.st »
A little rudder far from the rocks is a lot better than a lot of rudder close to the rocks. L. David Marquet

Offline Lourens

  • Full Member
  • *
  • Posts: 156
  • The Netherlands
  • Liked: 206
  • Likes Given: 304
We're thinking along similar lines I think. The real question is the objective function. The best reference bitmap would of course be the actual frame, which we don't have. Perhaps comparing colour histograms would work (using a good part that we have for reference, and perhaps the earth mover's distance to compare), and I've also been thinking about using the contrast along macroblock edges. We also have some known edges, so another idea is to run an edge detect on the frame and then compute the inner product with a separate image of known edges. On the other hand, there aren't that many edges, and when the legs are moving, we don't know where they are exactly, so maybe that doesn't work so well...

Offline arnezami

  • Full Member
  • **
  • Posts: 282
  • Liked: 262
  • Likes Given: 341
Hi guys,

Since this is the subject I've been spending a lot of time on thinking I will react too. But I havent got a lot of time right now. So I will probably be more elaborate later.

What this whole problem boils down to is three phases (per frame):

1) Extracting: Correcting the predictable streamheaders and I/P-frame headers. This way we (or actually the decoder) can extract all the video data without missing packets.

2) Positioning: Finding all the macroblock-data (that is: finding all bitpositions where each mb-data starts) and then assigning the right macroblocks to the these starting bitpositions.

3) Repairing: Correcting left-over DC-values errors either by bit flipping or by changing the DC-values after decoding. This should implcitly also fix "bad inheritance": where a macro inherents its DC base value from the wrong macroblock (that is: left or top).

I believe that for 2 there is an algorithm that will find all likely correct starting positions of macroblocks, but it will not be able correctly assign macroblocks to these bitpositions. I believe it would be helpful if we auto-generate all the likely starting positions for each frame and make these available as a static file to the online editor. This would allow the user to choose from good starting positions.

As for 3, I think there may be a way to correct DC-values somewhat automatically. However bitflipping is a better way when possible of course. Massively correcting DC-values should only be done if 1 and 2 has been done for a frame. If this works as I hope this will really give it a "finished look".

There is also the issue with the MV's for P-frames. I haven't looked into this too much.

I will go into more detail when I have more time ;).

Regards,

arnezami
« Last Edit: 05/23/2014 09:06 AM by arnezami »

Offline DragonRider

  • Member
  • Posts: 19
  • London
  • Liked: 9
  • Likes Given: 1
 I have to say, amazing work in this thread, I really did not think it was possible to recover a video like this.

Offline Sohl

  • Full Member
  • **
  • Posts: 238
  • Liked: 73
  • Likes Given: 209
I have to say, amazing work in this thread, I really did not think it was possible to recover a video like this.

Yes, the results of the work here greatly eclipse my expectations! Great job everyone.  :D I hope to join in in some way within a week or two.

Offline mhenderson

  • Member
  • Posts: 69
  • USA
  • Liked: 101
  • Likes Given: 18
@arnezami -

In the water, the typical "good" macroblock has a length of 45-100. It isn't too hard to find chains of 4 to 10 valid macroblocks. No error, length of 45-100.

In iFrame 92, for example, the first half dozen rows have an average length of 72. The average gives an estimated starting POS for each Col:Row block.

Fill the gaps between these short chains of good blocks with "estimated water color" Col:Row:-2:0:0:0:0:12:-6. (IMHO, this color should be standardized across the iFrames for consistency / smooth video.)

Badabing.

Similar logic applies to the rocket body.

Offline SwissCheese

  • Full Member
  • *
  • Posts: 164
  • Liked: 249
  • Likes Given: 80
I was thinking about the movie, could someone with some artistic talent (not my case :P) make 1-2 title images, and maybe also some credits images at the end (for example a few before/after images, printscreens of the wiki, web application, ...)? It should have the same format as the images, i.e. 704x480.

Edit: a first version of a movie with all recovered frames (the other are not used)
« Last Edit: 05/23/2014 03:45 PM by SwissCheese »

Tags: