Author Topic: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread  (Read 954482 times)

Offline mlindner

  • Software Engineer
  • Full Member
  • ****
  • Posts: 1980
  • Space Capitalist
  • Silicon Valley, CA -- previously in Ann Arbor, MI
  • Liked: 624
  • Likes Given: 239
It's actually a lot easier than that. If no one else does it by Friday Eastern time then I'll be modifying ffmpeg to add a frame specifier to the mmb options. Then we simply pile all the mmb options into one long entry and run ffmpeg with it. viola, produced video. This would also allow us to try tweaking some p-frames if we so wish.

Producing a video now with what we have won't be very attractive yet. We need to fine tune a lot of the frames still. I'm doing that now for iframe8, which should hopefully look really really good. This means doing things like whenever you use -1 and replace the block, fine tune all the luma and chroma settings for it to have the least effect on surrounding blocks.

So you're saying at this point, ffmpeg doesn't have the ability to re-encode with replacement iframes?

You might be able to manually have it use those iframes and re-encode it that way. I'm not sure how to use ffmpeg to do so. We haven't been producing replacement iframes anyway, only modifications to iframes that need to be applied every time.
Internal combustion engine in space. It's just a Bad Idea.TM - Robotbeat

Offline Sohl

  • Full Member
  • **
  • Posts: 241
  • Liked: 77
  • Likes Given: 215
Excellent work people!  I agree the historical significance of this video makes restoration efforts worthwhile even if the Orbcom flight provides a much cleaner landing video.  I'd probably enjoy this bit-hacking myself if I wasn't up to my neck in my day-job programming work!   :-[

Offline John_L

You might be able to manually have it use those iframes and re-encode it that way. I'm not sure how to use ffmpeg to do so. We haven't been producing replacement iframes anyway, only modifications to iframes that need to be applied every time.

Right, well, I'm nowhere near an expert at mpeg encoding, but it was my assumption that once the iframes were done, we'd re-encode the other frames using the iFrames as a reference, or would reinserting the fixed iframes automatically  fix the other frames (because they're now referencing the fixed iframes), which would mean we would just need a way to reinsert the iframes into the video.  Luckily, we have michaelni onboard helping out... nice to have the curator of ffmpeg onboard... ;)
« Last Edit: 05/07/2014 06:45 PM by John_L »

Online grythumn

  • Full Member
  • **
  • Posts: 211
  • Liked: 124
  • Likes Given: 145
Frame 9, cleaned up some single-bit errors up top.

20:04:-1,24:04:16023,X:16410:01,16:5:-1,17:05:18692,X:19903:1,X:27244:1,
1:15:-1,3:16:79127,41:19:-1,8:20:96220,15:04:16023

-Bob

Offline wronkiew

  • Full Member
  • *
  • Posts: 186
  • 34.502327, -116.971697
  • Liked: 105
  • Likes Given: 124
Frame 9, cleaned up some single-bit errors up top.

20:04:-1,24:04:16023,X:16410:01,16:5:-1,17:05:18692,X:19903:1,X:27244:1,
1:15:-1,3:16:79127,41:19:-1,8:20:96220,15:04:16023

-Bob

Nice work! The mixed up part of the right leg is likely also a bit error.

Online grythumn

  • Full Member
  • **
  • Posts: 211
  • Liked: 124
  • Likes Given: 145
Technique: Find suspect macroblock. Isolate bit range. Plug them into the for loop. Put the string ",X,$i:1," in the middle of your MMB string, where it goes from a bit range point point of view. Run the loop, wait a bit, flip through the resulting pictures. Can usually narrow it down to 4 or 5. If nothing works, try changing the macroblock to the left, or nulling out the macro block... if it's a multibit error, it's a lot more permutations to try to figure out.

For example, my last run:
for i in {32260..32407} ; do ./ffmpeg -debug mb_pos_size -err_detect ignore_err -s:0 704:480 -mmb 20:04:-1,24:04:16023,X:16410:01,16:5:-1,17:05:18692,X:19903:1, X:27244:1,X:$i:1,1:15:-1,3:16:79127,41:19:-1, 8:20:96220,15:04:16023 -i iframe_9.mpg4-img -f image2 img-%03d-$i.png 2> /dev/null ; done

-Bob

Offline wronkiew

  • Full Member
  • *
  • Posts: 186
  • 34.502327, -116.971697
  • Liked: 105
  • Likes Given: 124
Technique: Find suspect macroblock. Isolate bit range. Plug them into the for loop. Put the string ",X,$i:1," in the middle of your MMB string, where it goes from a bit range point point of view. Run the loop, wait a bit, flip through the resulting pictures. Can usually narrow it down to 4 or 5. If nothing works, try changing the macroblock to the left, or nulling out the macro block... if it's a multibit error, it's a lot more permutations to try to figure out.

For example, my last run:
for i in {32260..32407} ; do ./ffmpeg -debug mb_pos_size -err_detect ignore_err -s:0 704:480 -mmb 20:04:-1,24:04:16023,X:16410:01,16:5:-1,17:05:18692,X:19903:1, X:27244:1,X:$i:1,1:15:-1,3:16:79127,41:19:-1, 8:20:96220,15:04:16023 -i iframe_9.mpg4-img -f image2 img-%03d-$i.png 2> /dev/null ; done

-Bob

If you start with an error-free image, you can also eliminate any bit-flips that introduce errors by grepping through the output. I have a script that does just that.

Online dorkmo

  • Full Member
  • ****
  • Posts: 690
  • Liked: 318
  • Likes Given: 819
i made a super basic spreadsheet that just concatenates the x:y:bitpos. i was just guna use it for me but heres the link. https://docs.google.com/spreadsheets/d/1sP5lyTbC3ICyca03ZJlWwWaeIn_H6MSeqcp2ACT1HOQ/edit?usp=sharing

Offline mvpel

  • Full Member
  • ****
  • Posts: 1124
  • New Hampshire
  • Liked: 1295
  • Likes Given: 1686
The University of Wisconsin has a very large HTCondor pool which would probably be very useful to run through this processing. I'll reach out and see if there's anyone over there who can leverage their tens of thousands of available CPU cores to crank out macroblock iterations.
"Ugly programs are like ugly suspension bridges: they're much more liable to collapse than pretty ones, because the way humans (especially engineer-humans) perceive beauty is intimately related to our ability to process and understand complexity. A language that makes it hard to write elegant code makes it hard to write good code." - Eric S. Raymond

Offline mlindner

  • Software Engineer
  • Full Member
  • ****
  • Posts: 1980
  • Space Capitalist
  • Silicon Valley, CA -- previously in Ann Arbor, MI
  • Liked: 624
  • Likes Given: 239
Technique: Find suspect macroblock. Isolate bit range. Plug them into the for loop. Put the string ",X,$i:1," in the middle of your MMB string, where it goes from a bit range point point of view. Run the loop, wait a bit, flip through the resulting pictures. Can usually narrow it down to 4 or 5. If nothing works, try changing the macroblock to the left, or nulling out the macro block... if it's a multibit error, it's a lot more permutations to try to figure out.

For example, my last run:
for i in {32260..32407} ; do ./ffmpeg -debug mb_pos_size -err_detect ignore_err -s:0 704:480 -mmb 20:04:-1,24:04:16023,X:16410:01,16:5:-1,17:05:18692,X:19903:1, X:27244:1,X:$i:1,1:15:-1,3:16:79127,41:19:-1, 8:20:96220,15:04:16023 -i iframe_9.mpg4-img -f image2 img-%03d-$i.png 2> /dev/null ; done

-Bob

Oh that's an awesome idea. Hadn't thought of doing it for bit flipping. I was already doing that for moving start locations of blocks around.
Internal combustion engine in space. It's just a Bad Idea.TM - Robotbeat

Offline CraigLieb

  • Full Member
  • ****
  • Posts: 859
  • Dallas Fort Worth
  • Liked: 779
  • Likes Given: 974
Not really able to follow the technical process being executed, but I have been putting the most recent IFrame images posted into a powerpoint (using it like a flip book). One thing that jumps out is some artifacts that are at the top center and offset to the left of each of the frames. Maybe they can be used like a key to lock in some image locations like the time stamp at the bottom on currently hard interpret frames.

I have placed a few of the frames top strips next to each other here so you can see what I mean.
« Last Edit: 05/07/2014 08:05 PM by CraigLieb »
Colonize Mars!

Offline mlindner

  • Software Engineer
  • Full Member
  • ****
  • Posts: 1980
  • Space Capitalist
  • Silicon Valley, CA -- previously in Ann Arbor, MI
  • Liked: 624
  • Likes Given: 239
Not really able to follow the technical process being executed, but I have been putting the most recent IFrame images posted into a powerpoint (using it like a flip book). One thing that jumps out is some artifacts that are at the top center and offset to the left of each of the frames. They may be like a key that can be used to lock in some image locations like the time stamp at the bottom. I have captured a few of the frames just top strips next to each other.

Yay for dust and dirt on the lens.
Internal combustion engine in space. It's just a Bad Idea.TM - Robotbeat

Offline arnezami

  • Full Member
  • **
  • Posts: 284
  • Liked: 262
  • Likes Given: 346
Hi guys,

I would like to explain what we are actually doing. Because some of you might not completely grasp it. And I think I can best explain it by an illustration. Otherwise it gets too technical too quickly. I will do it in terms of rockets. ;)

--

Imagine a book. A thick book. It's a book containing very detailed technical instructions on how to build a new type of rocket.  It contains 13 chapters. One for each rocket part. Each chapter consists of an introductory paragraph explaining what the part is, how it should look like and how it works. None of the following paragraphs can be understood if this paragraph has not been read. The following paragraps are highly dependent on eachother: if you haven't read them in sequence they will be meaningless since they always refer heavely to the previous one to understand.

All sentences are written in a foreign language. And all punctuation is removed. Also all captitals are made lower case. The letters in the words are coded: each letter can only be decoded if you know the previous letter. For example this code: when reading a letter add the letternumber (a=1, b=2 etc) of the previous letter to this one to get the real letter. All whitespaces are removed. So each paragraph is just a string of seemingly random letters.

Ok. Now you're thinking: "This is insane. Who would read such a book?"

But this book was never intended to be read by a human being! It was supposed to be read by a machine that would also automatically build the rocket parts. Here is the real problem though: the printing machine that printed the book had a major error: it sometimes printed the wrong letter! So the book has thousands of random errors in it! Unsuprisingly when the book is now given to the rocket-part-building-machine it says: "Error". And that's basicly it.

Ok it turns out that the rocket-part-building-machine made a few incomplete and defective drawings of parts before giving the error. Mostly useless. So it seems that this is impossible to fix. Trying to figure out which letters are randomized by hand is insanely hard and time consuming.

What we did though was the following: instead of fixing the book we changed the machine so it would (1) give us better errors and (2) allow us to overrule certain instructions. We fed the machine with the first paragraph of a chapter and it started to draw the part. When we thought it went wrong we gave it the instruction to skip that portion of the drawing. Because apparently it had read a wrongly printed letter.

Of course we did not know how far it had to skip. So we guessed several times and hoped it would start drawing something sane again. And each time it was drawing something that looked sensible we knew it was reading uncompromised letters. We got an understanding which parts of the paragraph were wrongly printed and then mostly avoided them. By doing this many times and collaborating with several people we have now been able to let the machine draw several almost complete and fairly accurate parts. We do not know yet if it is going to be possible to let the machine read the other paragraphs in the chapters (they are much harder). But getting these drawings is already very satisfying.

--

I hope this illustration helps you (who might not understand the deeply technical stuff) to understand what the problem really is. How insane it is. And I think it's a quite good representation of what is actually the case.

Regards,

arnezami

---

PS. Below is a translation from the illustration into the real problem.

- Book: Video

- 13 chapters: 13 parts of the video

- First paragraph of each chapter: I-frame (independent picture)

- Other paragraphs: P- and B-frames (all dependent on each other)

- Foreign language: the language of video decoding (things like "discrete cosine transform", "motion vectors", "chrominance and luminance blocks" etc)

- Removal of punctuation and capitals: there are essentially no clear "markers" in the bitstream: no marker to detect the beginning of a horizonal line in the picture.

- Words: macroblocks (16 x 16 pixel blocks)

- Letter coding and removal of whitespaces: the coding of the bits is such that you need every previous bit to know what the current one means withing a word. The words are also variable in length. There is essentially no clear marker between each macroblock.

- Printing errors: random bits that were flipped due to bad reception

- Rocket-part-building-machine: the video decoder (the open source ffmpeg in our case)

- Drawings: the pictures the video decoder makes after trying to decode a frame
 
The transportstream is not mentioned. But effectively many pages of the book were missing. We mostly fixed that: they were actually "hidden" and we discovered most of them.
« Last Edit: 05/07/2014 10:46 PM by arnezami »

Online AncientU

  • Senior Member
  • *****
  • Posts: 6217
  • Liked: 4008
  • Likes Given: 5545
Thanks, that is perfect even for those of us who don't build rockets!
Ever thought of writing users' manuals?
"If we shared everything [we are working on] people would think we are insane!"
-- SpaceX friend of mlindner

Offline Adaptation

  • Full Member
  • *
  • Posts: 160
  • Liked: 39
  • Likes Given: 38
 

The University of Wisconsin has a very large HTCondor pool which would probably be very useful to run through this processing. I'll reach out and see if there's anyone over there who can leverage their tens of thousands of available CPU cores to crank out macroblock iterations.

This could work, they could run it through an edge detection filter to find ones with good time stamp in the proper location and the dust artifacts on the lens then bias the rest of the results against having strong vertical or horizontal lines.

A little bit of evolutinary code could reduce the power needed drastically, simply did this change make it better or worse.  If it was better use it as the starting point for the next set of changes. 
« Last Edit: 05/07/2014 08:29 PM by Adaptation »

Offline moralec

The University of Wisconsin has a very large HTCondor pool which would probably be very useful to run through this processing. I'll reach out and see if there's anyone over there who can leverage their tens of thousands of available CPU cores to crank out macroblock iterations.

This could work, they could run it through an edge detection filter to find ones with good time stamp in the proper location then biase the rest of the results against having strong verticle or horisontal lines.

We could then load "candidate images" in a server and crowdsource it. (like tomnod)

Online Chris Bergin

Hi guys,

I would like to explain what we are actually doing. Because some of you might not completely grasp it. And I think I can best be explained it by an illustration. Otherwise it gets too technical too quickly. I will do it in terms of rockets. ;)

--

Imagine a book. A thick book. It's a book containing very detailed technical instructions on how to build a new type of rocket.  It contains 13 chapters. One for each rocket part. Each chapter consists of an introductory paragraph explaining what the part is, how it should look like and how it works. None of the following paragraphs can be understood if this paragraph has not been read. The following paragraps are highly dependent on eachother: if you haven't read them in sequence they will be meaningless since they always refer heavely to the previous one to understand.

All sentences are written in a foreign language. And all punctuation is removed. Also all captitals are made lower case. The letters in the words are coded: each letter can only be decoded if you know the previous letter. For example this code: when reading a letter add the letternumber (a=1, b=2 etc) of the previous letter from this one to get the real letter. All whitespaces are removed. So each paragraph is just a string of seemingly random letters.

Ok. Now you're thinking: "This is insane. Who would read such a book?"

But this book was never intended to be read by a human being! It was supposed to be read by a machine that would also automatically build the rocket parts. Here is the real problem though: the printing machine that printed the book had a major error: it sometimes printed the wrong letter! So the book has thousands of random errors in it! Unsuprisingly when the book is now given to the rocket-part-building-machine it says: "Error". And that's basicly it.

Ok it turnes out that the rocket-part-building-machine made a few incomplete and defective drawings of parts before giving the error. Mostly useless. So it seems that this is impossible to fix. Trying to figure out which letters are randomized by hand is insanely hard and time consuming.

What we did though was the following: instead of fixing the book we changed the machine so it would (1) give us better errors and (2) allow us to overrule certain instructions. We fed the machine with the first paragraph of a chapter and it started to draw the part. When we thought it went wrong we gave it the instruction to skip that portion of the drawing. Because apparently it had read a wrongly printed letter.

Of course we did not know how far it had to skip. So we guessed several times and hoped it would start drawing something sane again. And each time it was drawing something that looked sensible we knew it was reading uncompromised letters. We got an understanding which parts of the paragraph were wrongly printed and then mostly avoided them. By doing this many times and collaborating with several people we have now been able to let the machine draw several almost complete and fairly accurate parts. We do not know yet if it is going to be possible to let the machine read the other paragraphs in the chapters (they are much harder). But getting these drawings is already very satisfying.

--

I hope this illustration helps you (who might not understand the deeply technical stuff) to understand what the problem really is. How insane it is. And I think it's a quite good representation of what is actually the case.

Regards,

arnezami

---

PS. Below is a translation from the illustration into the real problem.

- Book: Video

- 13 chapters: 13 parts of the video

- First paragraph of each chapter: I-frame (independent picture)

- Other paragraphs: P- and B-frames (all dependent on each other)

- Foreign language: the language of video decoding (things like "discrete cosine transform", "motion vectors", "chrominance and luminance blocks" etc)

- Removal of punctuation and capitals: there are essentially no clear "markers" in the bitstream: no marker to detect the beginning of a horizonal line in the picture.

- Words: macroblocks (16 x 16 pixel blocks)

- Letter coding and removal of whitespaces: the coding of the bits is such that you need every previous bit to know what the current one means withing a word. The words are also variable in length. There is essentially no clear marker between each macroblock.

- Printing errors: random bits that were flipped due to bad reception

- Rocket-part-building-machine: the video decoder (the open source ffmpeg in our case)

- Drawings: the pictures the video decoder makes after trying to decode a frame
 
The transportstream is not mentioned. But effectively many pages of the book were missing. We mostly fixed that: they were actually "hidden" and we discovered most of them.


Such a really good post! :)

Offline John_L

And to continue the analogy... a new book will be written this Saturday....lol

Offline mlindner

  • Software Engineer
  • Full Member
  • ****
  • Posts: 1980
  • Space Capitalist
  • Silicon Valley, CA -- previously in Ann Arbor, MI
  • Liked: 624
  • Likes Given: 239
Technique: Find suspect macroblock. Isolate bit range. Plug them into the for loop. Put the string ",X,$i:1," in the middle of your MMB string, where it goes from a bit range point point of view. Run the loop, wait a bit, flip through the resulting pictures. Can usually narrow it down to 4 or 5. If nothing works, try changing the macroblock to the left, or nulling out the macro block... if it's a multibit error, it's a lot more permutations to try to figure out.

For example, my last run:
for i in {32260..32407} ; do ./ffmpeg -debug mb_pos_size -err_detect ignore_err -s:0 704:480 -mmb 20:04:-1,24:04:16023,X:16410:01,16:5:-1,17:05:18692,X:19903:1, X:27244:1,X:$i:1,1:15:-1,3:16:79127,41:19:-1, 8:20:96220,15:04:16023 -i iframe_9.mpg4-img -f image2 img-%03d-$i.png 2> /dev/null ; done

-Bob

One more note. X doesn't behave quite as you think. The bitmask is inverted. X:pos:1 actually flips the bit 8 positions away from pos. It reads the next 8 bits starting from pos with pos being the MSB of those 8 bits. If you want to flip just the target bit, do X:pos:80.
Internal combustion engine in space. It's just a Bad Idea.TM - Robotbeat

Offline R7

  • Propulsophile
  • Senior Member
  • *****
  • Posts: 2738
    • Don't worry.. we can still be fans of OSC and SNC
  • Liked: 950
  • Likes Given: 662
And to continue the analogy... a new book will be written this Saturday....lol

Let's hope they have changed the page layout to contain additional information to help detect sentence boundaries, errors and such. This macroblock necromancy goes only so far.  :)
AD·ASTRA·ASTRORVM·GRATIA

Tags: