You did really great! Those headers were totally mangled, but after your fixes your result file is pretty clean from a TS point of view.
I hope you don't mind but I've processed it a little more and fixed a couple of PCC discontinuities that were remaining. I've also removed the AF from any packets where either the AF is longer than the packet, or when the AF shows a wildly invalid PTS time. This is hopefully the correct thing to do, but please take a look and let me know what you think.
Question: when we use -mmb X:offset:pattern, are we flipping bits before or after the entropy decoding stage? Is this simply equivalent to flipping bits in the input file? Or is it essentially just a low-level way of editing the MB directly?
Quote from: moralec on 05/23/2014 08:33 pmLegs for days....http://spacexlanding.wikispaces.com/Frames+72-91Can you animate these frames? Legs are starting to open frames 85-90...
Legs for days....http://spacexlanding.wikispaces.com/Frames+72-91
Brute force bit flipper inside the decoder:Another idea is to do the bruteforce bitflipping inside the decoder. The main reason to do the brute forcing inside the code of the decoder is that it is likely to be much faster. Lets say you have four blocks of -1 (for example 15:12, 16:12, 17:12 and 18:12 in I-frame 150). To the left and right of it is are good macroblocks (14:12 at 45301 and 19:12 at 46628). You remove the four -1s so it will give an error again. Now you say to the decoder (in broteforce mode) try bitflipping 1-3 bits between the starting position of 15:12 (45472) and the starting position of 19:12 (46628) and each time you do this try to decode from 15:12 and see if you find 4 macroblocks and end at exactly at 19:12 (46628).
Quote from: Untribium on 05/24/2014 04:16 amQuote from: Req on 05/24/2014 02:24 amActually, it occurs to me that the straight up "ps" approach to tracking workers may have problems with the above script, because the workers are constantly starting and stopping ffmpeg, which ps is looking for.Another easy trick to track spawned workers is just to use PHP.To check workers you can use:while [ `ps aux | grep process_one\.php | grep -v grep | wc -l` -ge $threads ]And to spawn workers you can use:# spawn a worker(via php passthrough for easy thread tracking) on the current offsetphp /root/process_one.php $i &process_one.php looks like this:-snip-Again, the overhead php adds to spawning a process in this case is negligible because the workers will be running for a good while each and you're spawning thousands, not millions or billions.Looks like this: http://coldattic.info/shvedsky/pro/blogs/a-foo-walks-into-a-bar/posts/7 might be an option as well. Generate the addresses using seq and then pipe it to xargs which then runs a fixed number of ffmpeg processes at a time. I'll check it out tomorrow, should get some sleep first I am interested to see the results. I'm just so busy that my "recreation" basically involves this thread and a few other news sites at the moment. Maybe you have found something that will be very useful to my endeavors in the future, at least on one level of scale! Although I must admit, I do enjoy coding this type of thing.
Quote from: Req on 05/24/2014 02:24 amActually, it occurs to me that the straight up "ps" approach to tracking workers may have problems with the above script, because the workers are constantly starting and stopping ffmpeg, which ps is looking for.Another easy trick to track spawned workers is just to use PHP.To check workers you can use:while [ `ps aux | grep process_one\.php | grep -v grep | wc -l` -ge $threads ]And to spawn workers you can use:# spawn a worker(via php passthrough for easy thread tracking) on the current offsetphp /root/process_one.php $i &process_one.php looks like this:-snip-Again, the overhead php adds to spawning a process in this case is negligible because the workers will be running for a good while each and you're spawning thousands, not millions or billions.Looks like this: http://coldattic.info/shvedsky/pro/blogs/a-foo-walks-into-a-bar/posts/7 might be an option as well. Generate the addresses using seq and then pipe it to xargs which then runs a fixed number of ffmpeg processes at a time. I'll check it out tomorrow, should get some sleep first
Actually, it occurs to me that the straight up "ps" approach to tracking workers may have problems with the above script, because the workers are constantly starting and stopping ffmpeg, which ps is looking for.Another easy trick to track spawned workers is just to use PHP.To check workers you can use:while [ `ps aux | grep process_one\.php | grep -v grep | wc -l` -ge $threads ]And to spawn workers you can use:# spawn a worker(via php passthrough for easy thread tracking) on the current offsetphp /root/process_one.php $i &process_one.php looks like this:-snip-Again, the overhead php adds to spawning a process in this case is negligible because the workers will be running for a good while each and you're spawning thousands, not millions or billions.
seq 8800 9100 | xargs -n 1 -I BITPOS --max-procs=4 ffmpeg -threads 1 -debug mb_pos_size -err_detect ignore_err -s:0 704:480 -mmb ...
Why are we succeeding?I think there are two main reasons why we are succeeding in what was deemed to be impossible. The first is that we really went outside-the-box with our approach. The second is that our techniques are such that we can collaborate in our efforts.
OK, here's my fixed version of the part 15 TS file. As far as I can see there's a damaged P-frame starting in packet 108 (offset 0x4f50 bytes into the file), but it's damaged at the MPEG4 level and I'm only fixing TS-level things at the moment.I've set its AF to 7 bytes so that the MPEG4 data will be aligned properly, but I've not put a PCR in it because it's very damaged (and as far as I can see, ffmpeg doesn't seem to mind the lack of a PCR).It should be ready for MPEG4 experts to start playing with it - if anyone wants to correct the header of this packet, a new P-frame should hopefully emerge. Let us know how you go!
Edit: Ok I looked at your fixed part15 file. There were still many unfixed Ts-packets, most with the correct PID and CC but wrong flags set. The TS-Header of a data packet should really only look like 47 03 e8 1X, with 2 exceptions, the first and last of a frame which should be easily identified.
For the other "unfixed" packets I'd just left the erroneous Payload Unit Start Indicator bits in there. I agree with you that they should be removed - I had left them in there because (as far as I can see) they don't trip ffmpeg up, and if I'm in doubt I like to change as little as possible. I'm always worried about changing too much, it always feels nicer to make as small a change as I can.
There's a true and original state underneath all that corruption, and that's where we should aim as long as we can aim with confidence and precision.
This is just a proof of concept. I deciphered a few of the timestamps on the images and interpolated the rest. Due to interlacing and propagating errors, even if we get good mbs in the pframes the timestamps are still going to be gibberish most of the time. So, I propose we just redraw them. Any concerns about this?
Seems like a very good idea, with a few constraints:- they need to be correct- they are only applied at time of generating the movie file, so not to mess up de frame reconstruction process. In the end it is much better to have them fixed instead of reapplied.
For the final work product (if there is such a thing) I suggest a video that repeats the sequence several times interleaved with some text screens that explain who / what / why at a high level. 1) Begin with a brief paragraph about the circumstances and the challenge.2) Full video first pass, show the original video before restoration.3) Another paragraph, introduce the NSF forum and crowdsourced effort.4) A series of before / after shots of individual frames to indicate progress with captions highlighting tools used, round-the-globe contributions, people contributing from their own skillsets....5) Full video, showing a "true" restoration (no enhancements)6) Full video, show a side-by-side presentation. in the top left corner of screen (20%), show a graphic animation of the landing process in the main frame bottom right (80%) show the restored video again. For the graphic animation, perhaps the YouTube video by Oberg N titled "F9R in KSP" is pretty close to what we would need. Perhaps this can be captioned with the interpretation. 7) A final pass, full screen of the team's video including enhancements such as a frame counter at the bottom right and wronkiew's nice clock in the bottom left. End with credits thanking core people - slow scroll - and every username on the forum - fast scroll.