Author Topic: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread  (Read 916579 times)

Online 411rocket

  • Full Member
  • **
  • Posts: 261
  • Retired RCEME w/ tours in Cyprus, Croatia, Bosnia
  • Vancouver Island
  • Liked: 79
  • Likes Given: 120
Newbie here - I have experience working with MPEG-TS streams and I've written a few custom tools that can fix corruption at the transport stream layer. Is it worth me working on the split TS files that shanuson created from the raw edit8.ts?

I've run his files through the tools, and they fix a lot of stuff like bad PIDs, bad program continuity counters, wrong adaptation field lengths, things like that. The start positions of various badly-corrupted P-frames become visible, but they're still quite damaged.

I guess I'm asking, would the recovery effort benefit from this?

In my way of thinking, anyone with skills that can help the effort, should be encouraged to do so. Some, seem to enjoy the challenge of this task, but remember to take breaks, to prevent burnout (or showing up at work in a daze). As life still happens, regardless of this fantastic group effort. 

Offline princess

  • Member
  • Posts: 65
  • Liked: 106
  • Likes Given: 25
Excellent, I've been trying to clean them up some.  But I'm totally in the dark.  I've done what I can for parts _11 and _12.  But I'm only really playing with a hex editor.  So I don't even know if what I'm doing is helpful, beyond the fact that it's cleaned up a bunch of the continuity errors.  It did return a few additional frames, but I can't see them (no computer Fu, i'm capable of following exact directions but that's it).  I've attached my attempts.  Someone let me know if they're any improvement or if I'm just wasting my time.  Not that that will necessarily keep me from goofing around with them.

That's great, thanks for posting those. I've had a quick look at part 11 and what you've done to it - I've attached the output of my TS fix tool when run against shanuson's part 11.

The good part is that it's managed to fix a few more TS-level issues! It deliberately doesn't change the contents of the data stream, so it hasn't fixed some of the MPEG4 headers that you'd got. I'll see if I can update the tool so it can incorporate your MPEG4 changes.

One thing I did notice is that sometimes you removed the adaptation field from an 0x03e8 packet when it looks too long, but as far as I can see it is sometimes legitimately long. For example, packet 87 at file offset 0x00003fe4 is marked as having a 69-byte adaptation field. It's human nature to think "whoa, that can't be right" and zero the AF, but if you do this and then look at the data it has a huge run of 0xff bytes before the data starts (from deruch's edit):

Packet 87 at 0x00003fe4: PID 0x03e8 Pay3:4500ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff

I believe the correct thing to do in this case is to leave the AF at 69 bytes (from my edit):

Packet 87 at 0x00003fe4: PID 0x03e8 AF[69] Pay3:ff01c0d80f09017b09418cde36def8f3fb90982fa7bf8dbee58fbdef1bcf7c6d

However, in other cases you can see that a packet header has become corrupted so that it adds a freaky AF, sometimes one that's longer than the 188-byte TS packet itself! So it is hard to automate this kind of checking, it has to be done manually. Just be super-careful when removing the AF from a packet, as you might introduce a huge run of padding into the MPEG4 datastream.

If we sort out the lower-level TS problems, your MPEG4 header fixes will be very useful, so we'll probably need to work together on this one!

Offline Req

  • Full Member
  • ****
  • Posts: 400
  • Liked: 413
  • Likes Given: 2581
I've been attempting to tune this bash script, with an eye towards performance.

This script is searching for two bits between 8800 and 9065 on frame 52.  It's a lot of combinations, and I am getting roughly 600 frames per minute.

Quote
for i in {8800..9064} ; do for j in {8801..9065} ; do ffmpeg.exe -debug mb_pos_size -s:0 704:480 -mmb X:76768:80,X:22120:80,X:45038:80,X:42196:80,X:66234:80,X:50298:80,X:$i:80,X:$j:80, 0:7:-2:-10:-10:-10:-10:8:-5,2:7:15506,9:7:-1,15:7:16391,17:7:-1,28:7:16704, 35:7:-2:-10:-10:-10:-10:8:-5,29:9:21074,34:9:-1,39:9:21626,5:28:-1::63,10:28:80196 -i iframe52.mpg4-img 2> output-$i-$j.txt -f image2 /dev/null -y ; done ; done

The output images have been entirely disabled, and the directive to ignore errors has been removed.  I am not 100% sure that the latter is "safe" but it does significantly increase the speed of the process.

Note that there are spaces in the mmb string that would need to be removed to run this.

After the run is done, I first do a grep for 42:03:9061 , then a grep for 00:29:81738 .  The second one is just in case the first "known good" position ends up changing due to correctly flipped bits.  Edit: This second test is probably worthless.

If anybody has suggestions for how to speed this up, I am all ears!  The search space can likely be reduced but I'm really unsure of what would be advisable.

Just sticking to two bits right now... going for three in a search space this size would take something like 21 days at a rate of 600 tries per minute.

Farm it out to multiple cores...

A quick paste(and edited a bit for your case) from something I made for a friend to run N rsyncs at once - in this case it's looking for "ffmpeg" in the processlist to track threads.

# use command-line argument for threads if specified
if [ "$1" == "" ]
then
  threads=3
else
  threads=$1
fi

..<snip>..

for i in {8800..9064}
do
  # check to see if enough threads are running and wait if so
  while [ `ps aux | grep ffmpeg | grep -v grep | wc -l` -ge $threads ]
  do
    sleep .1
  done

  # spawn a worker on the current offset
  /root/process_one $i &
  sleep .1
done

# don't exit back to the prompt until all of the workers are finished
while [ `ps aux | grep ffmpeg | grep -v grep | wc -l` -gt 0 ]
do
  sleep .1
done


Maybe farm out the "for j in" part this way.  Use however many logical cores (this includes hyper-threading) that your system has +1 (so for a 4-core i7 you'd want to use 9).  You'll have to eliminate your temp file per an earlier recommendation or use $i for it's filename if you want to run several at once.  Also, yes, as previously mentioned by another poster, if you're going to be using a temp file you want to make sure that it's on a ramfs mount with atime disabled(if applicable in your distro).

Even if ffmpeg is already using multiple cores, you are likely to see at least double your current performance.  If ffmpeg is not using multiple cores, you should see a fairly linear increase relative to number of cores used.  If ffmpeg is already using multiple cores, you should probably run the script with just 2 or 3 threads, not cores+1.

This should work well since you have a decent amount of work to farm out per worker if the farmed out work is the "for j in" part.  If you needed to farm out tons of really short workers you'd want to use memcached listening on a socket to track the number of workers running to eliminate the time of the "ps" call and invoke them from compiled code like C++ or hiphop'd php or something instead of a shell script.

Also, I'm not sure what the output of this looks like, but you should probably append whatever useful output this stuff generates into a logfile and run the main script in a screen, since it may take a while, and you don't want it aborting just because you got disconnected from your SSH session.  You want the logfile because it's a pain in the ass to scroll up in a screen.  This also allows you to detach the screen and just tail -f the logfile.  Note that >> appends stdout and 2>> appends errout, so you'll need to use both if you also want error output in the log.
« Last Edit: 05/24/2014 02:15 AM by Req »

Offline deruch

  • Senior Member
  • *****
  • Posts: 2022
  • California
  • Liked: 1564
  • Likes Given: 2844
Good catch, Princess.  There's probably a couple of other places where I made the same mistake.   :'( 

Have you been to the http://spacexlanding.wikispaces.com yet?  There's a section there, towards the bottom of the active work that deals with the headers and .ts files.   
Shouldn't reality posts be in "Advanced concepts"?  --Nomadd

Offline Req

  • Full Member
  • ****
  • Posts: 400
  • Liked: 413
  • Likes Given: 2581
Actually, it occurs to me that the straight up "ps" approach to tracking workers may have problems with the above script, because the workers are constantly starting and stopping ffmpeg, which ps is looking for.

Another easy trick to track spawned workers is just to use PHP.

To check workers you can use:
while [ `ps aux | grep process_one\.php | grep -v grep | wc -l` -ge $threads ]

And to spawn workers you can use:
# spawn a worker(via php passthrough for easy thread tracking) on the current offset
php /root/process_one.php $i &

process_one.php looks like this:

<?

function parseArgs($argv)
{
        array_shift($argv);
        $out = array();
        $i = 0;
        foreach ($argv as $arg)
        {
                $out[$i] = $arg;
                $i = $i + 1;
        }
        return $out;
}

// parse arguments from commandline
$args = parseArgs($argv);
if ($args[0])
{
        $offset = $args[0];
}

exec('/root/process_one ' . $offset);

?>

Again, the overhead php adds to spawning a process in this case is negligible because the workers will be running for a good while each and you're spawning thousands, not millions or billions.
« Last Edit: 05/24/2014 02:31 AM by Req »

Offline mvpel

  • Full Member
  • ****
  • Posts: 1124
  • New Hampshire
  • Liked: 1295
  • Likes Given: 1686
Try using HTCondor - when you install it on your Linux box after downloading from http://research.cs.wisc.edu/htcondor/ as an RPM, it will set itself up to manage jobs on the local machine. You submit the jobs and it will feed them into however many cores you have as needed.

Edit: Sorry, typo in the URL.
« Last Edit: 05/24/2014 03:03 AM by mvpel »
"Ugly programs are like ugly suspension bridges: they're much more liable to collapse than pretty ones, because the way humans (especially engineer-humans) perceive beauty is intimately related to our ability to process and understand complexity. A language that makes it hard to write elegant code makes it hard to write good code." - Eric S. Raymond

Offline Req

  • Full Member
  • ****
  • Posts: 400
  • Liked: 413
  • Likes Given: 2581
Try using HTCondor - when you install it on your Linux box after downloading from http://research.cs.uwisc.edu/htcondor/ as an RPM, it will set itself up to manage jobs on the local machine. You submit the jobs and it will feed them into however many cores you have as needed.

Link is broken.

I can tell you right now though, it'll be quite easier to make 3 10-20 line files in this case.

My background is hosting extremely large scale sites and applications(100,000+ concurrent users), and I also had a sole source contract with the state department for several years to aggregate and analyze "sources of interest" to the tune of >150GB/day(of text) including twitter, facebook, forums, etc for sentiment analysis/event prediction and notification/outlier detection and characterization/personal network visualization and characterization/etc to provide intelligence for select embassies in the middle east.  To complicate matters, it had to "understand" all of the various dialects of arabic/farsi/urdu/kurdish/turkish/etc along with the standard english/german/spanish/etc obviously.  I had to develop a lexicon system that had native speakers characterizing words in the order of their significance in the dataset.  I do have some sense of what it looks like when you want to spread a task across a few cores, or 14 cabinets full of servers.

Back on topic... Implementation isn't even really the crux of the matter, most of the thought and design goes into HOW you will stage the data and scale the task to operate efficiently given your dataset and task.  This particular task can be readily scaled just using the two loops he already uses.  Implementation on the shell won't have a noticeable impact on performance for this data set, and 30-60 lines of code which is mostly copy and paste at this point is pretty ridiculously easy implementation.
« Last Edit: 05/24/2014 04:12 PM by Req »

Offline Untribium

  • Member
  • Posts: 32
  • Switzerland
  • Liked: 32
  • Likes Given: 78
Actually, it occurs to me that the straight up "ps" approach to tracking workers may have problems with the above script, because the workers are constantly starting and stopping ffmpeg, which ps is looking for.

Another easy trick to track spawned workers is just to use PHP.

To check workers you can use:
while [ `ps aux | grep process_one\.php | grep -v grep | wc -l` -ge $threads ]

And to spawn workers you can use:
# spawn a worker(via php passthrough for easy thread tracking) on the current offset
php /root/process_one.php $i &

process_one.php looks like this:

-snip-

Again, the overhead php adds to spawning a process in this case is negligible because the workers will be running for a good while each and you're spawning thousands, not millions or billions.

Looks like this: http://coldattic.info/shvedsky/pro/blogs/a-foo-walks-into-a-bar/posts/7 might be an option as well. Generate the addresses using seq and then pipe it to xargs which then runs a fixed number of ffmpeg processes at a time. I'll check it out tomorrow, should get some sleep first :)

Offline Req

  • Full Member
  • ****
  • Posts: 400
  • Liked: 413
  • Likes Given: 2581
Actually, it occurs to me that the straight up "ps" approach to tracking workers may have problems with the above script, because the workers are constantly starting and stopping ffmpeg, which ps is looking for.

Another easy trick to track spawned workers is just to use PHP.

To check workers you can use:
while [ `ps aux | grep process_one\.php | grep -v grep | wc -l` -ge $threads ]

And to spawn workers you can use:
# spawn a worker(via php passthrough for easy thread tracking) on the current offset
php /root/process_one.php $i &

process_one.php looks like this:

-snip-

Again, the overhead php adds to spawning a process in this case is negligible because the workers will be running for a good while each and you're spawning thousands, not millions or billions.

Looks like this: http://coldattic.info/shvedsky/pro/blogs/a-foo-walks-into-a-bar/posts/7 might be an option as well. Generate the addresses using seq and then pipe it to xargs which then runs a fixed number of ffmpeg processes at a time. I'll check it out tomorrow, should get some sleep first :)

I am interested to see the results.  I'm just so busy that my "recreation" basically involves this thread and a few other news sites at the moment.  Maybe you have found something that will be very useful to my endeavors in the future, at least on one level of scale!  Although I must admit, I do enjoy coding this type of thing.
« Last Edit: 05/24/2014 04:42 AM by Req »

Offline mvpel

  • Full Member
  • ****
  • Posts: 1124
  • New Hampshire
  • Liked: 1295
  • Likes Given: 1686
I've used my bvi/wireshark approach to fix the transport stream headers on fixed_edit8_part_229.ts, and the "clean47" version is attached below. There were good-sized chunks of bad headers at the outset of the file, and I'm hoping that it can reveal something more of the top half of that frame, though I'm not particularly hopeful.
« Last Edit: 05/24/2014 06:10 AM by mvpel »
"Ugly programs are like ugly suspension bridges: they're much more liable to collapse than pretty ones, because the way humans (especially engineer-humans) perceive beauty is intimately related to our ability to process and understand complexity. A language that makes it hard to write elegant code makes it hard to write good code." - Eric S. Raymond

Offline princess

  • Member
  • Posts: 65
  • Liked: 106
  • Likes Given: 25
I've used my bvi/wireshark approach to fix the transport stream headers on fixed_edit8_part_229.ts, and the "clean47" version is attached below. There were good-sized chunks of bad headers at the outset of the file, and I'm hoping that it can reveal something more of the top half of that frame, though I'm not particularly hopeful.

You did really great! Those headers were totally mangled, but after your fixes your result file is pretty clean from a TS point of view.

I hope you don't mind but I've processed it a little more and fixed a couple of PCC discontinuities that were remaining. I've also removed the AF from any packets where either the AF is longer than the packet, or when the AF shows a wildly invalid PTS time. This is hopefully the correct thing to do, but please take a look and let me know what you think.

Offline Shanuson

  • Full Member
  • **
  • Posts: 272
  • Liked: 184
  • Likes Given: 468
I've used my bvi/wireshark approach to fix the transport stream headers on fixed_edit8_part_229.ts, and the "clean47" version is attached below. There were good-sized chunks of bad headers at the outset of the file, and I'm hoping that it can reveal something more of the top half of that frame, though I'm not particularly hopeful.

You did really great! Those headers were totally mangled, but after your fixes your result file is pretty clean from a TS point of view.

I hope you don't mind but I've processed it a little more and fixed a couple of PCC discontinuities that were remaining. I've also removed the AF from any packets where either the AF is longer than the packet, or when the AF shows a wildly invalid PTS time. This is hopefully the correct thing to do, but please take a look and let me know what you think.

what do you mean be remoced the AF? not every AF has a PTS.


Yet the file looks really really clean, only at 2 points there was a 47 23 e8 1x instead of 4703e81x
I did not check if the AF length is correct. looked ok. In the end if the AF length is to small and some FFs get in the img file, that will be ok and only force a reassignment of one MB or so and be handled by the nice group that is fixed P-frames.

Offline Shanuson

  • Full Member
  • **
  • Posts: 272
  • Liked: 184
  • Likes Given: 468
here is cleaned up part 7.

I will redo part 1 and 2 to also fixed all TS-headers.

The end result should be 15 fixed parts that only have to be put together to get a final ts file to which no tsfix etc has to be applied.

Cheers
Shanuson

Offline princess

  • Member
  • Posts: 65
  • Liked: 106
  • Likes Given: 25
I've also removed the AF from any packets where either the AF is longer than the packet, or when the AF shows a wildly invalid PTS time. This is hopefully the correct thing to do, but please take a look and let me know what you think.

what do you mean be remoced the AF? not every AF has a PTS.

I'm working on the theory that if a packet has a "long" AF, and it indicates it has a PTS in the AF header, and the PTS is obviously completely wrong, then probably what's happened is that the "AF" bit has been flipped on a normal data packet that shouldn't contain an AF.

The aim is to try to recover more of the MPEG4 data. If an error flips the "AF present" bit, then the normal MPEG4 data gets interpreted as an AF length, and a number of bytes get removed from the MPEG4 data stream. If we can detect when this has happened, we can recover more of the MPEG4 data, which might result in more valid blocks popping into place in I-frames and P-frames.

Offline princess

  • Member
  • Posts: 65
  • Liked: 106
  • Likes Given: 25
here is cleaned up part 7.

The data packets all look good to me - great job! I've cleaned up the big runs of 0x1fff padding packets so that there are no more invalid PIDs or invalid packets in the stream. It won't make any difference to the MPEG4 data (as your edits have already done that), but it means that the file is clean at the TS level.

Offline Oersted

  • Member
  • Full Member
  • ****
  • Posts: 847
  • Liked: 447
  • Likes Given: 264
My background is hosting extremely large scale sites and applications(100,000+ concurrent users), and I also had a sole source contract with the state department for several years to aggregate and analyze "sources of interest" to the tune of >150GB/day(of text) including twitter, facebook, forums, etc for sentiment analysis/event prediction and notification/outlier detection and characterization/personal network visualization and characterization/etc to provide intelligence for select embassies in the middle east.  To complicate matters, it had to "understand" all of the various dialects of arabic/farsi/urdu/kurdish/turkish/etc along with the standard english/german/spanish/etc obviously.

Cool to have the NSA contributing to this video clean-up now...  ;-)

Offline Shanuson

  • Full Member
  • **
  • Posts: 272
  • Liked: 184
  • Likes Given: 468
here is cleaned up part 7.

The data packets all look good to me - great job! I've cleaned up the big runs of 0x1fff padding packets so that there are no more invalid PIDs or invalid packets in the stream. It won't make any difference to the MPEG4 data (as your edits have already done that), but it means that the file is clean at the TS level.


Yes, i clean it by hand so i don't worry much about the 1fff parts. only look it up for CC if there is some ambiguity if a TS packet is data or something else. 
There are 2 cases where the AF bit should be set for a packet with PID 3e8: The first packet of a frame, and the last packet of a frame. The frist contains a PTS and stuff. the last only stuffing FF's in front of the last data bytes. The frist one is always there, but some frames dont need stuffing at the end.

I'm half way through part1 fixing the TS packets. :D More tonight.
Cheers,
Shanuson

Offline princess

  • Member
  • Posts: 65
  • Liked: 106
  • Likes Given: 25
Here is my cleaned version of the part14 TS file. Comments gratefully received!

Offline Lourens

  • Full Member
  • *
  • Posts: 156
  • The Netherlands
  • Liked: 206
  • Likes Given: 304
Question: when we use -mmb X:offset:pattern, are we flipping bits before or after the entropy decoding stage? Is this simply equivalent to flipping bits in the input file? Or is it essentially just a low-level way of editing the MB directly?

Offline saliva_sweet

  • Full Member
  • ****
  • Posts: 544
  • Liked: 415
  • Likes Given: 1309
Out of sheer desperation I decided to go deeper into the macroblock data to see if there's any way to find macroblock start positions, which would be a tremendous help for flipping bits. No amount of flipping can save a block if its start position is wrong. Unfortunately, but also unsurprisingly I did not find much signal in the bitstream to do this. But here are my results.

I took the first 623 macroblocks from frame 169. They are known to be good and haven't been tampered with. The sample is a bit small and mostly represents ocean data though. MB size range is 23 - 725 bits, mean 89, median 75. Data isn't random as expected, there is a slight tendency for ones to follow ones and zeros to follow zeros, but it's only like 2% over 50/50 so no use for flipping.

All macroblocks are type 3, not sure if this is due to small sample and rarity of type 4 macroblocks or they are not used in live encoding. This is about the only concrete result, but all it tells us is that a macroblock does not start with more than 2 zeros.

69% of macroblocks start with a one. 71% of macroblocks end with a one. The first part of macroblock is mcbpc, the ones used are (mcbpc:count):
001 : 37
011 : 32
010 : 123
1 : 431
This is followeb by ac_pred_flag (fixed length 1 bit), it's set to zero in 89% of macroblocks
Then comes cbpy. Here are the ones I saw with counts.
cbpy: 1010 : 43
cbpy: 000011 : 14
cbpy: 0011 : 4
cbpy: 0110 : 65
cbpy: 1000 : 45
cbpy: 1001 : 24
cbpy: 0111 : 20
cbpy: 1011 : 51
cbpy: 0100 : 20
cbpy: 00101 : 13
cbpy: 00011 : 11
cbpy: 0101 : 20
cbpy: 00100 : 6
cbpy: 000010 : 12
cbpy: 11 : 269
cbpy: 00010 : 6
It's set to 11 in 43% of macroblocks. The first bit of block data (1st bit of dct_dc_size_luminance code) is set to 1 in 65% of macroblocks. I didn't go further at this time. On the whole the best signature based on these data for macroblock start was thus 1|10111, where | is the macroblock border. 11% of macroblocks start with this pattern (about 1 in 9). So it could in principle provide a 64X reduction in search space at the cost of 89% of sensitivity. Not sure if that's worthwhile.

Tags: