Excellent, I've been trying to clean them up some. But I'm totally in the dark. I've done what I can for parts _11 and _12. But I'm only really playing with a hex editor. So I don't even know if what I'm doing is helpful, beyond the fact that it's cleaned up a bunch of the continuity errors. It did return a few additional frames, but I can't see them (no computer Fu, i'm capable of following exact directions but that's it). I've attached my attempts. Someone let me know if they're any improvement or if I'm just wasting my time. Not that that will necessarily keep me from goofing around with them.
I've been attempting to tune this bash script, with an eye towards performance.This script is searching for two bits between 8800 and 9065 on frame 52. It's a lot of combinations, and I am getting roughly 600 frames per minute.Quotefor i in {8800..9064} ; do for j in {8801..9065} ; do ffmpeg.exe -debug mb_pos_size -s:0 704:480 -mmb X:76768:80,X:22120:80,X:45038:80,X:42196:80,X:66234:80,X:50298:80,X:$i:80,X:$j:80, 0:7:-2:-10:-10:-10:-10:8:-5,2:7:15506,9:7:-1,15:7:16391,17:7:-1,28:7:16704, 35:7:-2:-10:-10:-10:-10:8:-5,29:9:21074,34:9:-1,39:9:21626,5:28:-1::63,10:28:80196 -i iframe52.mpg4-img 2> output-$i-$j.txt -f image2 /dev/null -y ; done ; doneThe output images have been entirely disabled, and the directive to ignore errors has been removed. I am not 100% sure that the latter is "safe" but it does significantly increase the speed of the process.Note that there are spaces in the mmb string that would need to be removed to run this.After the run is done, I first do a grep for 42:03:9061 , then a grep for 00:29:81738 . The second one is just in case the first "known good" position ends up changing due to correctly flipped bits. Edit: This second test is probably worthless.If anybody has suggestions for how to speed this up, I am all ears! The search space can likely be reduced but I'm really unsure of what would be advisable.Just sticking to two bits right now... going for three in a search space this size would take something like 21 days at a rate of 600 tries per minute.
for i in {8800..9064} ; do for j in {8801..9065} ; do ffmpeg.exe -debug mb_pos_size -s:0 704:480 -mmb X:76768:80,X:22120:80,X:45038:80,X:42196:80,X:66234:80,X:50298:80,X:$i:80,X:$j:80, 0:7:-2:-10:-10:-10:-10:8:-5,2:7:15506,9:7:-1,15:7:16391,17:7:-1,28:7:16704, 35:7:-2:-10:-10:-10:-10:8:-5,29:9:21074,34:9:-1,39:9:21626,5:28:-1::63,10:28:80196 -i iframe52.mpg4-img 2> output-$i-$j.txt -f image2 /dev/null -y ; done ; done
Try using HTCondor - when you install it on your Linux box after downloading from http://research.cs.uwisc.edu/htcondor/ as an RPM, it will set itself up to manage jobs on the local machine. You submit the jobs and it will feed them into however many cores you have as needed.
Actually, it occurs to me that the straight up "ps" approach to tracking workers may have problems with the above script, because the workers are constantly starting and stopping ffmpeg, which ps is looking for.Another easy trick to track spawned workers is just to use PHP.To check workers you can use:while [ `ps aux | grep process_one\.php | grep -v grep | wc -l` -ge $threads ]And to spawn workers you can use:# spawn a worker(via php passthrough for easy thread tracking) on the current offsetphp /root/process_one.php $i &process_one.php looks like this:-snip-Again, the overhead php adds to spawning a process in this case is negligible because the workers will be running for a good while each and you're spawning thousands, not millions or billions.
Quote from: Req on 05/24/2014 02:24 amActually, it occurs to me that the straight up "ps" approach to tracking workers may have problems with the above script, because the workers are constantly starting and stopping ffmpeg, which ps is looking for.Another easy trick to track spawned workers is just to use PHP.To check workers you can use:while [ `ps aux | grep process_one\.php | grep -v grep | wc -l` -ge $threads ]And to spawn workers you can use:# spawn a worker(via php passthrough for easy thread tracking) on the current offsetphp /root/process_one.php $i &process_one.php looks like this:-snip-Again, the overhead php adds to spawning a process in this case is negligible because the workers will be running for a good while each and you're spawning thousands, not millions or billions.Looks like this: http://coldattic.info/shvedsky/pro/blogs/a-foo-walks-into-a-bar/posts/7 might be an option as well. Generate the addresses using seq and then pipe it to xargs which then runs a fixed number of ffmpeg processes at a time. I'll check it out tomorrow, should get some sleep first
I've used my bvi/wireshark approach to fix the transport stream headers on fixed_edit8_part_229.ts, and the "clean47" version is attached below. There were good-sized chunks of bad headers at the outset of the file, and I'm hoping that it can reveal something more of the top half of that frame, though I'm not particularly hopeful.
Quote from: mvpel on 05/24/2014 04:53 amI've used my bvi/wireshark approach to fix the transport stream headers on fixed_edit8_part_229.ts, and the "clean47" version is attached below. There were good-sized chunks of bad headers at the outset of the file, and I'm hoping that it can reveal something more of the top half of that frame, though I'm not particularly hopeful.You did really great! Those headers were totally mangled, but after your fixes your result file is pretty clean from a TS point of view.I hope you don't mind but I've processed it a little more and fixed a couple of PCC discontinuities that were remaining. I've also removed the AF from any packets where either the AF is longer than the packet, or when the AF shows a wildly invalid PTS time. This is hopefully the correct thing to do, but please take a look and let me know what you think.
Quote from: princess on 05/24/2014 09:19 amI've also removed the AF from any packets where either the AF is longer than the packet, or when the AF shows a wildly invalid PTS time. This is hopefully the correct thing to do, but please take a look and let me know what you think.what do you mean be remoced the AF? not every AF has a PTS.
I've also removed the AF from any packets where either the AF is longer than the packet, or when the AF shows a wildly invalid PTS time. This is hopefully the correct thing to do, but please take a look and let me know what you think.
here is cleaned up part 7.
My background is hosting extremely large scale sites and applications(100,000+ concurrent users), and I also had a sole source contract with the state department for several years to aggregate and analyze "sources of interest" to the tune of >150GB/day(of text) including twitter, facebook, forums, etc for sentiment analysis/event prediction and notification/outlier detection and characterization/personal network visualization and characterization/etc to provide intelligence for select embassies in the middle east. To complicate matters, it had to "understand" all of the various dialects of arabic/farsi/urdu/kurdish/turkish/etc along with the standard english/german/spanish/etc obviously.
Quote from: Shanuson on 05/24/2014 10:56 amhere is cleaned up part 7.The data packets all look good to me - great job! I've cleaned up the big runs of 0x1fff padding packets so that there are no more invalid PIDs or invalid packets in the stream. It won't make any difference to the MPEG4 data (as your edits have already done that), but it means that the file is clean at the TS level.
What this whole problem boils down to is three phases (per frame):1) Extracting: Correcting the predictable streamheaders and I/P-frame headers. This way we (or actually the decoder) can extract all the video data without missing packets.2) Positioning: Finding all the macroblock-data (that is: finding all bitpositions where each mb-data starts) and then assigning the right macroblocks to the these starting bitpositions.3) Repairing: Correcting left-over DC-values errors either by bit flipping or by changing the DC-values after decoding. This should implcitly also fix "bad inheritance": where a macro inherents its DC base value from the wrong macroblock (that is: left or top).