One issue seems to be corrupt motion vectors (see the legs), would it be possible to implement something to change the motion vectors values? or at least set them to 0?
Quote from: Shanuson on 05/21/2014 06:52 amMy tools are more or less my hands and a few python scripts. and yes, i was conservative or not concerned about that problem yet. Did you only fix some 4byte TS-headers? or more? And only for some part or the whole stream? If only for one part. could you apply those changes to the Part_X.ts file so we can more easily merge your and my changes together?In the part9 I uploaded, it was strictly the transport stream headers, since that's what was easily discernable with Wireshark, and I only dealt with that one segment of the stream. Working through the procedure by hand, though, gave me a sense of what the algorithm would need to be to implement it, and extend it to the PES and MPEG headers. I'm currently thinking that breaking out the entire stream into an array of nested C structs will make it easier to manipulate and parse, and apply the known anchor points to the bits.I recall you mentioning a section where the 188-byte alignment goes out the window at one point - something about AJMartin's 28- or 56-byte frames, perhaps? Do you have the offset where that occurs?
My tools are more or less my hands and a few python scripts. and yes, i was conservative or not concerned about that problem yet. Did you only fix some 4byte TS-headers? or more? And only for some part or the whole stream? If only for one part. could you apply those changes to the Part_X.ts file so we can more easily merge your and my changes together?
Hopefully, there'll be some appropriate thank you for all this work coming the workers' way. A guided tour or an invite to a launch, perhaps?
^ Amazing, awesome, astonishing - yes, I'm impressed with the achievement here (and more to come!).Hopefully, there'll be some appropriate thank you for all this work coming the workers' way. A guided tour or an invite to a launch, perhaps?
Hello everyone, latest update of the gif (frames 169-173,175-177,180-181,183-199)It looks really cool!Note that many p-frames only need very little correction, some even don't need any.One issue seems to be corrupt motion vectors (see the legs), would it be possible to implement something to change the motion vectors values? or at least set them to 0?
Does it make sense to start working on fixing p-frames before their counter bytes and stuff was checked (wiki -> extraction of p-frames)? I was going to start on frames 52 and up, since the iframe looks very promising, so I'm wondering if I should try to fix the headers first...Also, this gif looks terrific!
Quote from: Untribium on 05/21/2014 04:45 pmDoes it make sense to start working on fixing p-frames before their counter bytes and stuff was checked (wiki -> extraction of p-frames)? I was going to start on frames 52 and up, since the iframe looks very promising, so I'm wondering if I should try to fix the headers first...Also, this gif looks terrific!You can start with this one, the problematic ones are those where not all of the 19 p-frames are detected.
The new online editor isn't showing bitpositions. Could we get the old one back until it's fixed?
Quote from: Chris Bergin on 05/17/2014 01:51 pmComing up on the 150,000 view mark for this thread/effort. Noting it as there's a small active team of people really working hard on this, but also a HUGE amount of people watching that work. Always nice to have a large appreciative audience, both inside and outside the space industry.You'd be astonished how many views are from IP addresses in a place called Hawthorne, California! I was curious how this thread ranked overall in the history of SpaceX threads (currently as of this writing, reads are >180,000) so I sorted the other SpaceX dedicated threads and only the 100,000+ and Missions sections have threads that exceed 180,000. Currently this thread is the 27th most read SpaceX thread and I think the NSF server (sorry Chris ) will explode when you all get the video restored "even" to the level of SwissCheese's latest gif (which is awesome BTW). see: Quote from: SwissCheese on 05/20/2014 04:09 pmI updated http://spacexlanding.wikispaces.com/Frames+169-188 Graphic representation of the historical significance of the work that the video restoration group is doing (see pic).
Coming up on the 150,000 view mark for this thread/effort. Noting it as there's a small active team of people really working hard on this, but also a HUGE amount of people watching that work. Always nice to have a large appreciative audience, both inside and outside the space industry.You'd be astonished how many views are from IP addresses in a place called Hawthorne, California!
I updated http://spacexlanding.wikispaces.com/Frames+169-188
Do we have a list of all people who worked on this?
Today’s corporation exists in an increasingly complex and ever-shifting ocean of change. As a result, leaders need to rely more than ever on the intelligence and resourcefulness of their staff. Collaboration is not a “nice to have” organizational philosophy. It is an essential ingredient for organizational survival and success.
Elon asks who's leading the effort?Eight Tips for Collaborative LeadershipQuoteToday’s corporation exists in an increasingly complex and ever-shifting ocean of change. As a result, leaders need to rely more than ever on the intelligence and resourcefulness of their staff. Collaboration is not a “nice to have” organizational philosophy. It is an essential ingredient for organizational survival and success.That's what's been interesting about the effort - each individual has brought their individual talents, experience, knowledge, and skills to bear on different pieces of the problem to the extent that they are able to contribute, and everyone has been learning from everyone else.Outsized kudos are due, of course, to the people who have set up the framework that allows this collaboration to occur in the first place - Michael Niedermyer for his ffmpeg patches to simplify the work on the frames, Iain Cole for his magnificent online editor, Moralec for his work in maintaining the Wiki page, etc. But are they "leaders" in the traditional sense of the word? They're fellow collaborators.I think the effort and the fruit that it has borne so far is just a really excellent example of the power of "crowdsourcing" (I think someone will eventually have to add this video repair effort to the Wikipedia page for that term. ), and a testament to how many people around the world are cheering for Elon and his entire team at SpaceX, and the vision of a multiplanetary human society.I remember fondly my first Mars Society convention years ago in Boulder, and my work with the California Mars Society around 2000, and it's just magnificent to see how much closer we are to that goal today thanks to the efforts of not only Elon but also countless other people who were inspired - by Ray Bradbury, or Robert Zubrin, or Kim Stanley Robinson, or in any number of other ways - to each work in their own way and in their own capacity towards achieving it.In a way, this video repair mission is just another example of that same phenomenon. It's great to see it unfold right before our eyes.
Quote from: ajmartin on 05/21/2014 12:36 amLooking at the null packets, I noticed some patterns in the bit errors that might be useful for this kind of effort.Specifically, many (but not all) of the bit-flips come in groups of 3, with one bit flipped, then 13 unaffected, then the next two also flipped. This is most obvious in a hex-dump of the null packets, which are supposed to be filled with 0xff bytes: patterns such as "7f fc" or "fb ff e7" are common. But the same bit-flip patterns apply where errors overlay the TS headers, and presumably also when they occur in image data.Combinations (with XOR) of these 3-bit errors are also seen; mathematically, these error patterns are multiples of the polynomial x^15 + x + 1 over the binary field. This could be the result of FEC coding applied to a signal with too many errors for successful correction. I haven't figured it out completely but there are indications that the data may have been coded in 28-byte or 56-byte blocks (not respecting the TS packet boundaries); in particular, errors that don't fit this pattern seem to be found near offsets that are multiples of 28 bytes.There is still the problem of figuring out if the change helped or not, but this should help to reduce the search space. In low-error regions, there's a good chance that some of the flaws can be fixed with a single instance of this pattern.That is brilliant. And something that never occurred to me. It could be incredibly important. Just yesterday (posted here: http://forum.nasaspaceflight.com/index.php?topic=34597.msg1201317#msg1201317) through a lot of trial and error I ended up doing this to frame72: X:11807:80,X:11822:80. We could really use more statistics though. Need to know what is the probability of single flips vs patterned flips. Perhaps a graph showing probability of flipped bit in positions following another flipped bit going up to 28 bytes.
Looking at the null packets, I noticed some patterns in the bit errors that might be useful for this kind of effort.Specifically, many (but not all) of the bit-flips come in groups of 3, with one bit flipped, then 13 unaffected, then the next two also flipped. This is most obvious in a hex-dump of the null packets, which are supposed to be filled with 0xff bytes: patterns such as "7f fc" or "fb ff e7" are common. But the same bit-flip patterns apply where errors overlay the TS headers, and presumably also when they occur in image data.Combinations (with XOR) of these 3-bit errors are also seen; mathematically, these error patterns are multiples of the polynomial x^15 + x + 1 over the binary field. This could be the result of FEC coding applied to a signal with too many errors for successful correction. I haven't figured it out completely but there are indications that the data may have been coded in 28-byte or 56-byte blocks (not respecting the TS packet boundaries); in particular, errors that don't fit this pattern seem to be found near offsets that are multiples of 28 bytes.There is still the problem of figuring out if the change helped or not, but this should help to reduce the search space. In low-error regions, there's a good chance that some of the flaws can be fixed with a single instance of this pattern.
He only gets the space of a tweet, why not just have him link to this thread or the wiki page?