Author Topic: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread  (Read 947581 times)

Online Chris Bergin

Hey, guys,

I'm a Long Time Lurker, haven't posted since Armadillo Aerospace was first taking off.  I have been following this thread with delight and amazement, and wanted to show/compare just how much you all have done.  so I set this up:



Cheers! Keep up the awesome work!

That's superb! Getting that tweeted out! ;D

Welcome to the site's forum and thanks for making the effort with that!

Offline Quialiss

  • Member
  • Posts: 75
  • Liked: 82
  • Likes Given: 29
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1581 on: 06/13/2014 04:33 AM »
https://docs.google.com/spreadsheets/d/1RZWykghV6XTuMRjikDleiX_J-ifG0g548Fta5WTRY88/edit?usp=sharing

I mentioned earlier that a spreadsheet would be really nice for doing the 'final' edits, so I put one together.  It shows all the frames, full size, thanks to the wonderful work done previously to make the auto-updating spreadsheets.  It looks like the images were scaled down after loading so it shouldn't be any worse for bandwidth than the All Frames spreadsheet.

SwissCheese is the master of divining the correct alignment of the p frames, and is still working on some of part 3-4, so I haven't made notes on them yet, but other than that I've put my notes on them for things to still fix, and anyone and everyone can go add more things to the task list as they're noticed. 

While on that note, fixing the gray blocks(copy the correct DC values from a nearby block that's correct) and too dark/too intense colours(bitflip in a luma or chroma subblock, or remove a bad block upstream) in the p frames is pretty simple, each frame only takes a few minutes, perfect if you don't have too much time to spend.  ;)


Offline wronkiew

  • Full Member
  • *
  • Posts: 186
  • 34.502327, -116.971697
  • Liked: 105
  • Likes Given: 121
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1582 on: 06/13/2014 05:50 AM »
I couldn't find these names in the forum (I think they originated from the Wiki, if you know who these people are on the forum, PM me)
Asmegin
dgdpg
Exclavion
Gnonthgol
lewing
Noslyl
pshrpd
tentonine
Turix

Exclavion posted a message on the previous page!

Thank you IainCole for taking on this stinky job.

And yes, Condorcet is a pain to implement, but it is the fairest voting system known to mathematics.

Offline wronkiew

  • Full Member
  • *
  • Posts: 186
  • 34.502327, -116.971697
  • Liked: 105
  • Likes Given: 121
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1583 on: 06/13/2014 06:37 AM »
Ohsin posted a video of the ascent over in the interpretation thread, and it prompted me to see if I could cross-check the digit 2 bitmap. Two and four never appear unscrambled in the landing video, so I had to piece them together from multiple pframes. It turns out I had two bad pixels, as you can see below. This also confirms that the clock digits have identical pixels on the even and odd rows.

Offline Untribium

  • Member
  • Posts: 32
  • Switzerland
  • Liked: 32
  • Likes Given: 78
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1584 on: 06/13/2014 01:35 PM »
-snip-
If there is anyone who would not like to be on the list for whatever reason then drop me a PM and I will take that into account when adding up the numbers, also don't include me in your lists as I was lucky enough to receive something already (which I feel quite guilty about if I'm honest)
-snip-

Don't feel bad, you deserve every bit of whatever it is you got :)) The online editor was absolutely crucial to the success of this effort, it helped getting a lot of people involved (including myself) :)

Thank you IainCole for taking on this stinky job.

And yes, Condorcet is a pain to implement, but it is the fairest voting system known to mathematics.

Agreed :) The problem is that if you pick the top 15 by number of votes, then one can increase their "chance of winning" by not voting...I think...correct me if I'm wrong :)
« Last Edit: 06/13/2014 01:36 PM by Untribium »

Offline moralec

-snip-
If there is anyone who would not like to be on the list for whatever reason then drop me a PM and I will take that into account when adding up the numbers, also don't include me in your lists as I was lucky enough to receive something already (which I feel quite guilty about if I'm honest)
-snip-

Don't feel bad, you deserve every bit of whatever it is you got :)) The online editor was absolutely crucial to the success of this effort, it helped getting a lot of people involved (including myself) :)

Thank you IainCole for taking on this stinky job.

And yes, Condorcet is a pain to implement, but it is the fairest voting system known to mathematics.

Agreed :) The problem is that if you pick the top 15 by number of votes, then one can increase their "chance of winning" by not voting...I think...correct me if I'm wrong :)

Offtopic, but every voting system has issues. Actually Arrow (http://en.wikipedia.org/wiki/Arrow's_impossibility_theorem) proved that, almost by construction, there is no mechanism that is capable of satisfying "fairness" the basic fairness criteria. :)

Let's just let IainCole take care of this.... we are not here for the prizes anyway.
« Last Edit: 06/13/2014 07:00 PM by moralec »

Offline maximlevitsky

  • Member
  • Posts: 85
  • Liked: 86
  • Likes Given: 103
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1586 on: 06/13/2014 10:38 PM »
Since the launch did slip by 'few days' (as expected  ;) ), do you think the next landing will be at day time?
(unless it slips again by few months, but that isn't funny anymore  :( )

And its starts to be not funny indeed :(

Offline Lourens

  • Full Member
  • *
  • Posts: 156
  • The Netherlands
  • Liked: 206
  • Likes Given: 304
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1587 on: 06/13/2014 10:55 PM »
It looked nice and difficult, so I had a go at iframe 1 (from raw_final_fixed.zip):  8)

-mmb X:553:80,X:668:80,X:692:80,X:706:C0,X:732:80,X:746:C0,
X:921:80,X:936:C0,X:939:80,X:953:C0,X:988:80,X:1002:C0,X:993:80,X:1007:C0

Those seem to get the first five MBs in a somewhat working order. Mostly triple-flips, that pattern does seem to work quite well indeed! Nice find! I'm not sure about the earlier single flips though, maybe there's still something wrong there.

I'm a little confused now with all the spreadsheets, wiki, online editor, moderated submission system and how they're all linked together. Do I put the above on the wiki, on the spreadsheet, both? Where, how?

I did these using my local FFmpeg, which has arnezami's detailed block dump BITLOG in it. I've noticed that these aren't in the online editor, at least not in version 1. They're very helpful though, I found the above mainly by looking at which blocks had lots of DCT coefficients or large DC values. Is there any way the BITLOG patch could be applied to the online editor as well?


Offline Llian Rhydderch

  • Full Member
  • ****
  • Posts: 901
  • Terran Anglosphere
  • Liked: 612
  • Likes Given: 5976
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1588 on: 06/13/2014 10:58 PM »
Since the launch did slip by 'few days' (as expected  ;) ), do you think the next landing will be at day time?
(unless it slips again by few months, but that isn't funny anymore  :( )

And its starts to be not funny indeed :(

Wrong thread.  Launch slips are discussed elsewhere.
Re arguments from authority on NSF:  "no one is exempt from error, and errors of authority are usually the worst kind.  Taking your word for things without question is no different than a bracket design not being tested because the designer was an old hand."
"You would actually save yourself time and effort if you were to use evidence and logic to make your points instead of wrapping yourself in the royal mantle of authority.  The approach only works on sheep, not inquisitive, intelligent people."

Offline Quialiss

  • Member
  • Posts: 75
  • Liked: 82
  • Likes Given: 29
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1589 on: 06/13/2014 11:51 PM »
I'm a little confused now with all the spreadsheets, wiki, online editor, moderated submission system and how they're all linked together. Do I put the above on the wiki, on the spreadsheet, both? Where, how?

Wiki is the original system for keeping track of things, it requires you to format the mmb's so they don't run off the page, and you have to update the images separately.  It's also where the auto updating youtube videos are sourced from.

The spreadsheets(linked to on the wiki) just require you to paste in the MMB in the appropriate cell, and will update the images next time you load the spreadsheet. 

The moderated submission system is attached to the spreadsheets, but nobody's submitted anything through it yet... I believe the idea was convenience, and also to encourage people who aren't quite sure they've got it right to submit through it rather than directly editing the spreadsheets. 

So.. both for now.  Phasing out the wiki by having the videos source from the spreadsheets would be nice, as they're much easier to update, but it's okay as is. 



Also, you're crazy.   ;D  Of parts 1-2, the p frames of part 2 are probably the most recoverable, but if you're intent on the i frames, you might get some help referencing the raw video that SpaceX posted on youtube that has a couple more i frames on either side, for ideas on what you're looking at in frame 1. 


Offline cmobile

  • Member
  • Posts: 2
  • Liked: 6
  • Likes Given: 0
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1590 on: 06/14/2014 08:04 AM »
So.. After going back and forth and trying to work out a way to get a list of 15 people who will receive SpaceX goodies, it has come to this, a vote!

It sucks to have to whittle it down to 15, but this is the situation we find ourselves in.

...


I think we should send a list of everyone who made contributions and highlight the top 15 (I'll bet this number was somewhat arbitrary).  SpaceX can do with the list whatever they wish.  Hopefully the runner-ups will get at least a letter of acknowledgment.

And someone needs to post an un-boxing video of the loot; we're all on the same team regardless of who gets to keep the trophy!

Offline JohnKiel

  • Member
  • Posts: 15
  • United States
  • Liked: 10
  • Likes Given: 0
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1591 on: 06/14/2014 02:40 PM »
The spreadsheets(linked to on the wiki) just require you to paste in the MMB in the appropriate cell, and will update the images next time you load the spreadsheet. 

Just a slight clarification on this point:

After changing the MMB, the image might update after reloading the spreadsheet, but it could take anywhere from 5 to 10 minutes for this to happen. 

The problem is Google's image proxy caching the previous image.  Even though my spxi proxy script (a script that pulls a current MMB for a frame from the appropriate spreadsheet, and then either pulls that image from a local cache, or asks IanCole's server to generate new image data for the MMB) says "must revalidate", Google's proxy seems a little more aggressive about caching.

Offline JohnKiel

  • Member
  • Posts: 15
  • United States
  • Liked: 10
  • Likes Given: 0
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1592 on: 06/14/2014 03:29 PM »
So.. After going back and forth and trying to work out a way to get a list of 15 people who will receive SpaceX goodies, it has come to this, a vote!

It sucks to have to whittle it down to 15, but this is the situation we find ourselves in.

Everyone who wishes to vote should send me a PM on the forums with an ordered list of people (forum usernames please if possible) who they think deserve to get goodies. It doesn't need to be 15 long (the longer the better, so long as it is ordered) because I will collate the results of each list and pick the top 15. Also you can't vote for yourself =) Here is a list of names that has already been put forward in various places, you don't need to pick from this list, it's just here as a guide, if there is someone else you think is deserving then feel free to add them.

Adaptation
ajmartin
arnezami
dorkmo
gospacex
grythumn
jakusb
Jared (http://www.reddit.com/user/sharebrained)
John_L
lgjy98d (myroslav)
Lourens
mhenderson
michaelni
Mlindner
moralec
morningdew76
mvpel
princess
Quialiss
Saliva_Sweet
seanpg71
Shanuson
SwissCheese
theshadow27
Turix
Untribium
wronkiew

I couldn't find these names in the forum (I think they originated from the Wiki, if you know who these people are on the forum, PM me)
Asmegin
dgdpg
Exclavion
Gnonthgol
lewing
Noslyl
pshrpd
tentonine
Turix

If there is anyone who would not like to be on the list for whatever reason then drop me a PM and I will take that into account when adding up the numbers, also don't include me in your lists as I was lucky enough to receive something already (which I feel quite guilty about if I'm honest)

Polls close on Sunday so vote now! I will then publish the anonymised results and ask the lucky 15 to PM me their gender (I'm not going to guess), shirt size, and delivery address. Something to bear in mind, assuming that everyone gets something similar to what I did (I hope this is the case), the package will NOT fit through a letter box (not even close ;)) so make sure the delivery address is somewhere where the package can be taken and signed for.

Probably should put req (req is hosting the spxi image proxy scripts used for displaying images in the spreadsheets) on that list as well.
« Last Edit: 06/14/2014 03:31 PM by JohnKiel »

Offline Lourens

  • Full Member
  • *
  • Posts: 156
  • The Netherlands
  • Liked: 206
  • Likes Given: 304
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1593 on: 06/14/2014 09:30 PM »
I'm a little confused now with all the spreadsheets, wiki, online editor, moderated submission system and how they're all linked together. Do I put the above on the wiki, on the spreadsheet, both? Where, how?

<snip>

So.. both for now.  Phasing out the wiki by having the videos source from the spreadsheets would be nice, as they're much easier to update, but it's okay as is. 

Thanks for the explanation! I've decided to not yet put my results up there until I'm a bit more sure that they make sense. See also below.

Quote
Also, you're crazy.   ;D  Of parts 1-2, the p frames of part 2 are probably the most recoverable, but if you're intent on the i frames, you might get some help referencing the raw video that SpaceX posted on youtube that has a couple more i frames on either side, for ideas on what you're looking at in frame 1.

Well, I'm still working on automated bit flipping, and a frame that no human could or would do sounds like just what we would like to have an automatic system for. I think I'm getting a little bit closer to something workable, although I doubt it will ever be fully automated. But that's okay. And even if we never get I-frame 1 back, it may still be useful for filling in some of the gaps in the more interesting I-frames.

My current program tries only triple flips, and judges its performance not by looking at the picture, but by analysing arnezami's BITLOG output. I took a frame that we have a pretty good solution for (I-frame 52), and I collected some statistics on the various MB fields, e.g. which values occur for dc_lum, and how often? I use that information to judge whether a macroblock is likely to be correct. If it has a lot of common values, I assume that it's okay, if not, I'll run through the possible triple-flips and see if they help. I add the one that gives the most improvement to the mmb, and repeat, until the MB is good enough. Then I move on to the next one.

As you can see below in the top-left corner, it does seem to be improving things, but it's not quite there yet. One major omission is that it's not including AC values yet when checking how good a macroblock is, and the "good enough" threshold isn't tuned very well yet. As a result, there's some high frequency noise in the "fixed" blocks that probably shouldn't be there, and the whole thing gets desynchronised enough to give an unfixable green tint from MB 10 on. So even though the first ten MBs look good in this picture, they probably aren't completely. Also, if it does desynchronise somewhere, then the later grey blocks may well be an example of my program creating something that isn't really there. On the other hand, the bit flip rate here is not that big, on the order of one triple flip per 100 bits.

The code is an ugly hack in Bash, but if anyone wants it, just let me know. Meanwhile I'll keep fiddling with it :).

Offline Quialiss

  • Member
  • Posts: 75
  • Liked: 82
  • Likes Given: 29
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1594 on: 06/14/2014 10:05 PM »
Well, I'm still working on automated bit flipping, and a frame that no human could or would do sounds like just what we would like to have an automatic system for. I think I'm getting a little bit closer to something workable, although I doubt it will ever be fully automated. But that's okay. And even if we never get I-frame 1 back, it may still be useful for filling in some of the gaps in the more interesting I-frames.

My current program tries only triple flips, and judges its performance not by looking at the picture, but by analysing arnezami's BITLOG output. I took a frame that we have a pretty good solution for (I-frame 52), and I collected some statistics on the various MB fields, e.g. which values occur for dc_lum, and how often? I use that information to judge whether a macroblock is likely to be correct. If it has a lot of common values, I assume that it's okay, if not, I'll run through the possible triple-flips and see if they help. I add the one that gives the most improvement to the mmb, and repeat, until the MB is good enough. Then I move on to the next one.

As you can see below in the top-left corner, it does seem to be improving things, but it's not quite there yet. One major omission is that it's not including AC values yet when checking how good a macroblock is, and the "good enough" threshold isn't tuned very well yet. As a result, there's some high frequency noise in the "fixed" blocks that probably shouldn't be there, and the whole thing gets desynchronised enough to give an unfixable green tint from MB 10 on. So even though the first ten MBs look good in this picture, they probably aren't completely. Also, if it does desynchronise somewhere, then the later grey blocks may well be an example of my program creating something that isn't really there. On the other hand, the bit flip rate here is not that big, on the order of one triple flip per 100 bits.

The code is an ugly hack in Bash, but if anyone wants it, just let me know. Meanwhile I'll keep fiddling with it :).

I didn't realize you were doing them automatically!  I wouldn't mind seeing your script. 

Have you tested it on any of the more intact frames?  That should help figuring out the 'correctness' thresholds, giving you a reference frame to look at to find any places where your script is making incorrect flips.

Offline saliva_sweet

  • Full Member
  • ****
  • Posts: 552
  • Liked: 423
  • Likes Given: 1328
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1595 on: 06/14/2014 10:45 PM »
Well, I'm still working on automated bit flipping, and a frame that no human could or would do sounds like just what we would like to have an automatic system for. I think I'm getting a little bit closer to something workable, although I doubt it will ever be fully automated. But that's okay. And even if we never get I-frame 1 back, it may still be useful for filling in some of the gaps in the more interesting I-frames.

Very interesting approach, that I think has a lot of potential, but also needs a lot of effort to get working. I agree with Quialiss, you should be working on good frames first to see if it works and improve the method, like iframe 181. We don't even know if iframe 1 has anything recoverable at all. There seems to be a fairly sharp threshold where the FEC fails completely (possibly starts introducing even more errors) and data becomes totally unrecoverable by current methods. With enough flips and patience we could turn random noise into the mona lisa, but that's not what we want to accomplish.

Offline wronkiew

  • Full Member
  • *
  • Posts: 186
  • 34.502327, -116.971697
  • Liked: 105
  • Likes Given: 121
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1596 on: 06/15/2014 04:25 AM »
Probably should put req (req is hosting the spxi image proxy scripts used for displaying images in the spreadsheets) on that list as well.

Also I noticed that deruch is missing from the list. Deruch contributed to the ts cleanup effort.

Offline Req

  • Full Member
  • ****
  • Posts: 400
  • Liked: 413
  • Likes Given: 2581
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1597 on: 06/15/2014 06:35 AM »
So.. After going back and forth and trying to work out a way to get a list of 15 people who will receive SpaceX goodies, it has come to this, a vote!

It sucks to have to whittle it down to 15, but this is the situation we find ourselves in.

Everyone who wishes to vote should send me a PM on the forums with an ordered list of people (forum usernames please if possible) who they think deserve to get goodies. It doesn't need to be 15 long (the longer the better, so long as it is ordered) because I will collate the results of each list and pick the top 15. Also you can't vote for yourself =) Here is a list of names that has already been put forward in various places, you don't need to pick from this list, it's just here as a guide, if there is someone else you think is deserving then feel free to add them.

Adaptation
ajmartin
arnezami
dorkmo
gospacex
grythumn
jakusb
Jared (http://www.reddit.com/user/sharebrained)
John_L
lgjy98d (myroslav)
Lourens
mhenderson
michaelni
Mlindner
moralec
morningdew76
mvpel
princess
Quialiss
Saliva_Sweet
seanpg71
Shanuson
SwissCheese
theshadow27
Turix
Untribium
wronkiew

I couldn't find these names in the forum (I think they originated from the Wiki, if you know who these people are on the forum, PM me)
Asmegin
dgdpg
Exclavion
Gnonthgol
lewing
Noslyl
pshrpd
tentonine
Turix

If there is anyone who would not like to be on the list for whatever reason then drop me a PM and I will take that into account when adding up the numbers, also don't include me in your lists as I was lucky enough to receive something already (which I feel quite guilty about if I'm honest)

Polls close on Sunday so vote now! I will then publish the anonymised results and ask the lucky 15 to PM me their gender (I'm not going to guess), shirt size, and delivery address. Something to bear in mind, assuming that everyone gets something similar to what I did (I hope this is the case), the package will NOT fit through a letter box (not even close ;)) so make sure the delivery address is somewhere where the package can be taken and signed for.

Probably should put req (req is hosting the spxi image proxy scripts used for displaying images in the spreadsheets) on that list as well.

...Which JohnKiel made and continues to maintain and expand, adding database support and other features. :)

Offline wronkiew

  • Full Member
  • *
  • Posts: 186
  • 34.502327, -116.971697
  • Liked: 105
  • Likes Given: 121
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1598 on: 06/15/2014 07:31 AM »
I am testing a change to the auto-generator script to pull mmbs from the spreadsheets instead of the wiki. This is not enabled for the clockfixer or the YouTubifier yet.

Standard video:
http://adama.nocdirect.com/~wronkiew/spx_crs3/landing_base_set_alt.mp4

Slow-mo:
http://adama.nocdirect.com/~wronkiew/spx_crs3/landing_base_set_alt_slow.mp4

JPEG frames:
http://adama.nocdirect.com/~wronkiew/spx_crs3/landing_base_set_alt_jpg.zip

Generated mmb files:
http://adama.nocdirect.com/~wronkiew/spx_crs3/mmbs_alt/

There are a few minor differences in the generated mmbs between the wiki and the spreadsheets. Please take a look through the files and let me know if you see any problems. Otherwise it will go live tomorrow night.

Offline deruch

  • Senior Member
  • *****
  • Posts: 2216
  • California
  • Liked: 1720
  • Likes Given: 3540
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1599 on: 06/15/2014 10:13 AM »
Probably should put req (req is hosting the spxi image proxy scripts used for displaying images in the spreadsheets) on that list as well.

Also I noticed that deruch is missing from the list. Deruch contributed to the ts cleanup effort.
 

Thanks.  Iain originally included me on the list, but I PMed him to take me off of it.  I did like two little things, neither of them very well.  Really, I just wanted to be able to say that I helped/was a bit involved.  So, for me, mission accomplished.  Plus, it introduced me to this excellent forum, double bonus (for me anyways, not sure that it's to your benefit).
Shouldn't reality posts be in "Advanced concepts"?  --Nomadd

Tags: