The raw binary data is saved in the SIMH .tap format, which is wildly used by museums and computer simulations to mount magtapes. If we know what kind of data it is, we will also convert it to ASCII. Like the Switch Action Table tapes and Pione-QK799H tapes which contained EBCDIC text that we converted to ASCII in a separated file.
We don't know the image format used in the Pioneer 11 tapes (Pione-) and would love to process the raw data into a modern image file so everybody can see them. The Pione tapes came from NASA and the state uni of Arizona. They are from the mid to late 70's and mention the people "TOMASKO" shows up in the tape header, who turns out to beMartin G. Tomasko of ASU. And Zellner.
In the imaging mode, the data are converted to 64 levels of intensities (6 bits) and stored in a 6144-bit buffer onboard the spacecraft. The instrument overwrites this buffer as it starts each "vertical" scan with each rotation of the spacecraft. The memory read-in time is approximately one-half second, and the spacecraft rotation period is approximately 12.5 seconds, which means that there are approximately 12 seconds available for reading out the data from the memory. To read out these 6144 bits in the 12 seconds that are available requires a data rate of 512 bits per second. The IPP instrument receives 50 percent service rate on the spacecraft's telemetry downlink. Thus a 1024 bits per second telemetry downlink to Earth is the minimum data rate at which all the IPP data taken can be returned to Earth.
Cool, digital archaeology. I had a little fun looking at your Pione-QK7992H .tap file. While I was intrigued in the past by work on the Pioneer Anomaly (now resolved), it struck me that the interpretation of your tapes could be approached similarly.Quote from: apollo16uvc on 08/27/2018 07:06 pmThe raw binary data is saved in the SIMH .tap format, which is wildly used by museums and computer simulations to mount magtapes. If we know what kind of data it is, we will also convert it to ASCII. Like the Switch Action Table tapes and Pione-QK799H tapes which contained EBCDIC text that we converted to ASCII in a separated file.I wrote a tiny program to parse the data based on qk7992h.log for SIMH Magtape file format to try to gain more insights in its contents.Quote from: apollo16uvc on 08/27/2018 07:06 pmWe don't know the image format used in the Pioneer 11 tapes (Pione-) and would love to process the raw data into a modern image file so everybody can see them. The Pione tapes came from NASA and the state uni of Arizona. They are from the mid to late 70's and mention the people "TOMASKO" shows up in the tape header, who turns out to beMartin G. Tomasko of ASU. And Zellner. While I do not know the exact file format of the data in there, we can make some educated guesses and ask for more help from more informed experts on this forum. Dr. Tomasko was indeed mentioned with the Imaging Photopolarimeter (IPP) of Pioneer spacecraft and Section 7 and Appendix 1 at SP-349/396 PIONEER ODYSSEY describe many of its inner workings.QuoteIn the imaging mode, the data are converted to 64 levels of intensities (6 bits) and stored in a 6144-bit buffer onboard the spacecraft. The instrument overwrites this buffer as it starts each "vertical" scan with each rotation of the spacecraft. The memory read-in time is approximately one-half second, and the spacecraft rotation period is approximately 12.5 seconds, which means that there are approximately 12 seconds available for reading out the data from the memory. To read out these 6144 bits in the 12 seconds that are available requires a data rate of 512 bits per second. The IPP instrument receives 50 percent service rate on the spacecraft's telemetry downlink. Thus a 1024 bits per second telemetry downlink to Earth is the minimum data rate at which all the IPP data taken can be returned to Earth.Also, some example data from the IPP (with interpretation) can be found at Pioneer 10/11 Imaging Photopolarimeter DataNow, looking at the hexdump of qk7992h.tap, I see e.g. 0101340 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 000101360 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00*0101560 00 01 01 01 02 02 02 02 03 02 02 03 02 03 04 030101600 02 02 04 02 04 04 04 05 06 08 08 08 08 09 08 0b0101620 09 0a 0b 09 0a 0a 0e 0d 0e 10 0c 0f 0f 0e 13 120101640 13 11 14 12 14 14 15 15 15 15 16 16 14 14 15 170101660 16 18 1b 18 18 18 18 18 19 16 18 18 18 1b 16 180101700 1a 15 19 19 18 1a 16 17 1a 17 16 18 16 18 1a 1a0101720 17 14 15 14 18 15 17 15 10 15 13 13 11 11 10 0f0101740 10 10 0d 0f 0e 0e 0e 0c 0b 0d 0a 09 0a 0a 09 0a0101760 07 07 07 08 07 06 06 05 05 04 04 03 03 02 02 010102000 01 01 00 00 00 00 00 00 00 00 00 00 00 00 00 000102020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00While the interpretation of this enthusiast could be completely wrong this could loosely be interpreted as scanlines of gray pixels. It seems that with a bit more study of the old formats, a conversion to PNG or similar looks feasible.Help on the exact interpretation of the IPP data from the original engineers would be immensely useful though, and exciting!HTH
I have been trying to convert the data in .tap to a binary .bin file so people can look at it without having to know the SIMH format. If you could convert the binary data from the pione tapes SIMH files to a .bin file that would be great.
Quote from: apollo16uvc on 08/28/2018 09:29 amI have been trying to convert the data in .tap to a binary .bin file so people can look at it without having to know the SIMH format. If you could convert the binary data from the pione tapes SIMH files to a .bin file that would be great. Sure, I'll have a go to extend the simple parser to convert this data to a more portable format. Will update when I have something new!
You ought to get in touch with Keith Cowing, who did quite a bit of this data archaeology already.
Quote from: leovinus on 08/28/2018 04:21 pmQuote from: apollo16uvc on 08/28/2018 09:29 amI have been trying to convert the data in .tap to a binary .bin file so people can look at it without having to know the SIMH format. If you could convert the binary data from the pione tapes SIMH files to a .bin file that would be great. Sure, I'll have a go to extend the simple parser to convert this data to a more portable format. Will update when I have something new!Followup on yesterday's post and reaction. I managed to write a parser that dumps all tape blocks in ASCII form. This can be processed easily with awk/perl/python into something else. For the original Pione-QK7992H .tap file, here are the result files, and a first image interpretation First, I reformatted the .tap into qk7992h.raw.hex.plus.edcdic.asc. That still contains EBCDIC .The 5344 blocks of data look like blockNumber 0 length 360 error 0 :: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 etc blockNumber 1 length 360 error 0 :: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 etcNote that you guys processed 5358 records. I believe the difference is my parser error correcting code after block 131, which took most of the time to do build. I am sure there is an official way to find the next valid block but I followed a more ad-hoc way. It seems a minor difference in the result file qk7992h.raw.hex.plus.edcdic.asc. Also note that I chose to make an ASCII file instead of of .bin which would require yet-another-parser-program. Second, we have qk7992h.all.ascii.ascqk7992h.all.ascii.asc where the first line is blockNumber 0 length 360 error 0 :: 0001000107680768 1 2 2020 1530 1 C * * * * * 720* * * * * * * * * * 2 CLWR1530 VESTA,12MIN,SM.AP.,LO-DISP ZELLNER/MAY 21,1978 3 CCAM 2 IMAGE 1530 PROC VERSN 211 PARAM SET NO 8 4 C NO LAMP-EXP TIME=719 SECS 5 CImage aka pixel data is easily identifiable here by searching for "length 768" in the block identifier, e.g., blockNumber 21 length 768 error 0 :: 00 00 00 00 00 00 00 00 00 00 ...which has pixel values in the range [0..64]. The block identifier is, e.g., blockNumber <tape blocknumber> length <in bytes> error <was there a tape error or not> :: <delimiter>You'll note that a sequence of 768 blocks of length 768 makes an image.Third, throwing out all image data to get just the meta data and descriptions gives qk7992h.readable.txt which has various image descriptions, names etc. Interpretation by an expert would be useful Finally, I had a first go at extracting the first image of 768 lines 768 pixel values. I believe these are Red/Blue pairs. I made 2 quick PNG images which show a disc. Nice to see such an image after all the years since 1978. First PNG image and second PNG image. I am sure one of you can make better images out of the data as I do not know how to simulate the green channel. Hence, I aimed for a simple, gray scale image.I have not yet uploaded the C++ parser. A bit hackish but it did the job. Let me clean it up a bit. Based on the data files above, if someone has requests on what else the parser should do better then just let me know.What do you think?
You can clearly see in Leov's ASCII file how each image has metadata.I find it interesting that some images are clear, like at blockNumber 2694, but sometimes they look nothing like each other.blockNumber 3485 for example, looks nothing like 2694 and is looks like total garbage in the ASCII conversion. Any idea as to why this is?
Quote from: leovinus on 08/28/2018 09:17 pmQuote from: leovinus on 08/28/2018 04:21 pmQuote from: apollo16uvc on 08/28/2018 09:29 amI have been trying to convert the data in .tap to a binary .bin file so people can look at it without having to know the SIMH format. If you could convert the binary data from the pione tapes SIMH files to a .bin file that would be great. Sure, I'll have a go to extend the simple parser to convert this data to a more portable format. Will update when I have something new!Followup on yesterday's post and reaction. I managed to write a parser that dumps all tape blocks in ASCII form. This can be processed easily with awk/perl/python into something else. For the original Pione-QK7992H .tap file, here are the result files, and a first image interpretation First, I reformatted the .tap into qk7992h.raw.hex.plus.edcdic.asc. That still contains EBCDIC .The 5344 blocks of data look like blockNumber 0 length 360 error 0 :: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 etc blockNumber 1 length 360 error 0 :: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 etcNote that you guys processed 5358 records. I believe the difference is my parser error correcting code after block 131, which took most of the time to do build. I am sure there is an official way to find the next valid block but I followed a more ad-hoc way. It seems a minor difference in the result file qk7992h.raw.hex.plus.edcdic.asc. Also note that I chose to make an ASCII file instead of of .bin which would require yet-another-parser-program. Second, we have qk7992h.all.ascii.ascqk7992h.all.ascii.asc where the first line is blockNumber 0 length 360 error 0 :: 0001000107680768 1 2 2020 1530 1 C * * * * * 720* * * * * * * * * * 2 CLWR1530 VESTA,12MIN,SM.AP.,LO-DISP ZELLNER/MAY 21,1978 3 CCAM 2 IMAGE 1530 PROC VERSN 211 PARAM SET NO 8 4 C NO LAMP-EXP TIME=719 SECS 5 CImage aka pixel data is easily identifiable here by searching for "length 768" in the block identifier, e.g., blockNumber 21 length 768 error 0 :: 00 00 00 00 00 00 00 00 00 00 ...which has pixel values in the range [0..64]. The block identifier is, e.g., blockNumber <tape blocknumber> length <in bytes> error <was there a tape error or not> :: <delimiter>You'll note that a sequence of 768 blocks of length 768 makes an image.Third, throwing out all image data to get just the meta data and descriptions gives qk7992h.readable.txt which has various image descriptions, names etc. Interpretation by an expert would be useful Finally, I had a first go at extracting the first image of 768 lines 768 pixel values. I believe these are Red/Blue pairs. I made 2 quick PNG images which show a disc. Nice to see such an image after all the years since 1978. First PNG image and second PNG image. I am sure one of you can make better images out of the data as I do not know how to simulate the green channel. Hence, I aimed for a simple, gray scale image.I have not yet uploaded the C++ parser. A bit hackish but it did the job. Let me clean it up a bit. Based on the data files above, if someone has requests on what else the parser should do better then just let me know.What do you think?The metadata is much more readable now, they used some kind of window formatting around sets of text. Perhaps on their computers/printers this would look like a line? Something like PETSCII and IBM extended ASCII, which was used to make pseudo-graphics in MS-DOS programs.
Would be be possible to have a function where it doesn't add any block information before the data? so we just have the tape data.
I am incredible happy that we got our first pictures. Are you positive they are correct? Grayscale or not, they look awesome.
I notice the files are 255x255 while the metadata talks about 768x768, is this because of the missing colour channels?
Quote from: david-moon on 08/27/2018 07:15 pmYou ought to get in touch with Keith Cowing, who did quite a bit of this data archaeology already.Right!Look up the Lunar Orbiter Image Recovery Project.A quick plug into Google will pull up many links to this now completed project to digitize and process the data from old Lunar Orbiter data tapes, just like what you are doing. Best of luck!(Cowing is at NASAWatch.com.)
Quote from: Comga on 08/28/2018 10:20 pmQuote from: david-moon on 08/27/2018 07:15 pmYou ought to get in touch with Keith Cowing, who did quite a bit of this data archaeology already.Right!Look up the Lunar Orbiter Image Recovery Project.A quick plug into Google will pull up many links to this now completed project to digitize and process the data from old Lunar Orbiter data tapes, just like what you are doing. Best of luck!(Cowing is at NASAWatch.com.)Thank you! The link you gave has a typo though. It should be www.moonviews.com it seems.
Thanks for the update Leo, awesome work.I talked with Ted Stryk before, and he said QK7992H contains IPP data from zodiacal light observations. I am sure we can just take 4 individual parts out of the tool and mix them together. Once we have the complete image I will send it to ted Stryk to see what he has to say about it. or maybe it will be easier to modify the tool to support higher resolution output.
...While I do not know the exact file format of the data in there, we can make some educated guesses and ask for more help from more informed experts on this forum. Dr. Tomasko was indeed mentioned with the Imaging Photopolarimeter (IPP) of Pioneer spacecraft and Section 7 and Appendix 1 at SP-349/396 PIONEER ODYSSEY describe many of its inner workings....
Quote from: apollo16uvc on 08/29/2018 05:40 pmThanks for the update Leo, awesome work.I talked with Ted Stryk before, and he said QK7992H contains IPP data from zodiacal light observations. I am sure we can just take 4 individual parts out of the tool and mix them together. Once we have the complete image I will send it to ted Stryk to see what he has to say about it. or maybe it will be easier to modify the tool to support higher resolution output.How about this? Fixed my tool to allow the full PNG image size. This is again the first image, now in full 768x768, gray scale. Both based on the block numbers 21 to 788, and one contrast normalized. Would this make sense as an image of zodiacal light?first.image.768x768.grayscale.contrast.eq.pngfirst.image.768x768.grayscale.pngsecond.image.768x768.grayscale.contrast.eq.pngsecond.image.768x768.grayscale.pngthird.image.768x768.grayscale.contrast.eq.pngthird.image.768x768.grayscale.pngEDIT: added two more 768x768 from the same tape. These are all the 768x768 from the tape. There are a few sections 768x1536 which might be color coded. Will have a look
Quote from: leovinus on 08/29/2018 06:58 pmQuote from: apollo16uvc on 08/29/2018 05:40 pmThanks for the update Leo, awesome work.I talked with Ted Stryk before, and he said QK7992H contains IPP data from zodiacal light observations. I am sure we can just take 4 individual parts out of the tool and mix them together. Once we have the complete image I will send it to ted Stryk to see what he has to say about it. or maybe it will be easier to modify the tool to support higher resolution output.How about this? Fixed my tool to allow the full PNG image size. This is again the first image, now in full 768x768, gray scale. Both based on the block numbers 21 to 788, and one contrast normalized. Would this make sense as an image of zodiacal light?first.image.768x768.grayscale.contrast.eq.pngfirst.image.768x768.grayscale.pngsecond.image.768x768.grayscale.contrast.eq.pngsecond.image.768x768.grayscale.pngthird.image.768x768.grayscale.contrast.eq.pngthird.image.768x768.grayscale.pngEDIT: added two more 768x768 from the same tape. These are all the 768x768 from the tape. There are a few sections 768x1536 which might be color coded. Will have a lookHi Leo,I have been feeling ecstatic yesterday and today, but today really beats it. My long-term goal was to be able to digitize the tapes and present their contents in a modern format. You have done that! incredible work, I can't thank you enough.
Thanks for the update Leo, awesome work.I talked with Ted Stryk before, and he said QK7992H contains IPP data from zodiacal light observations.
I keep several forums updated on the status of these projects, may I add your files? I will credit you of course.
Yes I suspect a wide 768x image has two colour channels separated into two square parts. Should not be hard to merge into a colour image? will just have to add a false third colour layer manually, perhaps by processing the average out of the two colour layers we can determine the third color intensity value? It would be best to provide both grayscale layers so people can mess with it themselves.
This project has worked because everybody always provided transparent and raw data for the next person to take up and work with. It is key that everybody is transparent about what they have done to a file so we all understand each other. Everything has been volunteering work.Thanks, it has been a real high-speed roller-coaster. And I got several more tapes on the way.