Author Topic: Colourizing Apollo B/W photos with AI (Read 8641 times)

apollo16uvc · « **on:** 09/03/2019 04:58 pm »

Greetings everybody,

I am talking with several other people about setting up a machine-learning system that has been trained with colour Apollo photos. Once a data model has been generated I will try and have it colourize the black and white Apollo photos.

There were several thousand colour photos taken during the Apollo mission, I do not know if this will be enough.

Any links to high-resolution archives with colour and b/w photos will be great!
I would prefer archives that are easy to mass-download, such as FTP or simple links. I got 5TB of storage space.

Once I got enough photos to work with I will make some attempts.

Last week I finished building a new monster computer, and tomorrow an RTX 2080 Super should arrive. This card has Tensor cores which facilitate deep-learning.

To get an idea of what deep-learning can achieve, here is a model trained on 1.3 million photos:
https://colourise.sg/

Regards,
Apollo16uvc

libra · « **Reply #1 on:** 09/03/2019 05:09 pm »

You are bold, and this is a terrific project.

Sam Ho · « **Reply #2 on:** 09/03/2019 07:17 pm »

The Apollo Archive Flickr page has 1800dpi imagery arranged by film magazine, which conveniently sorts out the colour and b/w photos.

https://www.flickr.com/photos/projectapolloarchive/albums

The primary Apollo Archive web site has some indexing to help determine what is what.

http://www.apolloarchive.com/apollo_gallery.html

apollo16uvc · « **Reply #3 on:** 09/05/2019 08:02 pm »

Today my new high-end video card arrived.
An RTX 2080 Super by MSI, model Gaming X Trio.

The RTX lineup contains special Tensor cores which accelerate deep-learning.

Deep-learning/machine-learning/convoluted neural networks have been used to automatically colourize B/W photos by using hundreds of thousands of colour photos as a reference. Its even possible to colourize B/W video this way.

One such program is DeOldify: https://github.com/jantic/DeOldify
And this website: https://colourise.sg/

I don't know if it will work with the amount of colour photos avaible, but I will be giving it a shot. Generating a dataset from the colour photos may take several days. DeOldify advices the even higher end RTX 2080 Ti with 11 gigabytes of video memory, I got the model just below it with 'just' 8 gigabytes.

I think DeOldify is by far the best documented and wildy used image-learning program. So this is the one I will be using.

whitelancer64 · « **Reply #4 on:** 09/05/2019 08:06 pm »

Quote from: apollo16uvc on 09/05/2019 08:02 pm

Today my new high-end video card arrived.
An RTX 2080 Super by MSI, model Gaming X Trio.

The RTX lineup contains special Tensor cores which accelerate deep-learning.

Deep-learning/machine-learning/convoluted neural networks have been used to automatically colourize B/W photos by using hundreds of thousands of colour photos as a reference. Its even possible to colourize B/W video this way.

One such program is DeOldify: https://github.com/jantic/DeOldify
And this website: https://colourise.sg/

I don't know if it will work with the amount of colour photos avaible, but I will be giving it a shot. Generating a dataset from the colour photos may take several days. DeOldify advices the even higher end RTX 2080 Ti with 11 gigabytes of video memory, I got the model just below it with 'just' 8 gigabytes.

I think DeOldify is by far the best documented and wildy used image-learning program. So this is the one I will be using.

Is there a minimum number of photographs that you need to establish a database? There are tens of thousands of color pictures of the Apollo program.

apollo16uvc · « **Reply #5 on:** 09/05/2019 08:14 pm »

Quote from: whitelancer64 on 09/05/2019 08:06 pm

Quote from: apollo16uvc on 09/05/2019 08:02 pm
Today my new high-end video card arrived.
An RTX 2080 Super by MSI, model Gaming X Trio.

The RTX lineup contains special Tensor cores which accelerate deep-learning.

Deep-learning/machine-learning/convoluted neural networks have been used to automatically colourize B/W photos by using hundreds of thousands of colour photos as a reference. Its even possible to colourize B/W video this way.

One such program is DeOldify: https://github.com/jantic/DeOldify
And this website: https://colourise.sg/

I don't know if it will work with the amount of colour photos avaible, but I will be giving it a shot. Generating a dataset from the colour photos may take several days. DeOldify advices the even higher end RTX 2080 Ti with 11 gigabytes of video memory, I got the model just below it with 'just' 8 gigabytes.

I think DeOldify is by far the best documented and wildy used image-learning program. So this is the one I will be using.

Is there a minimum number of photographs that you need to establish a database? There are tens of thousands of color pictures of the Apollo program.

I don't think there is a minimum, but the more photos it can use as a reference the better. I am not sure how it works, but I think it basically learns which identifiable features and objects have which colours and gradiants.

We got a big benefit compared to photos taken on earth: similarity. Earth photos can contain millions of different subjects, colours, objects, people, surfaces, textures. DeOldify has a pre-made set made from 1.6 million photos.

Due to obvious reasons this is much less so with Apollo photos because they were all taken in the same enviroment (Space craft or lunar surface) and usually of the space suits (deep-learning is not so good with the human body/clothing and faces.) So I think we can get away with much less photos.

I will try and contact the creator of DeOldify and ask what our chances are of decent results...

edkyle99 · « **Reply #6 on:** 09/05/2019 08:30 pm »

This is interesting, and I understand the technical fun in the effort, but I have to ask the obvious question. Why?

The Apollo astronauts took black and white photographs on purpose. They used filters for that purpose and were trained for that purpose. Why alter their work?

Like "Psycho". Hitchcock filmed that movie in black and white for a reason. He thought the shower scene would be too distractingly gory in color. Audiences flocked in big numbers to see that unforgettable, deeply terrifying gray-scale masterpiece.

- Ed Kyle

leovinus · « **Reply #7 on:** 09/05/2019 08:49 pm »

Several thousand color photos is enough as ground truth. And I say that as expert in the AI field. I can built us a custom classifier from scratch to color these old pictures if necessary. Am currently traveling but love to contribute to a noble goal.

whitelancer64 · « **Reply #8 on:** 09/05/2019 08:54 pm »

Quote from: apollo16uvc on 09/05/2019 08:14 pm

Quote from: whitelancer64 on 09/05/2019 08:06 pm
Quote from: apollo16uvc on 09/05/2019 08:02 pm
Today my new high-end video card arrived.
An RTX 2080 Super by MSI, model Gaming X Trio.

The RTX lineup contains special Tensor cores which accelerate deep-learning.

Deep-learning/machine-learning/convoluted neural networks have been used to automatically colourize B/W photos by using hundreds of thousands of colour photos as a reference. Its even possible to colourize B/W video this way.

One such program is DeOldify: https://github.com/jantic/DeOldify
And this website: https://colourise.sg/

I don't know if it will work with the amount of colour photos avaible, but I will be giving it a shot. Generating a dataset from the colour photos may take several days. DeOldify advices the even higher end RTX 2080 Ti with 11 gigabytes of video memory, I got the model just below it with 'just' 8 gigabytes.

I think DeOldify is by far the best documented and wildy used image-learning program. So this is the one I will be using.

Is there a minimum number of photographs that you need to establish a database? There are tens of thousands of color pictures of the Apollo program.
I don't think there is a minimum, but the more photos it can use as a reference the better. I am not sure how it works, but I think it basically learns which identifiable features and objects have which colours and gradiants.

We got a big benefit compared to photos taken on earth: similarity. Earth photos can contain millions of different subjects, colours, objects, people, surfaces, textures. DeOldify has a pre-made set made from 1.6 million photos.

Due to obvious reasons this is much less so with Apollo photos because they were all taken in the same enviroment (Space craft or lunar surface) and usually of the space suits (deep-learning is not so good with the human body/clothing and faces.) So I think we can get away with much less photos.

I will try and contact the creator of DeOldify and ask what our chances are of decent results...

May I suggest you also contact Michael Light, who (in 1999) published a 250 page volume of Apollo photography, called Full Moon, and is an expert in the variance in the film, color correcting of variations between missions, etc.

http://www.michaellight.net/contact

https://www.hq.nasa.gov/alsj/apollocolor.html

Steve G · « **Reply #9 on:** 09/05/2019 09:27 pm »

I tried it with my own black and white photos and it's fun, but it lacks the ability to correct mistakes or wrong coloured clothings, etc. The scenery shots look more like a pastel conversion and lacks the vibrancy of true life. I had to take most into photoshop and isolate areas and correct, but its a great tool to get a colour image to manipulate rather than starting from scratch.

Steve G · « **Reply #10 on:** 09/05/2019 09:32 pm »

You can see by this Apollo 12 image it's not even close, so yes, controls to adjust the colours would be beneficial.

laszlo · « **Reply #11 on:** 09/05/2019 10:11 pm »

I have to join with Ed Kyle and ask "Why?" This has the chance of really messing with the scientific value of the images. The black and white images are a known quantity taken on film with known characteristics and processed using defined techniques. That's one reason why it's so easy to refute hoaxers who claim that various artifacts prove that everything was shot in a studio. At best, this project would provide hoaxers with a whole new data set with introduced inconsistencies. At worst, it could lead to a lot of wasted effort when someone is misled by an AI artifact into chasing an apparent scientific breakthrough. If you must do this, please put a watermark across the images indicating that they are experimental art projects.

As far as using the various archives, keep in mind that many of the images in them may need to be corrected. Unless you can trace the pedigree of the images and the processes used to set the white balance, brightness, etc. they are not really ready to be used as training data.

4throck · « **Reply #12 on:** 09/06/2019 10:02 am »

Quote from: Steve G on 09/05/2019 09:32 pm

You can see by this Apollo 12 image it's not even close...

Yes, doesn't work. I see no way of guessing the right lunar surface color for example. It changes with lighting angle, and doesn't depend on object recognition.

And as others pointed, the training dataset needs to be calibrated somehow. Colors are all over the place on Apollo photos, starting with the color charts and ending with the "gray" lunar surface., etc.

apollo16uvc · « **Reply #13 on:** 09/06/2019 11:29 am »

Yes we need to check with the colour calibration charts which films they used and which filters.

apollo16uvc · « **Reply #14 on:** 09/06/2019 03:58 pm »

Quote from: leovinus on 09/05/2019 08:49 pm

Several thousand color photos is enough as ground truth. And I say that as expert in the AI field. I can built us a custom classifier from scratch to color these old pictures if necessary. Am currently traveling but love to contribute to a noble goal.

Yes, any help would be great!

I have little expereince with Pyton (That which I have was using a custom-coded program to read NASA punch cards from photos and output them to a text file and image file.)

I don't have any expereince with machine-learning, their code, enviroments and networks. This is my first dive into it!

I am currently running through the Anaconda enviroment setup as specified in the DeOldify tutorial. I am running Jupyter on windows. Installing the Tensorboard prerequisite as we speak.

If I run into any problems i'll ask. If you talk with the creator of DeOldify you may be able to adjust some code and settings to get better results with the NASA photos. I don't know how the magic works, but it do.

As people have mentioned, I am worried how the program deals with the different sensetivities of B/W film. Each B/W has its own curve(As does colour film), which is rarely evenly sensitive over the entire light spectrum. Some films, which I believe were used on Apollo, had extended sensitivity in the ultraviolet and infrared spectrum.
This is something we got to take into account. We know which kodak film stock they used as the KODAK stock number is written on the calibration charts. Obviously the charts themselves will give clues on the curves.

I do not think machine specs will become a problem? see below:

Windows 10
Intel Core i7-9800X
32GB HyperX Predator 3200 MHz / PC4-25600 - CL16
1x Samsung 970 PRO 512GB M.2
2x 4TB Barracuda Pro
RTX 2080 Super

Regards,
Niels

apollo16uvc · « **Reply #15 on:** 09/07/2019 10:28 pm »

After two days of hard work and troubleshooting, including making a dual boot for Ubuntu to run Anaconda with Jupyter Notebook the rendering system has finally been setup.

For now I have been trying out the different settings and colourizing normal photos with the pre-made models.

Hopefully I can look into training a model with Apollo photos in the following weeks.

Some of the more decent results so far from my collection:

It generally likes colouring buildings.

It also seems to enjoy the sky, I do like the results. Its smart enough to keep the clouds white!

It does alright at differentiating between articles of clothing, animals and humans. Like some other AI I have tried it is quick to make clothing/faces red or blue. It doesn't always work.

I have noticed that contrast, clarity and brightness of the image are a contributing factor in how it renders colours. I am sure the type of film (Orthocromatic, Pancromatic) is also a contributing factor.

apollo16uvc · « **Reply #16 on:** 09/08/2019 10:09 am »

Quote from: leovinus on 09/05/2019 08:49 pm

Several thousand color photos is enough as ground truth. And I say that as expert in the AI field. I can built us a custom classifier from scratch to color these old pictures if necessary. Am currently traveling but love to contribute to a noble goal.

Some time ago Nvida showed some demo's of their AI that could interpolate frames to create a very confincing slow-mo effect. From 30fps to 60 or even 120. Any idea if this has been published as a working program? I saw some papers on it but no hint at any programs.

Last time I checked someone was talking about signing up to Nvida's deep-learning RTX developer program which may give you those tools eventually? but not right now...

I am interested in using that on some low-framerate 16mm footage.

Steve G · « **Reply #17 on:** 09/08/2019 11:47 am »

I tried about 50 old photos and am having all kinds of grief, especially with water. The RCAF 401 Squadron photo is nice except the uniforms are blue-gray not green and the sky is white instead of blue.

Steve G · « **Reply #18 on:** 09/08/2019 11:51 am »

I've had better luck on these. However, the baret is supposed to be red, the canoe red, and the girl's eyes bright blue. (The picture of the girl is far more effective in B&W in any case). So this program definitely needs some optional advance controls to tweak it's colour choice rather than it trying to out smart you.

Steve G · « **Reply #19 on:** 09/08/2019 12:09 pm »

Actually it didn't take long to isolate and colour correct the off colours. But where the river is green, the skin tones a mish-mash of peach and zombie gray, the labours would be intensive and not worth the while.