Cinematographers often spend a great deal of time making color decisions for a given film. Sometimes these choices are subtle, but often enough there are segments of films that have strong color themes, and I was curious as to how to view this in a single image.
An example of one such film is Jet Li’s Hero (shown below). There are blue scenes (stone courtyard in evening rain), red scenes (calligraphy school in the desert), and a stunning green scene (assassination attempt in the king’s chamber). On the flip side you have Saving Private Ryan which was intentionally greatly de-saturated and is essentially a dull brown/green throughout the entire film.
Extracting the Colors
Step one of my plan was to loop through a video file and extract the color information for each frame, or every hundredth frame to be more precise. Since this was the slowest part of the process, I wrote this information to a simple text file to be processed later.
import pyffmpeg
import csv
import Image
outfile = csv.writer(open(‘out.csv’, ‘w’))
stream = pyffmpeg.VideoStream()
stream.open(‘C:\movie.avi’)
i=0
while(stream.GetFrameNo(i*100)):
image = stream.GetFrameNo(i*100)
color = image.resize( (1,1), Image.ANTIALIAS).getpixel((0,0))
outfile.writerow([i,color[0],color[1],color[2]])
i=i+1
outfile.close
This produces a simple csv file with the sequence number and the red, green, and blue components of the average color of the frame.
Creating Raw Image
From the text file, I create a very simple image file that’s just a series of solid color horizontal lines.
import Image,ImageDraw
from random import randint as rint
import csv
reader = csv.reader(open(“out.csv”, “r”))
img = Image.new(“RGB”, (800,2347), “#FFFFFF”)
draw = ImageDraw.Draw(img)
for row in reader:
r,g,b = row[1], row[2], row[3]
draw.line((0,int(row[0]),800,int(row[0])), fill=(int(r),int(g),int(b)))
img.save(“out.png”, “PNG”)
This creates a pretty ugly image of the movie, but certain patterns are still pretty clear – at least to the human eye.
Clustering
The end product I had envisioned would be large bands of a handful of colors. For example, in the image of Hero above you can see it starts with dark blue, then moves into some reds, and then back into lighter blues, etc. I tried some various clustering algorithms within Python, but none yielded the result I was looking for. Mostly I would cluster the colors to say 5 base colors, and then redraw the image using only those colors. This would still yield a lot of small bands, so I tried different algorithms to remove bands of certain sizes, or even re-cluster the clustered results. I simply couldn’t get to what I was imagining.
So sadly, I gave up on coding this part, and jumped into Photoshop. Using the Cutout filter, I quickly reached the result I was hoping for. Using 4 or 5 levels, large solid bands are produced as expected. I did a little research into the Cutout algorithm to try to recreate it it, but that didn’t lead too far. So for now, as sad as it makes me, the final step of the process requires a single Photoshop filter.
In short, we take a movie file, loop through every 100 frames or so, and draw a line of that color to a new image. Then, once we have a long image of solid color lines, we use Photoshop to more-or-less band this image into only a few common solid colors.
Final Output
Saving Private Ryan – As expected, not terribly exciting. Lots of de-saturated greens.
The Wizard of Oz – I expected a little more from this one, but you can still see the big chapters. Intro, Emerald City, Witch’s castle, back to Emerald City. Also, the beginning is not gray because the “black and white” portions are actually sepia-toned.
Curse of the Golden Flower – Who would have thought there’d be so much gold!?
Jet Li’s Hero – My favorite result, and the movie I was most looking forward to processing because of how visually stunning the movie is.
Filed under: Art, Coding by Brandon
3 Comments »
