The Pixilation Shoot
On Saturday 2006-11-04, the first shoot of the Trouble With Unicorns took place. This was for the opening sequence of the project, which is in a 2D-animation visual style. Instead of characters made of cutout paper or other materials, (which might be seen as the traditional 2D animation style), this sequence is going to have live people animated in a stop-motion animation technique called pixilation. This will give the characters the surface appearance of being real, but since they are ‘pixilated,’ their motions will be interrupted and jagged, giving them an animated aesthetic. The characters were captured performing their motions in front of a chroma screen, so that they can be “cut out” and composited in with backgrounds at a later point in the post-production process.

Technical Considerations of Chroma Key Shooting
Here is a still of our makeshift chroma keying setup.

The large green wall you see in the background is actually 6 flats (pieces of plywood stood upright and screwed to stands) taped together. We painted the flats with near chroma-green paint from Ace Hardware. On commercial film sets, chroma screens are often painted with special chroma paint. This usually costs about 60 to 70 dollars per gallon. For the budget-minded individual, one could acquire near-equivalent chroma green paint from your nearest hardware store, for much less money. The amount of gloss on the paint that you buy can be an important consideration. If you buy high-gloss paint, you will get reflections and glare on the surface of your green-screen, which should be avoided at all costs. You might also get an increase in green light bounced back at your actors, which should also be avoided. A full matte green paint on the other hand, might bring about an increase in the amount of light necessary to illuminate the green screen easily. Taking into account these considerations, we decided to get one step above matte in the “matte-to-gloss” scale, which at the hardware store we went to was termed “eggshell gloss”.

You might also notice that our actor is standing on a wrinkled combination of clothes… er, cloths. Yes. These clothes represent somewhat a range of colors, from darker green to lighter green to blue. Contrary to what might seem wrong at first, you can actually key two colors (green and blue) simultaniously, fairly effectively. While not desirable, as this adds more complexity to the post-production workflow, it can be effective. We were forced to resort to it in our case, where we were running out of time, and had to find many strips of cloth to put on the floor for keying.

You might notice in the above screenshot that our actor is very far away from the chroma green wall. We did this so that we would get as little ‘spill’ as possible. Spill is green (or blue) light reflected back from the chroma screens onto the subject, and can create difficulties with color corection in post, or even unusable footage. Lighting is very important in Chroma Key shooting. There is constant battle between even lighting on the chroma screens, and shadows cast onto the chroma screens by the subjects. It is difficult to maintain a balance between even lighting on the screens and desireable lighting on the actors. There are a list of tips for chroma key shooting, that interested readers might peruse.

Technical Considerations of Image Capture
Shooting in the style of pixilation brings about unique challenges for the actor. Essentially, you are using people as puppets. Unlike puppets, people can move themselves. Also unlike puppets, people are more difficult to pose and leave in one position, because they tire easily. Therefore, when shooting pixilation, it is best if the actor has to pause and hold one position as little as possible. For the shoot we were able to acquire a Canon Digital Rebel XTi (400d), from the excellent independent filmmaker’s resource, Costco. Our initial plan was to shoot in Canon’s digital RAW format, so as to preserve as much of the color information as possible. However, we ended up shooting at the highest resolution JPEG setting instead, for the majority of the shoot. The Digital Rebel XTi can shoot at 3 frames per second, until its memory buffer fills, and it gets behind in writing the data to the Compact Flash card. When shooting raw files, the buffer can hold 10 images before it fills. When shooting Jpeg (large/fine), the buffer can hold 27 images. To shoot in RAW would have put significant strain on our actors, because every 10 frames, they would have to freeze mid-motion and hold their position until the buffer filled.

When shooting for chroma key (see discussion below), color information is just as important as luminance, or brightness information. Most video codecs (DV, for example), and the JPEG still image compression algorithms, apply a method of compression known as Chroma Subsampling. This is basically a technique of compressing the color information more than the brightness information, because our eyes are less sensitive to a loss of detail in the colorspace of an image than in the luminance-space. DV has a Chroma Subsampling Rate of 4:1:1. JPEG has an inferior chroma subsampling rate of 4:2:0, which basically means that there is half as much resolution both horizontally and vertically in the color information. The RAW format has no color compression at all, and thus its effective chroma subsampling rate is 4:4:4, meaning that there is no compression of the color channel.

What does all this mean for Chroma Keying? This means that if we were shooting at SD resolution (720×480 pixels), in the JPEG format, our effective image resolution for keying would actually be half of that (360×240). This would result in a rather blocky key!

Since an 8-bit Morgan the Unicorn is not our goal, it is fortunate that the Rebel XTi can shoot at 10 Megapixels of Resolution, which comes out to 3,888×2592 pixels, which is about equivalent to the 4K standard resolution that Film Studios scan 35mm film at for digital intermediates. Because our final output format for this sequence is going to be HD 1080p24 resolution (with eventual down-conversion to 480p24 to mesh with the remainder of the project, which will be shot on the Panasonic DVX100A), we reasoned that even with the 1/2 effective resolution in color brought about by the JPEG compression, we would have plenty of resolution to bring about an effective key.

We will see just how difficult it will be when we actually start compositing the pixilation characters and the backgrounds we are scheduled to shoot this coming Sunday (11-26). For the backgrounds Brad Hutchinson’s and my own plan of attack is to acquire 32bitHDRI images to work with in After Effects (in which program we will be doing the compositing and animation), by shooting in RAW and bracketing 4-6 exposures of each image we take of background materials, and then using the “Merge to HDR” command in Photoshop CS2. While this may seem like overkill (and probably is), it will be an interesting learning experience. More updates on technical matters relating to the Trouble with Unicorns shoot to follow at a later time.

~Jed Smith – Editor, Visual Effects Collaborator

Also present in the following video is a very rough composite of some of that gross mud with worms crawling around in it warped with a cheesy liquify effect (I would do this better if this were a serious composite), and some TV static generated with the Bad TV plugin from the excellent Tinderbox plugin collection. This just a very short clip, so don’t spend too much time waching it…

[See post to watch Flash video]

I was thinking about the more not so perfect footage, and in the context of my approach and the style of this crazy music video, nothing is beyond usefulness. Thus, I will use and manipulate pretty much any footage I capture, even if it doesn’t look perfect, as long as I can exploit its imperfections towards a useful and legitamate goal.

As you can see, the chroma screens are underexposed, as are the subjects, and as a result the image is somewhat noisy, which doesn’t help matters with the already artifacted DV source. Also, there are lighting inconsistencies in the chroma screen. Also, like an idiot, I decided it would be a good idea to backlight the subjects with a blue-gel; I thought this would make my planned key more believable. I thought that because the blue light was from a source other than spill from the bluescreen, it would be okay. It’s not okay. Blue is blue! That must be kept in mind in the future.

Anyway, using the magic of the Keylight plugin for After Effects (read this awesome tutorial, anyone who might be interested in seeing how it can function), and some creative masking, I was able to pull a decent key.

Pixels should not be this large or this blocky!

This is slightly better… I achieved this by tweaking the settings of keylight and adding a subtle matte-choker.

This is the key without the mask on the circuit board. As you can see, the blue spilled onto the board from my backlight (arr!) is causing the keying plugin to make the board transparent, something that we don’t want to happen.

Since the board is stationary within the shot for a large period of time, I duplicated the layer and applied a mask to the board, effectively making the key only affect areas outside of the circuit board area. The results look alright, but I will be tweaking it more, possibly using the technique of blurring an adjustment layer and applying a transfer mode of “color”, in order to blur the chroma slightly without blurring the luminance, and thus counteract the DV artifacts somewhat, as outlined in the keylight tutorial.

Today I enthusiastically enjoyed the DVX100A proficiency with Ben, Alex, Dre, and the almighty Dave Cramton. Tomorrow, I will pick up my work order for it, and break open that pack of 5 new Panasonic Master Quality DV tapes, leaving the solitary Sony DVCAM tape that I bought solitary… and unused. (take that, DVCAM!). I will be doing tests on Friday to see if the 24p advanced mode can deliver the realistic progressive psuedo-filmlike motion and blur that I’m looking for, as well as doing a test pulldown, edit, reverse pulldown, and author to progressive-scan DVD. I will of course post the results here.

In other news, I have been drooling all over my monitor, reading about the Panasonic AG-HVX200, the HD successer to the DVX100, which records in 4:2:2 DVCPRO on P2 cards at a data rate of 100MB/s, and is capable of recording real variable framerates, much like the Varicam, except about $60,000 cheaper at $6,000. I’m excited by the fact that, in 6 or 7 years, this technology will be somewhat outdated, and I will probably be able to buy one for under 2 thousand dollars.

The process of animation is going well, and with experience, I get better at making good lighting, and better at making good motion (at least it seems like I’m getting better). In approximately 24 hours of solid animating, I have shot ~1725 frames, not including the tests done in my home lab (which I may still use for some things), which comes out to about 2 minutes and 24 seconds. There is a long way to go, but at least progress is being made.

Following are some production stills, for your consideration. The purpose of these photos is for you all to get a visual sense for what is going on, as well as some of the problems I have faced, and solutions I have eximated. Therefore, apologies for the briefness of the text.

“Lumbar Laura” construction:
This is the nickname I have given the newest armature-puppet I have been constructing. She is quite advanced compared to the previous model. Thanks to the wonderful technology of the drill, she has limbs which bend in all directions, connected and reinforced with twisted wire. The drill was for making holes in the limbs to shove wire into and then glue. This is the best way I have found to make a strong and sturdy bond between two limb-components.

Her previously useless floppy unsupportive feet are now gone, replaced with a newer better metal model! This allows for much better support while standing, and a good flat wide structure to hot-glue to things.

Surgury was performed on her spinal column, to allow friction against free movement, in order that she might pose in any position!

This was made possible through the magic of corn syrup! (unfortunatly, the only corn syrup I had was in the form of stage blood…)

The older model’s hip-joints were also reinforced This unfortunately meant that she had to suffer through an assuredly uncomfortable experience with hot melted pine resin being melted onto the connexions between her joints.

The pine resin tends to dry and form a locked connection between the joints. Corn syrup remains at least somewhat viscous over a period of time. When combined, these two sticky things look pretty good!

This inhibition of the leg-hip joints allowed the puppet to walk in 12 poses per second!

The ability to walk was made possible through industrious use of the magical substance Hot Glue!

This shot was very difficult. I had never animated a puppet walking before. The ground was covered in leaves, which had a malicious tendency to move a great deal whenever I approached with the glue-gun or screwdriver..

For the first few steps, I put a litlte bit too much hot-glue on her feet, and had difficulty moving them for each step.

Pleasantly, carressing of a fern was had.

Here, you can see the green-screen in action. This required careful lighting, but will hopefully result in flexibility on, with the composited addition of… sky, and the various exciting expressive possibilities innate in that idea.

Ken’s feet were also reinforced with injections of wire into joints.

Ken is now a manly Man. Just look at his junk… wow.

Tomorrow is the user group meeting for various glorious equipment, and I am ripe with fear that I will not be able to express myself well (having done a great deal more animating than talking in the last few days), and will be despised and denied access to said glorious equipment.

After that ordeal is over, I will proceed to schedule more time in the 3d animation lab for this weekend, and work on my soundtrack, which is in need of attention… focused and dedicated attention. That is all for now.

::placeholder for all animation footage shot so far, edited in a narrative linear fashion::