This is a set of rather lengthy video tutorials about the process of using Nuke compositing to replace the content of a cell-phone screen. I have been doing a rather huge quantity of screen replacements at work, and thought it would be useful to share the overall process, some tips, and common pitfalls to avoid.
Some screen replacement techniques have advantages over others, but none is perfect. Most smartphones and tablets these days are very reflective. This can introduce problems. Here is an overview of different approaches, with their strengths and weaknesses.
Shooting the phone with the screen not illuminated results in a nearly perfect representation of the reflections and smudges that might be on the screen. Since there is no light being emitted from the screen, putting the reflections back on top of whatever screen you put back into the phone’s screen is as simple as adding them together. Unfortunately, if there are foreground objects occluding the screen, like fingers interacting with the screen, or even hair dangling in front of the screen, getting a matte to separate the foreground object from the screen can be quite challenging and time consuming. For fingers, you would have to roto wherever the finger goes in front of the screen. For hair, the situation would be the same. If the hair is wispy and you aren’t able to pull an acceptable luminance matte from differences in brightness between the hair and the screen, this could be quite challenging. Tracking can also be difficult, because it can be hard to tell where the actual corner of the screen is when it is not illuminated. Sometimes putting small tracking markers might be necessary if only one or two screen corners are in frame. Removing these markers can prove tricky, especially if there are dynamic screen reflections jumping all over the screen.
In short: Looks the best but is the most difficult.
Illuminating the phone screen and setting it to a solid green or blur color can allow you to use chroma key techniques to extract a matte for foreground objects that might occlude the screen. Since there is light being emitted through the screen though, it can be difficult to recover reflection information to put back over the new screen. Additional problems may be also created from the sickly green of the screen’s light spilling all over your foreground, if they are in shot. The best approach is to set the screen’s brightness to be as dark as possible while still seeing the color clearly in camera. This spills less, competes with reflection less, and hopefully still allows you to extract a key for foreground objects. Having the screen illuminated also allows you to easily determine the exact corners of the screen. Tracking markers might still be necessary if only 1 or 2 corners of the screen are in frame.
With both of the previous approaches, tracking can be quite difficult. If there is a large or even marginal amount of reflections, tracking the surface of the screen with a planar tracking solution like Mocha or NukeX’s PlanarTracker can be difficult. Planar tracking works by analyzing the relative movement of many points on a surface, and calculating the movement of the surface as a planar surface in three dimensional space. When there is not enough points of detail to track, or if the surface is not static, the track can get wobbly. As you might imagine, the screens of phones don’t often provide lots of points to track. Usually it’s just a few buttons on the bottom and maybe a speaker grill at the top of the phone, and then a lot of reflections moving everywhere.
One approach to make tracking easier is to fill the screen with tracking markers, like this:
This makes a planar tracking solution quite easy! Unfortunately, it makes getting reflections back from the screen impossible. It also makes extracting occluding foreground objects difficult. Possibly more difficult than shooting with a black screen, because if the foreground object is moving quickly and has highly motion-blurred edges, these edges will likely have to be replaced because the detail of the tracking surface of the screen will show through.
Essentially, like in most areas of visual effects, there is no easy solution. The best solution varies from situation to situation, and from shot to shot. Knowing the upsides and pitfalls to each approach can be valuable in choosing which approach you will take. Here is the process for replacing a low-luminance chroma-blue cell phone screen.
Nuke Screen Replacement Tutorial Part 1: Tracking The Screen
Nuke Screen Replacement Tutorial Part 2: Compositing and Integrating the Screen Content
I have decided to include the source assets and the script that I used in this tutorial, since understandably some people do not have access to a camera to shoot footage of their own with. Download it here:
To help you out with some difficult point tracks, here is a CornerPin gizmo for Nuke that has a keyframeable offset for each corner: CornerPinOffset.