The Ultimate Guide to Rotoscoping in DaVinci Resolve 17

In this tutorial, we will use a DaVinci Resolve rotoscoping technique to create a composited image where text appears to be between a subject and the background of a single video clip. To create a seamless effect, we will track the text along with the background motion so it appears to be part of the video. Next, we will go further with advanced techniques for rotoscoping complex movement.

Part 1: What is Rotoscoping?

Rotoscoping is the process of tracing over live-action footage frame by frame. This process is commonly used for compositing multiple images or layers together into one final video output. For example, you would need to use a rotoscope technique to remove unwanted objects from a video or to separate objects from a background in order to insert additional elements to the video. This is achieved by creating a mask around the subject and animating the mask to follow the movement in the video.

Part 2: Learn Simple Text Rotoscoping in DaVinci Resolve 17

Step 1: Mask your Subject using the Color Page

1. Select the clip you’re working with, and navigate to the Color page at the bottom of the screen.
2. Right-click the clip in the Nodes Editor and choose Add Node > Add Serial. Serial nodes are the basic building blocks for most effects and compositions in DaVinci Resolve.
3. Click the Window palette and select the pen tool. 
4. Check the Interactive Mode checkbox. Interactive mode allows you to manually remove, add, or edit tracking points.
5. Click the Tracker palette and ensure your playhead is positioned at the first frame of your clip. Use the pen tool to draw a mask around your subject.

6. Click Track Forward. DaVinci Resolve will automatically generate tracking points. While this will provide a great starting point, you will likely need to refine your mask.
7. Use the arrow keys or drag the playhead to scrub through your clip to ensure your mask animates with your subject. Identify any parts in the clip where the mask no longer lines up with your subject.
8. Click and drag individual points or shift-click to draw a bounding box around a group of points to shift several at once. Remove any extra points with the delete key.
9. To remove the background, right-click any empty space in the Node Editor and choose Add Alpha Output. The output appears as a blue circle in the Node Editor.
10. Click and drag to connect the key output (blue square) from your Serial Node to the Alpha Output.

Step 2: Create a Rotoscope Text Effect in the Fusion Page

1. Navigate to the Edit page. Add a copy of your original clip on a track below your masked clip. Now you can add pretty much anything between the 2 tracks!
2. Right-click your original clip on the bottom layer and choose Open in Fusion Page.
3. To add the Tracker node, select the MediaIn node and press Shift + Spacebar on the keyboard to bring up the Select tool. Search for “Tracker” and choose Add.
4. Position your Tracker by clicking and dragging it from the upper right corner. Find a spot in the background near your subject. Look for a point with some contrast or interest for your tracking point.
5. The smaller rectangle is what the tracker will track, and the larger rectangle is the area in which it will look.
6. In the Inspector, select Track Forward.

7. Once DaVinci Resolve has tracked through your clip, click the Operation tab in the Inspector. Set the Operation to Match Move and ensure the merge is FG over BG.
8. Create a Text node by dragging the Text icon from the toolbar to the Nodes Editor. 
9. Drag the output from the Text node to the Tracker node to connect them.
10. Add your text by selecting the Text node and typing it in the Inspector. Make any size, font, or color adjustments as well. Drag to position the text in the frame.
11. Return to the Edit page or Color page to check out your text effect!

Here is the final result:

Part 3: Learn Complex Rotoscoping in DaVinci Resolve 17

In reality, most DaVinci Resolve rotoscoping tasks are not as simple as the example above. In a case where your video clip has significant movement, try some of these tips and tricks:

Use Polygon Nodes

Masking with a Polygon node allows you to create much more complex shapes with curves and defined edges. Steps to create a mask with a Polygon node:

1. In the Fusion page, click and drag the Polygon tool to the Nodes Editor to create a Polygon node.
2. Trace your subject with the pen tool. For simplicity, try to keep the number of points in your mask to a minimum.
3. Use the Bezier handles to refine your mask shape.

Use Multiple Masks

Working with multiple simpler masks is typically more manageable than working with one complex mask. Steps to create multiple masks:

1. Observe the movements of your subject to determine how many masks are needed. For instance, you may want to use a separate mask for a subject’s arm. 
2. On the Fusion page, click and drag the Polygon tool to the Nodes Editor for each mask needed.

Tip: Make sure you name each node so you can easily tell them apart.

3. Connect your nodes to one another by dragging the output from one node to the next. The last mask will ultimately connect to the MediaIn node.

Clean Up Your Mask with Soft Edge and Border Width

Edges can be tricky around textures like hair and fabric, but there are a few tools to help out in the Inspector under the Controls tab.

1. Drag your mask node to the MediaIn node to see the alpha channel and check out your mask edges. 
2. If some background is visible in your mask, select the Mask node and adjust the Border Width slider in the inspector. Lowering the border Width can help remove any visible background by narrowing the border.
3. For uneven edges like hair, it may be helpful to blur the mask edge with Soft Edge. Increase the slider to smooth your mask edge. Note that only a slight adjustment should be needed; too much Soft Edge can make the image appear blurry.

Remove Sections of Your Mask

When working with a complex subject with a lot of movement, there may be sections of your mask where the background shows through. You can resolve this by creating a small mask and subtracting it. Here are the steps:

1. Create a new Polygon node and position it between your other nodes and your MediaIn.
2. Trace the shape you want to subtract with the pen tool.
3. With your Mask node selected, change the Paint Mode in the Inspector to Subtract.

Rotoscoping is a great tool to have due to its many practical applications in video production. Use rotoscoping techniques to remove unwanted elements or to layer text or special effects in your videos. DaVinci Resolve 17 includes multiple tools in the Color page and Fusion page for creating seamless rotoscope effects.