How to Rotoscope in After Effects (The Basics) – After Effects Tutorial

After Effects 25/10/2019 4 min read

So what is rotoscoping?  

Well, this term can mean a couple slightly different things in different fields, but in video editing it’s basically the process of drawing around an object, character, or other part of your scene so that you can isolate it. Usually people will use this term when they’re talking about tracing something that’s moving and that doesn’t have a simple geometric shape.

A really typical example would be drawing a mask around a person so that you can separate them from the scene that their in. This has a huge amount of application to lots of other processes and tools inside of After Effects, so learning rotoscoping early on in your After Effects career is pretty essential.

So let’s go through an example together.   

How To Rotoscope

Say for example that we want our subject to have text behind them, but putting the text layer beneath our clip just makes it invisible. We need to actually tell After Effects that our subject is unique from the rest of the scene. This is where rotoscoping would come in handy.  

To start, grab your pen tool and make sure that your clip is highlighted. Having it highlighted will ensure that when you start to draw a shape around your subject, you’re actually masking them out as opposed to just creating a solid shape Layer.

So with the pen tool, whenever you click, you create a new point, and wherever you click next will connect it to the previous point. Do this around the entire perimeter of your object in order to mimic their shape. A great way to get really detailed with this is to have three shortcut keys memorized. Z, for the zoom tool. With this tool you can click and drag left or right to zoom in or out. Then hit H, to pull up the hand tool, which will let you pull around your frame and get perspective without touching any of the elements in your composition. And finally, hit G to go back to your pen tool and keep going. Another thing to note is that if you simple click and move on it will make a perfectly sharp angle to your next mask point. But if you click and drag, you can give curvature to your points. This can help if you’re masking around circular or rounded objects.  

Now that you’ve roto’d your subject, click the first tracking dot and connect the shape to the beginning and you’ll create your mask. From here your subject should be separated from the rest of the scene, but you’ll also see that the rest of the scene goes away. This is because a mask tells After Effects that that’s the only thing you want it to show. You can inverse the mask pretty easily if you want the opposite effect by going down to your dropdown, mask, and then hitting inverse.  

But what we actually want is for everything to still look the same but just to be able to work with our subject in isolation. To do that, we’re going to duplicate our video layer by clicking it, then hitting control or command D. Now we should have two identical video layers. And if we go to the bottom layer, go to the mask, and then click and delete it, we can see that our scene looks completely normal, but it’s actually still got the mask that we originally set up.  

Great, now let’s bring in some text and then move it in between our two video layer. And we can see that our effect looks like our subject is actually closer to use than the text is. But, it doesn’t look perfect, as there’s some places where the mask is cutting into our subject and other places where it’s too far away from our subject and cuts out the text. While finessing these points is a part of the task of rotoscoping, what can really help is feathering your mask. Go down to your mask options and increase or decrease the feathering depending on your situation and what looks best for your shot. For us, increasing the feathering a bit makes things look really natural.  

But here’s the problem. We’ve done this for the frame that we’re currently on, but this is a video, and if we play our video, the mask stays exactly where it is and we’re left with a terrible looking effect. Ladies and gentlemen, the reason a lot of people don’t like rotoscoping is because it needs to be accounted for in every single frame of the video that it’s needed. So what this looks like is bringing it to the frame where you first started working on it, hitting the keyframe stopwatch for the mask, and then going frame by frame to move your mask around so that it fits right for each individual frame.  

That can seem insane to some people, so let me give you some pointers to hopefully make the whole process a lot more manageable.  

If you’ve got a full sized keyboard, you can actually go forwards and backwards by 1 frame at a time by hitting the page up and page down buttons. This can help you quickly navigate forwards and backwards in an instant

Next, every time you make a change, as long as your stopwatch is highlighted blue, After Effects will automatically make a keyframe when you make a change to a new frame, but what it also does is average the changes made over the course of any frames where there are no keyframes. So if you have very simple movement of your subject over the course of a few seconds, you can simply have your starting frame, move way down the line, and adjust for the end frame. Depending on your shot, the mask might actually work if your motion is consistent and linear. If it’s not, dive into where it doesn’t meet your expectations. You can also use this method to a smaller degree for between 5-10 frames at a time to significantly cut down the amount of finessing you’ll have to do.  

But guys, that’s been just a basic overview of how to rotoscope in After Effects. Like I mentioned before, this is a skill that’s essential the deeper you get into After Effects, so the more you practice, the easier you’ll make things for yourself in the future. But that’s it for me guys. Feel free to check out all our other tutorials over at Motionarray.com.

We hope you found this video helpful.  If you did, we’ve got lots of other tutorials for Premiere Pro, After Effects, and filmmaking in general! If you have any questions, let us know in the comment section below.