Jan 19 / Christian Bull

Markers of Mastery: Exploring the Art and Science of VFX Tracking

DOTS! DOTS EVERYWHERE!

DOTS! DOTS EVERYWHERE!


It’s customary for every behind-the-scenes breakdown of VFX to showcase someone or something covered in dots or crosses, creating a magical transformation into amazing VFX!

This week, let’s explore the essence of these dots—what they are, why they exist, and when they shouldn’t.

I don’t think I’m gonna blow your mind if I tell you that they are tracking markers, which means they are… marks. Made for tracking.

This could be done with a fancy professional tracking marker, a laser dot, or…a normal marker pen. Look at the Bill Nighy tracking points above. It’s just a pen, but Davey Jones turned out alright!

Tracking just means following something that you’ve filmed. A tracking marker is just something that creates enough contrast for software to follow it.

And why do you need to follow points on your footage? There’s so many reasons!

Tracking Markers in Practice
Let’s take a look at some examples –
  • Blemish Removal – Notice how perfect actor’s skin is in big budget films? They still have spots, like the rest of us. But theirs get tracked, and that allows them to be replaced with a touch of painting (standard “Photoshop cloning”, but in motion).
  • Motion graphics – Maybe you want some text following someone in shot. You’ll need to track something on that someone to create a “path”, which your text can follow!
  • Cameras – “Camera tracking” is an odd one, since you can’t see the camera that’s doing the filming. So instead you track markers in the environment that the camera is filming. By analysing how those markers move, the tracking software can work out what the movement of the real camera would have been. It can then create a digital equivalent, which you can use to “film” your digital elements, and have them line up perfectly with your filmed footage. If you’re being really smart, measure the distance between tracking markers on set. That data can help your camera track.
  • Billboard/Sign replacement – Called “planar” tracking, this process involves tracking a flat plane, so that it can be replaced with something else. So let’s say you’re filming locally, but the film is set in another country. Sign posts will give the game away, so you would track them, and replace them with images/writing from the country you’re pretending to be in
  • Motion Capture – Although motion capture suits don’t use tracking markers, facial tracking (such as the Bill Nighy example) does. You use markers to follow the movement of the face, and that movement drives your digital character. AI based motion capture uses video, which doesn’t need tracking markers, but works much better if you’re wearing patterned clothing (because that makes tracking markers!)

I could go on, and on, and on. There’s so many times when you’ll want to track a point so that it can be replaced with something else!

The Art and Science of Tracking Markers
As simple as the concept may be, there is an art and a science to understanding and using tracking markers, and there are consequences for not using them properly.

When to use tracking markers
  • When you want to track something.
  • If you only need to track the movement of a point, you only need one. If you need to track the movement of a plane, you’ll need at least 3. If you want to track the movement of something 3 dimensional, you’ll need more (Blender insists on 8)
  • …that’s it! BUT always use them if you know you’ll need to track something. Tracking motion without them can be a nightmare

Robert Downey Jr as Iron Man. Don’t you dare use tracking markers on that beautiful face. They wouldn’t help you anyway…


When not to use tracking markers
  • If the shot already has enough texture in it for you to track off (something like a brick wall will probably create plenty of contrast. Just track off that, if you can)
  • If the area will be out of focus. You will probably hit this problem a lot in your VFX career – you spend an hour putting tracking markers around the set, and then realize it’s getting filmed with a shallow depth of field, and everything is out of focus, meaning that there’s not enough sharp contrast to track off!
  • If the area is moving in a way that you don’t want. If you want to have an Iron Man style helmet/mask, you want to track the head, not the face, because the helmet needs to follow the movement of the head. If you have markers on the actors forehead, and he or she gets surprised, the tracking markers will move up, causing the helmet to move up too, and slide off the actor’s face!
  • On a visually sensitive area. The tracking marker will need to be “painted out”, often by cloning another area of the image. That’s fine for a white wall, but right in the corner of someone’s mouth is a tricky area to paint even for a master painter! If you’re not confident of being able to clone or paint the marker out, stick it somewhere else
  • When you can’t see them. That seems silly, but I’ve had to track markers in horror films before when everything is black! If things are dark, use LEDs, and track those.

I hope you’ve understood the point of this week’s information, and it’s helping you get on track with your VFX journey. Every week I try to make my mark with this newsletter.

As always, let me know if you have any questions, or if I’ve missed any potential puns.