How Centropolis FX Creates Visual Effects

The Patriot contains an incredible range of visual effects created by the team at Centropolis FX. You can click here to learn more about the movie.

Understanding Effects

CFX began life as a company in 1996 and today has over 100 employees dedicated to the creation of digital visual effects in a wide variety of Hollywood movies. The team consists of artists, technicians, producers and managers who work together to create scenes that are realistic, stunning and totally convincing to the audience.

The word digital in the phrase digital visual effects means that CFX primarily uses computer hardware and software to create its effects. The general term used in the industry for what CFX does is CG, short for computer-generated. For example, when talking to the artists you will hear them say things like, "That entire scene is CG," or "Those are all CG soldiers," or "The actors are real, but everything else is CG." Computer-generated effects make imaginary characters like Godzilla possible, and they also create almost every effect that used to be done using models. The advantages of CG effects are their realism, flexibility and relatively low cost (compared to the alternatives).

The team at CFX works with the movie's director both during filming (production) and then extensively after filming (post-production) to create the effects. Involvement during production helps the director make creative decisions so that the effects can be integrated into the film more easily, and also allows CFX to add different markers and other features to each scene to make post-production work easier. For example, in certain scenes the camera might be fitted with encoders that will allow for easier integration of the effects that CFX creates. During post-production, the director works extensively with CFX to make sure that the effects in each shot have exactly the right look for the film.

CFX uses a variety of tools and techniques to create visual effects. The different capabilities of the company include:

Scanning and printing - CFX has the equipment to scan film at extremely high resolution (up to 12,750,000 dots per frame), store it, manipulate it digitally and then write it back out to film at the same resolution. A Kodak Cineon film scanner
3-D Character modeling and animation - The artists at CFX can create and realistically animate characters and then integrate them into scenes -- everything from Godzilla to a platoon of British soldiers can be added to a film. CFX can also create ships, water, background scenery, clouds, flags, buildings, vehicles, explosions and so on. These CG elements are totally realistic and are seamlessly integrated either into CG backdrops or filmed scenes. 3-D CG soldiers like these play a big part in "The Patriot"
3-D camera tracking - In order to lay 3-D characters into a filmed scene, there must be a model of how the camera moves and zooms when the scene was shot. This model can be created by adding encoders to the camera, or it can be created after the fact. In either case, CFX creates a 3-D model of the scene and how the camera moves within it.
Rotoscoping - Rotoscoping is the process of outlining and "lifting" elements of a filmed scene off the frame so that other elements can be added to the frame either in front of or behind the rotoscoped elements. We will see several demonstrations in the sections below. A rotoscoped section of a scene
Painting - Painting involves the creation of imaginary scenery. It also involves what was once called "airbrushing" -- the process of adding or removing things from a scene.
2-D Compositing - Compositing is the act of adding all of the different elements to a final scene. In the examples below we will see that many scenes have a dozen or more layers that are all added to create the final scene.

Combined together, these techniques allow a visual effects team to create nearly anything that the director can imagine. It is an amazing set of tools!

An Example

The easiest way to understand what a visual effects team can do is to look at an example. The first example that we will use involves a surprising number of techniques to completely change the landscape from a wooded field to a seaside town! This is a shot from the movie "The Patriot."

To get a feeling for the total transformation, take a look at these two short videos. The first shows the original shot as filmed. The second shows the scene as it appears in the movie:

The shot arrives as a piece of film, and the first step is to scan this film into digital form so it can be manipulated. Scanning is done frame by frame (movie film is shot at 24 frames per second), and then each frame is stored as an individual full-color image with a resolution of 2,048 by 1,556 pixels. The section of video shown in the two previous videos is 20 seconds long, or about 480 frames.

First, a variety of things might be done to the original shot to clean it up, correct the color and so on. Since the camera pans across the scene in this shot, the next step is to build a 3-D model of the camera so that all of the visual effects the team will create will mesh with the original scene exactly.

One of the first steps taken to add visual effects to the scene is the rotoscoping. An artist sits at a computer and, frame by frame, outlines the portion of the original shot that will be used in the final version. The breastworks, a portion of the field and several of the running soldiers will all be used, but the explosion seen on the left and everything else will be removed. The artist will outline these elements and essentially "lift" them out of the frame.

Another camera crew has created an ocean shot -- in this case the ocean is not far from the CFX studio in Hollywood! Once the ocean shot is cleaned up and color-corrected, it and the rotoscoped scene can be integrated:

The rotoscoped scene added to the ocean shot

This video shows the field replaced by water.

Another artist in the paint department has been working on a matte painting of the town. This is a high-resolution digital image created using a painting/illustration package.

Once the town is added in, the scene looks like this:

This video shows the town integrated into the shot

In this shot there are a number of boats encircling the town and firing on it. The boats are all computer generated. Each one is modeled and then added to the shot:

This video shows the boats on the water

The cannon fire for the boats is its own stand-alone effect that is created separately and then added to the shot.

This video shows the boats' cannon fire

A variety of other effects are added to the shot, including things like smoke over the town, people in the town, a large explosion on the right, etc.

The final step in the process is compositing all of the different components of the shot (the rotoscoped foreground, the water, the town, people in the town, the boats, the cannon fire, the smoke, the explosion, birds flying overhead and so on) layer by layer to create the final shot as it will appear in the movie. Once digitally composited, the shot is written back to film so that it can be spliced into the movie.

This video montage shows all the effects

The small images used here don't really do justice to the shot -- when you see "The Patriot" on a huge screen in a theater, you can see the incredible detail that has been added to make this shot look totally realistic.

As you might imagine, creating a scene like this is time-consuming. Even though the shot only appears on screen for 20 seconds, hundreds of hours go into the visual effects to create a rich, convincing visual image. In this video, several of the artists who worked on the shot describe how long the shot took. It is an amazing process that involves an entire team of people, and this process may be repeated several hundred times in a single film! In "The Patriot," 150 shots in the film involve visual effects like these.

Overview of the Visual Effects Process

A typical movie might have 1,000 to 1,500 shots. A shot might be one or two seconds long, or 30 to 60 seconds long. In "The Patriot," the longest shot that the team handled was about 1,000 frames. A given scene in a movie might be filmed with a number of different cameras so that there are wide vistas, close-ups, changes of perspective and so on. In the final movie, these different viewpoints are mixed together to create the scene. Therefore a single scene might contain dozens of individual shots. By splicing all of the shots together in the correct order you create the complete movie.

Hollywood movies are typically shot on 35mm film at 24 frames per second. The first step of the visual effects process is deciding which of the shots need to have visual effects applied to them. For example, in "The Patriot," 150 of the shots in the film required visual effects. The rolls of film for these shots are sent to CFX to be scanned. CFX uses a Kodak Cineon scanner capable of up to 4,096 dots of resolution (4,096 X 3,112 dots per frame). "The Patriot" was scanned at 2K resolution (2,048 x 1,556 dots per frame), and the 150 shots and all of the intermediate files consumed about 1.6 terabytes of disk space.

Once it is digitized, a shot may go through a number of 3-D artists. A big part of realistic computer generated effects is the creation of 3-D models and characters, a process that includes a number of steps. These include:

3-D Tracking - The tracking department uses markers added to the scene to create a 3-D model of the scene and then a 3-D camera. The goal is for the 3-D camera to exactly mimic the motion of the real camera so that the 3-D elements added to the scene look right and move correctly as the camera moves in the actual scene.
3-D modeling - A 3-D model is a collection of shapes that form the outside of the object being modeled. For example, the soldiers in "The Patriot" are formed from a combination of spheres, cylinders and other shapes that are molded on the computer screen into exactly the right configuration to look like a soldier.
3-D setup - Setup is the process of adding a "skeleton" of bones and joints to a 3-D model so that the different shapes in the model move correctly with respect to one another. In some cases the bones and joints are created by hand. In other cases they come from motion capture data (see below). Motion Capture To gather motion capture data, an actor is fitted with a suit that has reflective markers or lights at every joint. The actor moves on a special stage and 3-D cameras watch the actor from a number of different angles. Computer software is then able to track all of the markers and, with the help of a technician, bind them together into a stick figure that accurately duplicates the motion of the actor. The stick figure is the bones and joints that then control and animate a 3-D model. This video shows the transformation of motion capture data to a realistic soldier. A variety of different poses and activities are captured to provide variety, as shown in this video of four soldiers.
3-D animation - In the animation process, an artist choreographs the movement of a 3-D character.
2-D painting - Painting is used extensively by any visual effects team. With painting the team can create matte paintings for backdrops; paint out wires, harnesses, brackets and other safety equipment; paint over the "holes" sometimes created by rotoscoping; touch up things like grass that has lawn mower tracks in it (see Example 2 below)
2-D compositing - Compositing is the process of adding all of the different layers of a shot together to create the final shot. In the example shown in the previous section, the composited layers included the rotoscoped piece of the original shot, the water, the painting of the town, the boats on the water, the cannon fire from the boats, the people in the town, the smoke over the town and a new explosion. The compositing artist incorporates all of the elements in the right order so they overlay each other properly to create the final shot.

A visual effects team is responsible for all of the effects shots in a single film. At CFX, a team consists of a producer, several supervisors (for example a 3-D supervisor, a 2-D supervisor, etc.) and a number of artists. One of the first tasks for the team is the research and development process. For example, in the production of "The Patriot," a fair amount of R&D time was spent on creating soldiers and then groups of soldiers that look realistic when added to a scene. The team then uses what it creates during the R&D process to manipulate the shots it is responsible for. This is not a quick process -- a film takes about a year to create.

With the help of the supervisors and artists on the team as well as the film's director, the producer's job is to look at all of the shots, understand what the director wants to change in each one and then estimate the amount of time that all of the tasks will take. From these estimates a huge production schedule is built in a tool like Microsoft Project and the team executes against this schedule.

Aspect Ratios

The 35mm film that is used to film movies is the same as the 35mm film you use in a 35mm camera. Instead of being packaged in small 24 or 36 frame rolls, movie film comes in big reels, but it is exactly the same stuff. A movie camera exposes the film at a rate of 24 frames per second.

35mm film is 35mm wide and includes the sprocket holes along both edges. It has 24mm of usable film width once the space for the sprocket holes is subtracted. Therefore a typical frame of a film might be 24mm by 18mm -- it has an aspect ratio of 4:3 when shot.

When projected on a screen in a movie theater, the aspect ratio is much wider -- 16:9 for example. This is much wider than the 4:3 aspect ratio of the film. To compensate, a movie camera uses a special anamorphic lens that compresses a 16:9 image onto the 4:3 film. If you look at the film, therefore, a frame of a movie looks odd -- everything is tall and thin. Once the film is projected through a similar (but reversed) lens at the movie theater, it stretches out to its full 16:9 width and looks normal on the screen.

The Technology

Digital visual effects at the Hollywood level require an incredible technology infrastructure that includes both hardware and software. The machine room at CFX shows just how extensive the hardware investment is. The team needs massive amounts of hardware for four reasons:

The scanned film and the different layers that the team creates require gigantic amounts of disk space. A single frame of a film, once scanned and stored on a disk, consumes on the order of 10 megabytes of disk space. All of the shots of "The Patriot" together consume 1.6 terabytes (trillions of bytes) of disk space.
Individual artists need high-end desktop machines to work on and render their individual models and layers.
Rendering requires massive CPU resources. To render any animated 3-D figure or any effect like water or smoke, the CPU must generate millions of polygons, lines, points, etc and then light them correctly. And it must do this over and over again for each frame of the shot! For example, in "The Patriot" certain scenes incorporate hundreds of soldiers as well as things like boats, tents, flags, and so on. Each one moves independently, according to mathematical models expressed in the form of thousands or millions of pixels that are each calculated specifically.
Compositing - Compositing combines dozens of layers into a single shot. Because of the resolution involved -- millions of pixels and tens of millions of bytes per frame -- and the layering, both the CPU workload and the storage requirements are immense.

All of this means that the machine room at CFX is an amazing place!

Desktop machines are stored in the machine room to make maintenance and upgrades easier.

The Linux **render farm** formed from standard PC desktop units

Some of the many hard disks that form the online disk space available at CFX

This quick video walk-through of the machine room shows you the scope of the equipment required for the team at CFX to do its work.

The workhorses at CFX are a collection of SGI Onyx machines. These machines contain multiple processors and specialized graphics pipelines that speed the rendering and compositing tasks. Typically, a compositing artist will use an Onyx as his or her "desktop" machine, and other artists will run background rendering tasks on them as well. There is also a Linux render farm formed out of dozens of PC desktop machines. These machines work together to subdivide and execute rendering tasks.

Example 2

A big part of "The Patriot" is the battle scenes, and it turns out that most of the battle scenes contain computer generated soldiers. A good part of the R&D process for this film involved the creation of the individual soldiers and then groups of soldiers that would look realistic and convincing in the movie.

Creating CG soldiers that look realistic in a theater is not easy because the human eye is extremely sensitive to repetition and duplication. People are also very good at detecting artificial motion, even at a distance.

To create realistic groups of CG soldiers, the team at CFX started with real actors trained to perform in revolutionary war reenactments. These actors were outfitted and then filmed on a motion capture stage. The motion capture data allowed the team to create stick figures acting in a number of realistic ways -- walking, loading guns, falling and so on. For example, this video shows the motion capture data for four soldiers performing different activities.

The team fleshed out these stick figures with 3-D models of soldiers and then created four different variations (for both armies) with different heights and widths:

The different soldier models were then applied to the motion capture data to animate them, and then the soldiers were formed into groups.

Motion capture data, wire frame, solid model and final soldier

These two videos show the process:

One big part of the project was "dressing" the soldiers -- a process done by texture-mapping different fabrics and equipment onto the 3-D models. In "The Patriot," CG soldiers are only viewed from a certain distance, so advanced elements like cloth dynamics and facial expressions did not have to be incorporated into the models. If they had, the computational load -- already severe because of the number of soldiers -- would have increased dramatically.

The following shot shows how groups of soldiers get added to a scene. Here is the scene as it was shot:

This video shows the shot as filmed.

If you look carefully, you can see a number of bushes in the far field, as well as something as mundane as lawn mower tracks in the grass in the foreground! The visual effects for this scene involved everything from changing the sky to redoing the grass to removing the bushes and then adding in two armies. The following pictures show the process.

This video shows one group of soldiers being added to the shot.

This video shows the shot as it appears in the movie.

These small MPEG files have no way to show the richness and depth of the final scene as it would appear in a theater, nor do they demonstrate the realism of the two armies. When you see the scene in a theater, it is amazing!

Example 3

This example shows how the film's director can look at a scene and decide to completely change it using computer generated effects. In this case, the film crew shot a scene on a street in Charleston, SC.

This video shows the shot as filmed. The actors were dressed in period clothing, and the street was covered with a layer of dirt to hide the asphalt and take it back in time. Compare it to this video of the final shot as it appears in the movie -- the transformation is amazing!

The director made a decision to transform the end of this shot into something completely different -- instead of buildings down the left side of the street, what the director wanted is a dock, like this:

To create this effect, CFX did the following. First, using rotoscoping, the actors and the wall were lifted out of the frame of the original shot:

A new sky was added in to form the backdrop for the new scene:

Models of the docked boats were filmed against a green screen, rotoscoped and then added into the shot, and several buildings (both real and CG models) were added on the right:

Various crates and barrels were added to the dock to break it up, and there's even a new cloud in the sky if you look above the buildings:

CG flags were created and added to the ships:

All of this creates the final image:

These videos are interesting:

A scene transformation like this shows just how far a director can go in getting the right look and mood for a shot. About 90% of the scene is different from what was originally filmed. The magic of CFX is that the team can work together with the director to create completely new realities that look totally believable and convincing on the screen. The amount of effort it takes to work at this level is enormous, but as you can see in a film like "The Patriot," the final results are spectacular!

Partner Links

Creature Effects Makeup: An Interview with Matt Rose and Chad Waters

Links

The CFX Web site
Official Web site for THE PATRIOT
Motion Picture Technology - A nice overview
How Blue Screen Special Effects Work