Rotoscoping isn't just for making animated films. It was used to add special effects to Alfred Hitchcock's 1963 film "The Birds," to place all of the glowing lightsaber blades into the original "Star Wars" trilogy films and to add effects to many other movies.
In the days of movies shot and edited on actual film, rotoscoping was sometimes used to paint special effects onto animation cells over live-action footage, but it was also used to create mattes (or masks) to allow filmmakers to combine elements from one filmed scene with elements of another completely different filmed scene. A filmmaker might want to superimpose footage of a person on a soundstage over footage of a location background, such as the ocean or outer space or in front of an explosion. This combining of images or effects that were not initially shot together into one film frame is called compositing.
This sort of rotoscoping was an incredibly time consuming task that, like its animation equivalent, involved projecting each frame of film onto a glass plate and manually tracing items one frame at a time. An animator or special effects artist would trace any elements that needed to be isolated onto a transparent cell. The traced elements would be filled in with paint to create mattes that could be placed on top of another frame (such as footage of the desired background) to effectively block out that area on the frame and leave a spot for the foreground effect.
The process would often involve creating multiple reels of film, for example the original foreground and background image shots, film of the blacked out mattes and film of a negative version of the mattes with a blacked out background. A new reel of film would be exposed multiple times to combine all of the elements together. The background image would be exposed to the film with the black matte in front of it, and the foreground element would be exposed to the same frame of film with the negative version of the matte in front of it to expose the foreground element to the film in the spot that was left for it during the first exposure. In this way, the rotoscoped mattes were used to combine the foreground element and the background onto each frame of film.
It took a lot of work to get it right. Typical 35 millimeter film is projected at a rate of 24 frames per second, which translates to 1,440 frames per minute, so mattes would have to be painted for dozens, hundreds or even thousands of frames. In a single action shot, the moving items change position a little in each subsequent frame (a matte that changes position and shape from frame to frame is called a traveling matte). It can get even more complicated when multiple things are interacting and overlapping on screen, requiring multiple mattes per frame.
Not only were a lot of frames involved, but the outlines and painted mattes for each one needed to be meticulous, the lighting had to match up (or be matched up later through color correction), and all the physical elements (the mattes, film and equipment) had to be lined up exactly or the results would move oddly or otherwise look out of place.
But when done well, rotoscoping could be used to show things on film that would have been difficult or impossible to shoot in real life. And it is also for more mundane but necessary purposes, like removing cables, microphones, staging markers or other items left in a shot, intentionally or not, which might throw the audience out of the moment while watching the film.
The technique is still used to this day to an extent, but thankfully, more modern methods for rotoscoping, creating mattes and compositing elements came into being in the computer age.