Example of the temporal difference matte effect
Temporal Image Processing operations are time based video processing effects that use the current source movie as a source for processing. This kind of effect assumes you have a loaded source movie or are processing a movie file. You can run them without a loaded source movie but nothing interesting will happen if you do so. This is because the individual effects work with multiple movie frames from different times to create effected output that is dependent on the individual input frame images changing in some way over time.
Because temporal image operation effects work with multiple frames to generate a single effected output frame, they can be used to accent motion in a video sequence. The simple example shown above was generated with the temporal difference matte effect and is accenting the movement of the ice skater in the video sequence it was derived from. You can use Temporal Ip Op effects to amplify or suppress motion, to stretch or contract time in a video, to generate slit scan effects, or to convert video pans into static panorama imagery.
Another application for Temporal Ip Ops is called Stack Filtering. What this means is processing a stack of images with the time based effects. You can do this by creating a source movie file that consists of the individual images you want to Stack Filter. The Temporal Ip Ops will think that the stack of different individual images are sequential movie frames and try to process them accordingly. This unconventional approach to time-based image processing is a way to create some amazing unconventional visual effects.
Example of a temporal rank neighborhood filter used as a stack filter
The Temporal Ip Op Editor
The individual Temporal Ip Op effects available at the time this tip was written are listed above in the above screen shot of the Editor palette. The popup menu control at the top of the Editor lets you select the particular temporal effect you want to run. All of these effects are time-based which means they take multiple movie frames from different times as their source and perform some kind of processing where time is a component of the particular processing algorithm.
All of the individual effects have a collection of editable parameters you can adjust in the Editor Palette when in Temporal Ip Op operation mode. The editable parameters for the temporal Rank Filter are displayed below.
The bottom 3 parameters, Mix, Composite, and Composite Space are generic compositing parameters available for all of the different temporal ip ops. These have the same functionality as the equivalent generic parameters in other Studio Artist operations like Image Operations, Texture Synthesizer, MSG, etc. They control the effect output mix and compositing with the existing canvas layer image.
The Size and Frame Increment parameters are seen in many of the different temporal effects. Size refers to the size of the temporal frame window being used for the effect (how many frames are processed). The Frame Inc (increment) parameter refers to the number of frames advanced when moving from one input frame to another. So a Frame Inc of 2 would take every other frame, 3 would mean use every 3rd frame, etc. By increasing the Size and Frame Inc you would be processing more frames spaced further and further apart. The frame window specified by the Size parameter is usually centered around the current source frame unless there is a control that specifies something else.
The output of any given temporal effect can dramatically change depending on the size and frame spacing of the temporal processing window as well as what is happening visually in the frame images within the time window. Getting a feel for the different effects and what you can do with them requires some initial experimentation.
Be aware that some temporal ip ops will render slowly. This is especially true of the Slit Scan Complete and Time Axis Rotate Complete effects since they need to process all of the frames in the source movie. Increasing the size of the frame window being processed will also slow down a particular effect since more frames will have to be processed to complete the effect. There is a status message at the bottom right corner of the Status Area in the main Studio Artist window that will let you know if a temporal ip op is still processing frames.
You can always press the esc key if you think a temporal image operation is running too slowly and you want to stop it from processing.
The Temporal Effects
Here are some examples of what can be generated with some of the different temporal effects. Be aware that many of the filter effects may look radically different depending on which particular filter algorithm parameter you choose. And as mentioned above, what is happening in the source video (motion, pans, jitter) you use as input to a particular effect will greatly influence whether the temporally processed effect output is interesting or not.
Some effects or effect settings are designed to accent motion in the source video frames. Other settings may suppress motion from the effect output. For effects like the Scan tracker or the Slit Scan, the interaction between the direction of the motion in the video sequence and the direction of the scan control in the effect can lead to interesting or chaotic processed output. Camera shake or pan movement will be processed as motion in a temporal effect and may lead to desirable or undesirable output depending on the particular effect you are running and what you hope to visually achieve with the effect.
All of the examples below have an example output based on the same source video sequence of a runner. Because this standard source video may not really be optimal for showing off the particular effect some other examples are also presented to show off what can be done with the proper video footage for these effects.
Again i think this helps show off the fact that you need to develop a feel for how the motion of other characteristics in your source video interact with a particular effect and it’s internal parameters to build different style temporal effects. As always, there’s nothing like some experimentation with different settings to help develop tis understanding, and you may sometimes stumble upon interesting combinations of video motion and effect settings that lead to new and interesting results.
Temporal Difference
Computes a simple frame difference effect. You could think of this as being like a motion edge effect, where the brighter a pixel the more motion is associated with it. Not the most visually interesting effect, but a Src Movie Frame Difference modulator based on this effect is available in many Image Operation effects and this can be very useful for spatially modulating an image processing effect based on motion.
Temporal Difference Matte
Good at rendering the motion happening within the processed frame window in a single frame. Tends to work best if the camera is stationary without jitter and objects are moving in the camera view.
Example of a skating video processed with the difference matte
Temporal Displace
Time-based displacement mapping. The canvas image at the start of the processing is used as the frame displacement modulator.
Example of a sphere image used as a temporal warp modulator into a stack of graffiti images
Temporal Flow
You could think of this as a temporal orientation where the resulting effect output encodes the angle of movement in the frame. Also works well as a video motion edge effect.
Frame Jumble
Reorders the frames in the video sequence. Can be used to reverse the entire sequence or generate local stutter reversal or randomization.
Temporal Motion Rank Filter
Rank filters the motion in the video sequence.
Temporal Pixel Jumble
A random pixel temporal displacement effect.
Temporal Rank Filter
The temporal equivalents of some of the Rank Filter Image Operations. Rank filters are nonlinear filters. Depending on the particular algorithm you use the effect can change dramatically. Can be useful for representing or slowing down motion in a video sequence as well as creating bizarre special effects.
Example of a car jump video processed with a temporal min filter.
Temporal Rank Neighborhood Filter
Looks at a spatio-temporal neighborhood for each pixel and extracts a rank ordering based on spatial energy. So it’s a way to output the maximum or minimum energy details in the frame window being processed. Can be used to accent or diminish motion in a video sequence, or to create fascinating stack filter effects that provide super edge effects or severe averaging across the stack of images being processed.
Extreme rank neighborhood stack filter effect
Temporal Simplify
Similar to the Simplify Image Operation effect, but works it’s magic over multiple frames. This can lead to simplified cartoon style processed video output that has less flicker from frame to frame.
Temporal Scan Tracker
One of my favorite temporal effects. Depending on the parameter settings and the kind of video sequence you are processing this can be configured to produce many different kinds of visual effects. You can encapsulate the motion in a video sequence into a single static image. Or you can get an overview of what’s going on in a video sequence in a single static image. Or you can take a video pan and turn it into a panorama image. Or you can make a video sequence stutter.
Panorama generated from a video sequence.
Temporal Slit Scan Complete
Simulates scanning a slit across the complete sequence of video frames. This can be used to get an overview of what is happening in the complete video sequence as shown above or to freeze motion in the direction of the scan.
Vertical slit scan of video footage of someone eating. Note the interaction of the scan direction and the direction of the motion, which was chosen for optimal effects results.
Temporal Time Axis Rotate Complete
If you think of a video sequence as a 3D cube, where the z-axis represents time, then this effect rotates the spatial-temporal video cube so that time has been shifted to be in the plane of the 2D view. Can create some really ethereal images or video special effects.
For More Information
A tutorial tip on how to process an entire movie file with a temporal image operation effect is available here.
A Studio Artist News post that provides an overview of what you can do with temporal image operations is available here.