FrameFree® Technical Summary

FrameFree® provides the capability to present completely continuous motion imagery, even when the image capture is done using a frame-based system. With conventional animation techniques, a motion image is represented by quickly displaying a number of still image frames, just like a flip book. In other words, a human's vision is tricked as it stores an image for a short time (roughly 1/24 of second) until another image is presented.

FrameFree® overcomes the need to present images as distinct frames by generating mathematical functions specifying a continuous path between captured frames. Our imagery has no frames at all. This is much closer natural motion, where there are also no frames.

Many image-matching (pixel matching) algorithms have been proposed, but none of them can match FrameFree® for smooth representation of motion images, much less do it automatically and in real-time. FrameFree® is quite unique in that, by employing topological approaches in image analysis, it simultaneously examines both global and local features of the images in order to determine the ideal match.

Technology Background

Conventional methods to determine correspondence (matching) between images do not simultaneously examine both global and local features without manual intervention. This increases the human labor required to complete a match and severely limits the applicability of these methods. Common conventional methods include:

Edge Detection
Edge detection is now the most commonly used method to create intermediate images. However, because only a very small portion of correspondences can be detected and because noise is introduced, laborious human editing is required.

Block Matching
In block matching, a portion of an image is first designated as a block. Detection of this block is then attempted in neighboring frames based on image resemblance. However, a block can only be detected if it remains close to its original position. Global correspondence is not considered. MPEG uses a form of motion-compensated block matching.

Optical Flow
A traced object is assumed to be a rigid solid and the smoothness of its motion vector or optical flow is investigated. Again, only local features are detected.

Wavelet
Unlike the above methods, wavelets provide some consideration of both global and local features in that the original image is made into a hierarchy. However, an averaging filter is used to construct the hierarchy and inevitably masks important features from the original image, resulting in rather inaccurate image matching.

MPEG
In current markets, MPEG is the most common form of video compression. MPEG replaces a majority of image frames with frames that contain a reduced amount of data and that represent only the differences between periodically inserted key frames called "I" pictures. The biggest problem with MPEG is at the very essence of image processing: quality. In MPEG, the desired compression ratio and image quality cannot usually be met simultaneously. This result is a phenomenon called block distortion that is easily seen in highly compressed applications.

FrameFree® Technology Overview

In display devices and image processing, the "pixel" is the minimal element. Theoretically, there can be no finer correspondence than pixel-based correspondence. As long as the present pixel-based display technology lasts, pixel matching will generate the finest matching. FrameFree® was the best pixel matching methodology when it was first proposed in 1996 and remains so today. Our researchers have been continuously refining and developing the technology from the initial theory through to the real-time automatic types found in FrameFree® products available today.

Algorithm

When given two key image frames to be matched, the Critical Point Filters first analyze each key frame to construct a set of sub-images for that frame, each sub-image having a lower resolution. The set of sub-images creates a resolution hierarchy of sub-images. Figure 1 shows a resolution hierarchy for one key frame image. In this hierarchy, the resolution of the image is reduced with each step. This example shows the reduced resolution by selecting a single pixel from each group of four adjacent pixels in the previous image. In the final step, the result is a single pixel.



FrameFree® further expands on the concept of the image hierarchy by creating separate hierarchies for each critical point within a specified area. In Figure 2, a lower resolution image is created for each of the four types of critical points (max, min, saddle) found in a four-pixel area in the key frame image.



Once the hierarchies are created for each key frame, matching of the sub-images starts from the lowest level of the hierarchy. Matching at each level is estimated and the result is carried over to the next higher matching level. At the lowest level, each sub-image has only one pixel, so correspondence between sub-images is determined automatically. The matching then begins at the next level. At this level, each sub-image has four pixels, so correspondence has several possible patterns. A best match is found considering pixel value and location closeness among possible pixel pairs, smoothness of mapping from each pixel in one frame to the next, and previous matching results gained at the current level and in coarser resolution levels. The matching process is complete when pixel correspondence between the first-level images (the key images) is achieved.

Local correspondence is considered by calculating a matching of all possible pixel pairs at each resolution level, while global correspondence is considered by starting the matching from the coarsest level. In coarser levels, pixel location is generally close for any pixel pair, as fewer pixels exist at that level. Pixel location closeness, which is a key factor for evaluating local correspondence, is thus given less weight in the early matching stages in order to first focus on global correspondence.

In the image hierarchy construction, the critical points are detected and their pixel values are preserved in the matching so that distinct features are maintained and not leveled out, as occurs when using averaging filters.

In a motion image sequence, FrameFree® technology provides smooth motion by interpolating between each captured image frame or can provide compression by interpolating between select key frames and eliminating intermediate frames. In the case of compression, FrameFree® produces a better matching result when more key frames are input, i.e., when oversampling is done at the camera. A greater number of captured images in fact helps to reduce the overall amount of data needed and produces smoother quality images. "Pannya", a movie rendered with FrameFree® technology, won the Gold Prize in the 13th Tokyo International Fantastic Film Festival (November 2000), illustrating the use of FrameFree® in a setting requiring professional image quality. Many more cool movies using FrameFree® are on this website and in the works.

FrameFree® technology can also be used to create 3D images by taking a number of pictures of an object from many different angles (for example every 10 degrees) and then using FrameFree® to match adjacent images. Once the matching is complete, FrameFree® allows the object to be rotated and viewed from any angle giving a smooth 3D effect.