Frame rate conversion hitches

One or more submitted videos shows hitches due to frame rate conversion. This can be over the entire video, or certain parts of the video, or even certain elements in the image (animations on top of footage for example). This frame rate conversion takes place when different frame rate settings are applied in the video production process. But also when speed changes are applied on the footage in the editing software. The problem is rather complicated, and that is why detailed explanation is given below to make it clear.

WHAT IS FRAME RATE

Video is a moving visual media. Just like with film, a series of images is used one after the other to create the illusion of a moving picture. The frequency of images per second is called the frame rate. The images themselves are called frames. This frame rate is expressed in "frames per second" (fps). The more frames in a second, the smoother the motion appears to the eye. But there are several established Video standards in terms of frame rates. These different frame rates cannot simply be used interchangeably. For example, if you have recorded 30fps, but are playing 25fps, the video will slow down. After all, fewer frames are displayed in 1 second than recorded. To solve this problem a frame rate conversion must take place. A calculation to ensure that with a higher or lower frame rate than the original, the final display speed of the video is the same. There are various conversion methods for this, subdivided into these three: 1) Frame Sampling or Frame Step, 2) Frame Blending or Frame Mix or 3) a Motion Vector or Fluid Motion or Optical Flow or ... method. More explanation about these methods can be found at Framerate conversion distortions.

‘Frame Sampling’ or ‘Frame Step’ is the frame rate conversion method that causes the hitches in the video. With this method the amount of frames of the source and the output setting are compared. Frames are then repeated to get a higher frame rate, or frames are skipped to get a lower frame rate. See this example that clarifies how these conversions adversely affect the display of the final video.

FRAMERATE CONVERSIONS IN VIDEO PRODUCTION

With frame rate conversion it never gets better, no matter which method you choose. This becomes even worse when multiple conversions take place, where each conversion further affects the previous version. This way you get further and further away from the original. Increasingly more random hitches occur with the Frame Sampling method, for example. These hitches cannot be repaired afterwards. Therefore try to avoid framerate conversions as much as possible. This can be achieved by keeping all settings in the entire production process, from recording to final display on screen, in the same frame rate. Unfortunately, because there are different playback devices and screens, it is sometimes not possible to completely avoid conversions. But more than 1 conversion is not necessary. If in the production process 1) the recording or import, 2) the editing or effects and 3) the export are in the same frame rate, then only for displaying the video there could take place a conversion if necessary. What do we mean first of all by the 3 steps in video production?

1. Recording and import: Regarding recording, these are the settings of all cameras that are used. Make sure that they all record at the same frame rate, and preferably at the frame rate of the most important screen where you ultimately want to display the video properly. In the case of European television, that should be 50i or 50p. Regarding import, this concerns external stock footage that is used. Check whether this footage also has the same frame rate as the video edit project or sequence. And also check whether that material has not already been affected by a previous frame rate conversion. Unfortunately, this even occurs on the major stock footage suppliers. In the case of (3D) animations, it is important to set the rendering of it at the correct frame rate in the animation software used. Consult the manual of the relevant cameras or software for an explanation of how to set the frame rate, or indicate your preference with the relevant camera operator or motion graphics designer. 
2. Editing and effects: These are the settings of your project in your editing software. There is a lot of editing software available, and the setting options vary per program. View the manual or help function of your editing software for more information. Roughly, it's about having the overall project settings and program settings right. In addition, you can set the timelines or sequences separately in some software. Also check whether subsequences or nested sequences have the same frame rate settings. And then there are settings for external animation or color correction suites as external effects or plugins that are used in your editing project. Check the frame rate settings at all possible locations. Because a smooth animation that is rendered at another frame rate is then no longer displayed as smoothly in the final video.
3. Export: This is the process of final rendering and saving the video file. Here too you can set a frame rate again, and if this frame rate is different from your project or your timeline, then again conversion will take place. Therefore, also check when saving whether the correct preset is being used, with the correct frame rate that is equal to the recording and editing.

To eventually ensure that a maximum of only 1 frame rate conversion takes place in the entire video production, there are 2 ways to achieve that:

1. the recording is leading. In the entire production process, everything is set, and only external material is used in the frame rate of the source. Frame rate conversion then only takes place at the end of the production pipeline during the playback of the video.
2. either the display is leading. You can choose this if source material with different frame rates is used. Editing, effects and export are then identical to the display, there is no conversion nesesary during playback. The conversion takes place when the footage is imported and processed by the editing software at the beginning of the production pipeline.

Below, the two methods described above are shown schematically with a few example frame rates. Also if you do not adjust any frame rate setting, and if you adjust all settings to 1 frame rate:

SPEED CHANGES

Frame rate conversion does not only take place when converting from one frame rate to another. This also applies if you set a speed change of the video in the editing software. Even then, images must be deleted or repeated. To ensure that this happens as neatly as possible, we recommend setting this speed changes in doubles or halves. This means that a number of frames are repeated or skipped at a very constant rate, which is the least disturbing when viewing the video. So 2x, 4x or 8x as fast or slow. For example with these percentages, where 100% is the original speed: 25%, 50%, 100%, 200%, 400% 800%. In addition, in some editing software you can opt for different frame rate conversion methods. Here too you can look for which method gives the best results. See the manual or help function of your editing software for more information about available frame rate conversion methods.

DETECT FRAME RATE HITCHES

How can you see whether or not a video file or the editing runs smoothly? This is only visible on a monitor that runs at the same refresh rate as the frame rate of the video file or the edit. The refresh rate is the amount per second that a monitor reconstructs the image, so refreshing the display. The refresh rate is expressed in Hz (Hz); the number of changes per second. A computer monitor usually has a refresh rate of 60Hz. That means that everything you play on a computer screen, which is not 60fps, does not run smoothly because a conversion is performed by the computer's video card. The frame rate conversion that is used as standard is the Frame Sample method, as described above. For this reason, a computer screen is not a good reference for determining whether or not the video or edit is smooth when working with European frame rates (25p, 50p or 50i). We recommend working with an extra video output card in the computer or an external USB or Thunderbold device that works well with your editing software. The editing software then sends the rendered result of the editing, directly to the video output device. Consult the manual or help function of your mounting software for more information.
A video (reference) monitor or television should be connected to this video output device that actually runs at the refresh rate as the frame rate that the editing is set to. Smart TVs also usually run at 60Hz, and therefore have the same problem as a computer monitor.
It is also possible to see traces of frame rate conversion, to go through the video files frame by frame. If you regularly come across a repeating frame, or an extra large offset in the moving parts of the image, then frame rate conversion has already taken place, and the material is less suitable for use in the video.

FRAME RATE CONVERSION EXAMPLE

Finally, below an example of what the consequences may be of frame rate conversion with the "Frame Sampling or Frame Step" method. There are frames of 1 second from the video displayed where the frames of the original footage is numbered. Then we see that part converted into different frame rates, so more or fewer frames in that one second. Repeating frames are marked in red, and a purple line means that a frame has been skipped before.

Situation 1: There has been recorded at 25p.

>> Conversion 1: That 25p footage was imported into a 30p project or timeline. The editing software now has to repeat 5 frames per second (marked in red).

>> Conversion 2: This 30p timeline with already repeating frames is exported to a 24p file. Of the 30 frames in 1 second, 6 frames are now removed, where a pink line mark can be seen. In combination with the repeated frames from the previous conversion, this is the result.

>> Conversion 3: Finally, this already hitchy 24p material is converted to 50i for broadcasting, the frame rate of the TV channel. 26 frames are repeated again.

This is the result on TV. As you can see from the numbering, the sequence of these new frames is completely disordered. It skips and repeats a lot. As a result, all movements no longer look smooth.

 

Situation 2: There has been recorded at 25p.

>> No conversion: The timeline or project settings are also set to 25p. No conversion needed. 

>> No conversion: The export is also set to 25p. No conversion needed, all frames of the footage are still in tact.

>> Conversion 1: Eventually, it is converted to 50i for the display on the TV channel, the only conversion that takes place. Each frame is repeated 1 time.

This is the result of situation 2. As can be clearly seen, this looks much better. There is frame repetition, but that is at a very constant rate, so this result still looks very smooth.