A brand new paper out this week at Arxiv addresses a problem which anybody who has adopted the Hunyuan Video or Wan 2.1 AI video turbines can have come throughout by now: temporal aberrations, the place the generative course of tends to abruptly velocity up, conflate, omit, or in any other case mess up essential moments in a generated video:
Click on to play. Among the temporal glitches which can be turning into acquainted to customers of the brand new wave of generative video techniques, highlighted within the new paper. To the best, the ameliorating impact of the brand new FluxFlow method. Supply: https://haroldchen19.github.io/FluxFlow/
The video above options excerpts from instance take a look at movies on the (be warned: relatively chaotic) mission web site for the paper. We will see a number of more and more acquainted points being remediated by the authors’ technique (pictured on the best within the video), which is successfully a dataset preprocessing method relevant to any generative video structure.
Within the first instance, that includes ‘two kids taking part in with a ball’, generated by CogVideoX, we see (on the left within the compilation video above and within the particular instance beneath) that the native era quickly jumps via a number of important micro-movements, dashing the kids’s exercise as much as a ‘cartoon’ pitch. Against this, the identical dataset and technique yield higher outcomes with the brand new preprocessing method, dubbed FluxFlow (to the best of the picture in video beneath):
Click on to play.
Within the second instance (utilizing NOVA-0.6B) we see {that a} central movement involving a cat has indirectly been corrupted or considerably under-sampled on the coaching stage, to the purpose that the generative system turns into ‘paralyzed’ and is unable to make the topic transfer:
Click on to play.
This syndrome, the place the movement or topic will get ‘caught’, is among the most frequently-reported bugbears of HV and Wan, within the numerous picture and video synthesis teams.
A few of these issues are associated to video captioning points within the supply dataset, which we took a take a look at this week; however the authors of the brand new work focus their efforts on the temporal qualities of the coaching information as an alternative, and make a convincing argument that addressing the challenges from that perspective can yield helpful outcomes.
As talked about within the earlier article about video captioning, sure sports activities are notably tough to distil into key moments, that means that crucial occasions (equivalent to a slam-dunk) don’t get the eye they want at coaching time:
Click on to play.
Within the above instance, the generative system doesn’t know get to the following stage of motion, and transits illogically from one pose to the following, altering the perspective and geometry of the participant within the course of.
These are giant actions that acquired misplaced in coaching – however equally weak are far smaller however pivotal actions, such because the flapping of a butterfly’s wings:
Click on to play.
Not like the slam-dunk, the flapping of the wings is just not a ‘uncommon’ however relatively a persistent and monotonous occasion. Nevertheless, its consistency is misplaced within the sampling course of, because the motion is so fast that it is vitally tough to determine temporally.
These aren’t notably new points, however they’re receiving better consideration now that highly effective generative video fashions can be found to lovers for native set up and free era.
The communities at Reddit and Discord have initially handled these points as ‘user-related’. That is an comprehensible presumption, because the techniques in query are very new and minimally documented. Due to this fact numerous pundits have instructed numerous (and never at all times efficient) treatments for a few of the glitches documented right here, equivalent to altering the settings in numerous elements of numerous forms of ComfyUI workflows for Hunyuan Video (HV) and Wan 2.1.
In some circumstances, relatively than producing fast movement, each HV and Wan will produce gradual movement. Recommendations from Reddit and ChatGPT (which largely leverages Reddit) embrace altering the variety of frames within the requested era, or radically reducing the body fee*.
That is all determined stuff; the rising fact is that we do not but know the precise trigger or the precise treatment for these points; clearly, tormenting the era settings to work round them (notably when this degrades output high quality, as an illustration with a too-low fps fee) is barely a short-stop, and it is good to see that the analysis scene is addressing rising points this shortly.
So, moreover this week’s take a look at how captioning impacts coaching, let’s check out the brand new paper about temporal regularization, and what enhancements it would supply the present generative video scene.
The central thought is relatively easy and slight, and none the more serious for that; nonetheless the paper is considerably padded in an effort to attain the prescribed eight pages, and we are going to skip over this padding as obligatory.
The fish within the native era of the VideoCrafter framework is static, whereas the FluxFlow-altered model captures the requisite adjustments. Supply: https://arxiv.org/pdf/2503.15417
The new work is titled Temporal Regularization Makes Your Video Generator Stronger, and comes from eight researchers throughout Everlyn AI, Hong Kong College of Science and Expertise (HKUST), the College of Central Florida (UCF), and The College of Hong Kong (HKU).
(on the time of writing, there are some points with the paper’s accompanying mission web site)
FluxFlow
The central thought behind FluxFlow, the authors’ new pre-training schema, is to beat the widespread issues flickering and temporal inconsistency by shuffling blocks and teams of blocks within the temporal body orders because the supply information is uncovered to the coaching course of:
The central thought behind FluxFlow is to maneuver blocks and teams of blocks into sudden and non-temporal positions, as a type of information augmentation.
The paper explains:
‘[Artifacts] stem from a basic limitation: regardless of leveraging large-scale datasets, present fashions typically depend on simplified temporal patterns within the coaching information (e.g., mounted strolling instructions or repetitive body transitions) relatively than studying numerous and believable temporal dynamics.
‘This difficulty is additional exacerbated by the dearth of specific temporal augmentation throughout coaching, leaving fashions liable to overfitting to spurious temporal correlations (e.g., “body #5 should observe #4”) relatively than generalizing throughout numerous movement eventualities.’
Most video era fashions, the authors clarify, nonetheless borrow too closely from picture synthesis, specializing in spatial constancy whereas largely ignoring the temporal axis. Although methods equivalent to cropping, flipping, and colour jittering have helped enhance static picture high quality, they aren’t satisfactory options when utilized to movies, the place the phantasm of movement depends upon constant transitions throughout frames.
The ensuing issues embrace flickering textures, jarring cuts between frames, and repetitive or overly simplistic movement patterns.
Click on to play.
The paper argues that although some fashions – together with Steady Video Diffusion and LlamaGen – compensate with more and more advanced architectures or engineered constraints, these come at a price when it comes to compute and suppleness.
Since temporal information augmentation has already confirmed helpful in video understanding duties (in frameworks equivalent to FineCliper, SeFAR and SVFormer) it’s shocking, the authors assert, that this tactic isn’t utilized in a generative context.
Disruptive Habits
The researchers contend that easy, structured disruptions in temporal order throughout coaching assist fashions generalize higher to real looking, numerous movement:
‘By coaching on disordered sequences, the generator learns to recuperate believable trajectories, successfully regularizing temporal entropy. FLUXFLOW bridges the hole between discriminative and generative temporal augmentation, providing a plug-and-play enhancement answer for temporally believable video era whereas enhancing total [quality].
‘Not like current strategies that introduce architectural adjustments or depend on post-processing, FLUXFLOW operates straight on the information stage, introducing managed temporal perturbations throughout coaching.’
Click on to play.
Body-level perturbations, the authors state, introduce fine-grained disruptions inside a sequence. This type of disruption is just not dissimilar to masking augmentation, the place sections of knowledge are randomly blocked out, to stop the system overfitting on information factors, and inspiring higher generalization.
Assessments
Although the central thought right here does not run to a full-length paper, resulting from its simplicity, nonetheless there’s a take a look at part that we will check out.
The authors examined for 4 queries regarding improved temporal high quality whereas sustaining spatial constancy; means to study movement/optical circulate dynamics; sustaining temporal high quality in extraterm era; and sensitivity to key hyperparameters.
The researchers utilized FluxFlow to a few generative architectures: U-Web-based, within the type of VideoCrafter2; DiT-based, within the type of CogVideoX-2B; and AR-based, within the type of NOVA-0.6B.
For honest comparability, they fine-tuned the architectures’ base fashions with FluxFlow as an extra coaching section, for one epoch, on the OpenVidHD-0.4M dataset.
The fashions have been evaluated towards two standard benchmarks: UCF-101; and VBench.
For UCF, the Fréchet Video Distance (FVD) and Inception Rating (IS) metrics have been used. For VBench, the researchers targeting temporal high quality, frame-wise high quality, and total high quality.
Quantitative preliminary Analysis of FluxFlow-Body. “+ Authentic” signifies coaching with out FLUXFLOW, whereas “+ Num × 1” reveals completely different FluxFlow-Body configurations. Greatest outcomes are shaded; second-best are underlined for every mannequin.
Commenting on these outcomes, the authors state:
‘Each FLUXFLOW-FRAME and FLUXFLOW-BLOCK considerably enhance temporal high quality, as evidenced by the metrics in Tabs. 1, 2 (i.e., FVD, Topic, Flicker, Movement, and Dynamic) and qualitative leads to [image below].
‘As an example, the movement of the drifting automobile in VC2, the cat chasing its tail in NOVA, and the surfer using a wave in CVX turn out to be noticeably extra fluid with FLUXFLOW. Importantly, these temporal enhancements are achieved with out sacrificing spatial constancy, as evidenced by the sharp particulars of water splashes, smoke trails, and wave textures, together with spatial and total constancy metrics.’
Under we see picks from the qualitative outcomes the authors discuss with (please see the unique paper for full outcomes and higher decision):
Alternatives from the qualitative outcomes.
The paper means that whereas each frame-level and block-level perturbations improve temporal high quality, frame-level strategies are likely to carry out higher. That is attributed to their finer granularity, which allows extra exact temporal changes. Block-level perturbations, in contrast, could introduce noise resulting from tightly coupled spatial and temporal patterns inside blocks, decreasing their effectiveness.
Conclusion
This paper, together with the Bytedance-Tsinghua captioning collaboration launched this week, has made it clear to me that the obvious shortcomings within the new era of generative video fashions could not consequence from person error, institutional missteps, or funding limitations, however relatively from a analysis focus that has understandably prioritized extra pressing challenges, equivalent to temporal coherence and consistency, over these lesser considerations.
Till lately, the outcomes from freely-available and downloadable generative video techniques have been so compromised that no nice locus of effort emerged from the fanatic neighborhood to redress the problems (not least as a result of the problems have been basic and never trivially solvable).
Now that we’re a lot nearer to the long-predicted age of purely AI-generated photorealistic video output, it is clear that each the analysis and informal communities are taking a deeper and extra productive curiosity in resolving remaining points; with a bit of luck, these aren’t intractable obstacles.
* Wan’s native body fee is a paltry 16fps, and in response to my very own points, I observe that boards have instructed reducing the body fee as little as 12fps, after which utilizing FlowFrames or different AI-based re-flowing techniques to interpolate the gaps between such a sparse variety of frames.
First revealed Friday, March 21, 2025
