VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation
Learn how a hierarchical, interpretable reward model lifts both metric scores and human judgments in visual generation.
Aligning generated images and videos with human taste remains hard, and current reward models act as opaque scorers that can embed hidden biases. The authors built VisionReward to capture fine‑grained visual preferences while keeping the scoring process transparent. Their goal was to replace black‑box rewards with a system that can be inspected and tuned across multiple quality dimensions.
VisionReward organizes human feedback into a hierarchy of visual attributes, sharpness, composition, motion smoothness, and others, each scored separately. Linear weights combine these attribute scores into a single preference estimate, letting developers see exactly which factor drives a model’s rating. The framework trains on paired human judgments, learning the weight vector that best reproduces the observed rankings.
During generation, VisionReward serves as the reward function in a reinforcement‑learning loop, guiding diffusion or transformer models toward higher‑scoring outputs. A consistency strategy enforces that improvements on one attribute do not degrade others, preserving overall visual quality. The approach builds on prior RL‑from‑human‑feedback work but adds explicit multi‑dimensional control and interpretability.
In experiments VisionReward outperformed existing reward models on standard machine metrics and human studies. It lifted preference prediction accuracy over VideoScore by 17.2%, and text‑to‑video systems using VisionReward won 31.6% more pairwise comparisons than the same systems with VideoScore. These gains appeared across both image and video benchmarks.
If the reward model can be inspected, developers can diagnose why a generator prefers certain styles or frames, potentially reducing unexpected bias. The abstract leaves open how the linear weighting scales to dozens of attributes or how it handles conflicting human preferences in practice.
TakeawayHierarchical, weighted rewards give clear, tunable signals that markedly improve visual generation alignment.