PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection

Visual instruction tuning of multimodal LLMs now battles massive, overlapping datasets that waste compute. The authors target this redundancy, aiming to prune instruction data without the heavy proxy inference or extra training that current selectors demand. Their key observation is that visual feature anisotropy creates a global semantic drift, corrupting similarity estimates used for selection.

PRISM tackles drift by implicitly recentering the visual feature space, restoring intrinsic semantics as a clean redundancy signal. The framework operates entirely on raw visual embeddings, avoiding any downstream model passes or gradient steps. By treating the recentered vectors as a proxy for true semantic distance, PRISM identifies and discards duplicate instructions in a single pass.

The method builds on prior work that measures dataset overlap with learned proxies, but replaces those costly stages with a statistical correction of the feature distribution. No additional parameters are trained; the algorithm merely computes a mean shift and applies it before clustering or nearest‑neighbor checks. This design keeps the pipeline lightweight and fully training‑free, sidestepping the bottlenecks that plagued earlier selectors.

Experiments show PRISM cuts end‑to‑end GPU consumption by about 70% while delivering higher scores than models trained on the entire dataset. Across multimodal and language understanding benchmarks, the pruned models achieve 101.7% of the baseline performance, effectively beating the full‑data fine‑tuning result.

TakeawayImplicitly recentering visual features lets you prune instruction data fast and improve model performance without extra training.