GRIP-VLM: RL Speeds Up Vision-Language Models by 15%

Vision-Language Models are facing a bottleneck due to the massive processing of visual tokens. Here's the thing: a new framework called GRIP-VLM is tackling this challenge. GRIP-VLM uses a novel approach, formulating visual token pruning as a Markov Decision Process. This framework leverages a Group Relative Policy Optimization, or GRPO, paradigm. This reinforcement learning strategy helps navigate the discrete search space more effectively. What's interesting is GRIP-VLM includes a lightweight agent with a budget-aware scorer. This allows it to dynamically assess token importance and adapt to any compression ratio without full retraining. Evaluations show GRIP-VLM consistently outperforms other methods. It delivers up to a 15% inference speedup while maintaining accuracy. This could lead to more efficient and scalable Vision-Language Models.