Global Token Utility Learning in Low-Latency Vision Transformers

Vision Transformers achieve strong performance in image recognition, but their self-attention layers produce high computational cost because all patch tokens are repeatedly processed across transformer blocks. Token pruning can reduce redundant computation, yet many existing methods estimate token importance from local attention scores or fixed pruning ratios, which may remove visually important regions before deeper semantic representations are formed. This study investigates low-latency Vision Transformer inference through global token utility learning. We propose the Global Utility Pruning Transformer (GUPT), which estimates token contribution from three sources: cross-layer attention persistence, class-token dependency, and gradient-sensitive feature variation. A lightweight utility predictor is inserted before selected transformer blocks to assign each token a retention probability. Instead of pruning tokens independently at each layer, GUPT applies a global budget controller that adjusts pruning decisions according to accumulated token loss, predicted accuracy risk, and hardware latency feedback. The model is trained with a joint objective containing classification loss, token-retention regularization, and latency-aware penalty. Experiments were conducted on ImageNet-1K, CIFAR-100, and Stanford Cars, including 1.31 million training images and 93,000 validation images in total. The method was evaluated on DeiT-Tiny, DeiT-Small, DeiT-Base, and ViT-B/16. On ImageNet-1K, GUPT reduced DeiT-Small computation from 4.6 GFLOPs to 2.7 GFLOPs while maintaining 79.4% top-1 accuracy, only 0.5 percentage points lower than the unpruned baseline. For DeiT-Base, the average number of retained tokens decreased from 197 to 104, and inference throughput on an NVIDIA A100 increased from 1,420 to 2,310 images per second. On Jetson Orin, end-to-end latency decreased by 37.8%, showing practical value for edge deployment. Ablation analysis showed that removing the global budget controller caused a 1.3-point accuracy drop under the same FLOPs constraint, while removing gradient-sensitive utility estimation increased fine-grained classification errors by 8.6%. These results indicate that global token utility learning can provide a more stable trade-off among accuracy, computational cost, and inference latency in efficient Vision Transformers.

1. Bao, Y., & Wang, H. (2026). Orion: A High-Throughput, Fault-Tolerant Event Routing Architecture for Hyperscale Data Streams. Fault-Tolerant Event Routing Architecture for Hyperscale Data Streams (January 01, 2026).
2. Ahmed, B., Naqvi, S. R., Akram, T., Peng, L., & Almarshad, F. (2025). A hybrid deep learning-ViT model and a meta-heuristic feature selection algorithm for efficient remote sensing image classification. International Journal of Computational Intelligence Systems, 18(1), 122.
3. Yang, Q., & Du, Y. (2026). Predicting Rollback Risks and Controlling Gradual Rollout Traffic for Online Ranking Feature Deployments.
4. Zhao, Z., & Welsch, R. E. (2024). Hierarchical reinforced trader (hrt): A bi-level approach for optimizing stock selection and execution. arXiv preprint arXiv:2410.14927.
5. Zhang, Z., Wang, J., Li, Z., Wang, Y., & Zheng, J. (2025). Anncoder: A mti-agent-based code generation and optimization model. Symmetry, 17(7), 1087.
6. Ancilotto, A., Castagnini, E., & Farella, E. (2025, September). Efficient Attention on Microcontrollers via Linearized Transformer Blocks. In International Conference on Image Analysis and Processing (pp. 494-506). Cham: Springer Nature Switzerland.
7. Yuan, Y., Xu, T., Yin, J., & Huang, J. (2026). Modeling Loading Anomalies and Identifying Root Causes in Media Consumption Workflows on Large Social Media Platforms. Available at SSRN 7393318.
8. Wu, J., Wu, D., Zhang, J., & Peng, Y. (2026). Generative AI Feedback in Junior High School Artistic Creation Evaluation. Available at SSRN 7244838.
9. Huang, J., Yin, J., Xu, T., & Yang, J. (2026). Quality Assessment and User Adoption Prediction for Enterprise-Level Financial Reporting Data Products. Available at SSRN 7179118.
10. Cauteruccio, F., Marchetti, M., Traini, D., Ursino, D., & Virgili, L. (2025). Adaptive patch selection to improve Vision Transformers through Reinforcement Learning: F. Cauteruccio et al. Applied Intelligence, 55(7), 607.
11. Yao, G., Zheng, J., Wang, Z., Zhang, W., Han, R., Zhao, C., ... & Liu, R. (2026, March). V-pruner: A fast and globally-informed token pruning framework for vision transformer. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 40, No. 40, pp. 34396-34404).
12. Liang, S., Hsu, K. H., & Liu, Z. (2026). Rolling Scheduling and Capacity Balancing for Server Assembly Under Engineering Change Disturbances.
13. Bao, Y., Qiu, Y., & Wang, H. (2026). FLARE: A Real-Time Framework for Detecting Malicious Account Activities at Internet Scale. Available at SSRN 7185659.
14. Latibari, B. S., Homayoun, H., & Sasan, A. (2025). Optimizing vision transformers: Unveiling ‘focus and forget’for enhanced computational efficiency. IEEE Access, 13, 27908-27927.
15. Zhao, J., Fan, J., & Li, L. (2026). A Study on an Explainable Causal-Enhanced LLM Agent for Predicting the Forming Quality of Automotive Component Materials.
16. You, S. (2026). Verifiable Audit Mechanisms in AI Compliance Automation: Scalability. Available at SSRN 6547458.
17. Jiao, Y., Shi, T., Zhao, B., & Wang, A. (2026). Retrieval-Guided Structured Reasoning and Interpretable Representation Learning for Large-Scale Video–Language Models.
18. Khan, S., Qasim, I., Khan, W., Aurangzeb, K., Khan, J. A., & Anwar, M. S. (2025). A novel transformer attention‐based approach for sarcasm detection. Expert Systems, 42(1), e13686.
19. Liang, S., Du, Y., Chen, W., & Liu, Z. (2026). Order Allocation and Emergency Transportation Decisions Amid Multi-Source Procurement Disruptions.
20. Yuan, Y., Yin, J., Huang, J., & Xu, T. (2026). Weakly Supervised Anomaly Detection and Privacy Risk Scoring in Community Message Threads.
21. Takezoe, R., Li, Y., Bo, Z. H., Hou, A., Guang, M., & Long, K. (2026, April). Learnpruner: Rethinking attention-based token pruning in vision language models. In International Conference on Learning Representations (Vol. 2026, pp. 66381-66400).
22. Zhang, Z. (2026). A Study on the Impact of Cost Presentation on Decision-making Bias and Cancellation Behavior in Online Subscriptions. Available at SSRN 7349839.
23. Qi, C., & Qiao, X. (2026). Building and Operating a Large Scale Multi-Agent System: A Case Study from Industry. Available at SSRN 6795198.
24. Tarasov, M., Ahmadi-Asl, S., de Almeida, A. L., & Cichocki, A. (2026). Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretability. arXiv preprint arXiv:2608.30505.
25. Xu, T., Zhu, W., & Zhang, J. (2026). A Study on the Application of Alternative Data in Credit Assessment for the Unbanked Population.
26. Du, Y., Liu, Q., Dong, Y., & Chen, Z. (2026). Cross-Domain Transfer and Few-Shot Recognition of Defect Images in Complex Industrial Settings.
27. Mersha, M. A., & Kalita, J. (2026). Explainable AI: Context-aware layer-wise integrated gradients for explaining transformer models. Neurocomputing, 133050.
28. Xiong, W., Guo, Y., & Zeng, Y. (2026). A Study on the Energy Efficiency of MEP Systems and Coordinated Dispatch Mechanisms for Multi-energy Systems in High-density Urban Buildings. Available at SSRN 7120518.
29. Shabarinath, B. B., Muralidhar, P., Sharan, A. V., Harshitha, B. V., & Vardhan, G. S. (2026, April). Hardware-Optimized Hyperspectral Image Classification on Edge Devices. In 2026 13th International Conference on Computing for Sustainable Global Development (INDIACom) (pp. 1-6). IEEE.
30. Hong, Z., Liang, S., Xu, T., & Chen, H. (2026). A Study on Failover Verification and Recovery Objective Prediction for Cross-Region Cloud Services.