A vision transformer model for multilingual image-based text recognition

Abstract

Multilingual image-based text recognition is a tough problem with several practical applications. This work suggests an integrated ViT-YOLO model which integrates the strengths of the Vision Transformer (ViT) and You Only Look Once (YOLO) techniques to solve this challenge. The goal of the model is to correctly identify text in pictures with text in many languages. The ViT-YOLO model uses YOLO to locate text sections in pictures using patch extraction. Taking use of its robust image-understanding capabilities, the ViT model processes the derived patches for text recognition. To enhance the model's performance and robustness, a Generative Adversarial Network (GAN) is integrated for data augmentation. Experimental results demonstrate the superiority of the ViT-YOLO model over traditional methods and other deep learning models, achieving an impressive accuracy of 93.49%. These findings demonstrate that the proposed ViT-YOLO model holds significant promise in addressing multilingual text recognition challenges and paves the way for future advancements in multilingual image-based text recognition.