UI Vision

The UI Vision feature enables robots to visually analyze, identify, and interact with on-screen elements using AI and machine learning, similar to how a human would perceive and engage with a user interface. Instead of relying on traditional selectors or extensions, which can break when UI elements change, UI Vision recognizes elements based on visual cues like text, icons, buttons, and layouts.

Robility UI Vision utilizes the power of AI and machine learning to accurately detect, classify, and interact with UI elements. It employs AI-driven object detection to identify various components on the screen, OCR (Optical Character Recognition) to extract and interpret text, fuzzy text matching to handle variations in text, and layout detection to recognize structures within tables. Additionally, ML models are utilized for image matching and object classification, distinguishing elements such as icons, buttons, and input fields.

By integrating both AI and ML models, Robility UI Vision goes beyond traditional selector-based approaches, enabling comprehensive element recognition and text detection. This holistic approach allows the system to gain a full contextual understanding of the UI, making automation more resilient, adaptable, and efficient, even when UI layouts or designs change.

Key benefits of UI Vision

1. High-Precision Automation – Analyzes visual data to accurately identify patterns, objects, and text, ensuring precise task execution.
2. Versatile Interaction – Enables robots to detect and interact with UI elements across PDFs, images, documents, forms, and web applications.
3. Adaptive to Multiple Use Cases – Supports object recognition, image analysis, document processing across various environments.
4. Enhanced Stability in Virtual Environments – Overcomes challenges and seamlessly integrates with Citrix, VMware, and Microsoft Remote Desktop by eliminating reliance on unreliable image-based automation and selector targeting.
5. Selector-Free and Extension-Free – Eliminates dependency on rigid selectors and browser extensions, providing greater flexibility, adaptability, and reliability in automation.
6. Automates Dynamic Elements – Detects and interacts with changing UI elements like tables, checkboxes, dropdowns, and buttons without relying on fixed selectors.
7. Resilient to UI Changes – Adapts to evolving UI structures, reducing bot failures and maintenance efforts.