Computer Interaction

The Computer Interaction feature enables robots to understand natural language instructions and interact with desktop and web applications using AI. Instead of relying on predefined scripts or hardcoded selectors, it interprets user intent, identifies relevant UI elements, and performs the required actions.

Computer Interaction uses AI reasoning together with a Retrieval-Augmented Generation (RAG) knowledge base to retrieve contextual information and inform decisions during task execution. It can adapt to changes in application interfaces, handle unexpected scenarios, and recover from execution issues, supporting more flexible automation across different applications.

Key Benefits

1. Natural Language Automation – Provide automation instructions in natural language without defining every interaction step.
2. Dynamic UI Interaction – Identifies and interacts with relevant UI elements across desktop and web applications based on task context.
3. RAG-Powered Decision Making – Retrieves contextual knowledge from the RAG knowledge base to support task understanding and execution.
4. Adaptive Automation – Adjusts to changes in application interfaces and determines alternative interaction approaches when needed.
5. Self-Healing Execution – Detects unexpected UI states and uses contextual knowledge to recover and continue automation with reduced manual intervention.
6. Reduced Dependency on Selectors – Lowers reliance on predefined scripts and hardcoded selectors, reducing maintenance effort when applications change.
7. Cross-Application Consistency – Supports automation across a range of desktop and web applications, including interfaces that change over time.

Use Cases

1. Web Application Navigation – Use natural language instructions to navigate web applications, locate required options, and perform actions such as clicking, typing, and selecting.
2. Desktop Application Automation – Automate tasks across desktop applications by identifying relevant UI elements and performing actions without relying on predefined selectors.
3. Visual UI Interaction – Identify buttons, fields, menus, icons, and other visual elements on application screens and interact with them based on task context.
4. Dynamic Form Filling – Interpret instructions to locate and populate the appropriate fields in web or desktop forms.
5. Application-Based Data Entry – Retrieve information from one application and enter it into another by identifying the required UI elements and interaction points.
6. Cross-Application Data Synchronization – Move data between different applications by mapping source fields to destination fields based on task context.
7. Self-Healing Automation – Detect changes in application interfaces or unexpected UI states and determine alternative ways to complete the requested task.
8. Multi-Step Business Process Automation – Execute processes that span multiple applications by interpreting instructions, applying contextual reasoning, and performing the required interactions in sequence.
9. Legacy Application Automation – Automate applications where conventional selectors or standard UI automation methods are difficult to apply, using visual identification of UI elements instead.
10. Recovery from Unexpected UI States – Handle pop-ups, dialogs, or layout changes encountered during a task without requiring manual intervention to resume execution.