Robility Screen Play Task

The Robility Screen Play Task is an AI-powered automation tool that performs computer tasks based on simple, natural language instructions. Unlike traditional automation that follows rigid rules, this activity intelligently navigates and interacts with applications just like a human would, making it ideal for handling dynamic interfaces and changing layouts.

a. Perceive: The AI constantly scans the application screen to understand the current visual layout and elements.
b. Reason: It analyzes the information to decide the most efficient next step required to reach your goal.
c. Act: It performs the necessary interaction, such as clicking a button or typing text, to move the task forward.

Key Features

1. Agentic Loop: Define limits for steps, wait times, and timeouts for precise execution control.
2. Behavior Control: Manage automated planning, visual highlighting, and recovery from stagnant states.
3. Data Security: Protect sensitive information with customizable PII masking rules.
4. Adaptive Interaction: Seamlessly handles layout changes without the need for predefined navigation paths.
5. Outcome Tracking: Review detailed execution logs and receive a clear confirmation of task success.

Activity Details in Canvas

Prompt: Enter a natural language instruction that describes the action you want the AI to perform. You can enter a value directly or provide a String variable.

Success Condition: Enter the condition that determines whether the requested action was completed successfully. You can enter a value directly or provide a String variable.

Properties

Agentic Loop

Context / Constraints: Specify additional context, instructions, or constraints that the AI should consider while performing the task.

Max Steps: Specifies the maximum number of steps the AI can perform to complete the task. The default value is 20.

Note: Increasing the Max Steps value may increase AI-generated token usage and overall resource consumption. Set this value based on the complexity of the task.

Steps Wait (ms): Specifies the wait time between each AI step. The default value is 300 ms.

Success Condition: Enter the condition that determines whether the requested action was completed successfully.

Timeout (sec): Specifies the maximum time allowed for the AI to complete the task. The default value is 120 seconds.

Behavior

Enable Planning: Specifies whether the AI creates a plan before executing the task. It is a Boolean, Set to True or False.

Highlight Actions: Specifies whether actions performed by the AI are highlighted during execution. It is a Boolean, Set to True or False.

Max Verification Retries: Specifies the maximum number of times the AI can retry verifying an action. The default value is 2.

Stagnation Recovery: Specifies the recovery action performed when the AI is unable to make progress. The default value is BringWindowToFront.

Stagnation Threshold (Steps): Specifies the number of consecutive steps without progress before the stagnation recovery action is triggered. The default value is 3.

Masking

Enable Masking: Specifies whether masking is enabled for the activity. Set this property to True or False.
False: PII values are not masked.
True: PII values are masked based on the configured PII rule pack.

PII Rule Pack File Path: Specifies the file path of the PII rule pack used to identify and mask sensitive information.

a. When Enable Masking is set to True, the default PII rule pack is used if no customized rule pack path is provided.
b. To use a customized PII rule pack, specify its file path in this field.

Limitation and Workaround

When masking is enabled for a field, its original value is hidden and cannot be accessed by subsequent automation steps.

To ensure that values required by subsequent automation steps remain accessible, enable masking only for fields that require data protection.
a. Set Masking = True for fields whose values must be hidden.
b. Set Masking = False for fields whose original values must remain accessible to subsequent automation steps.

Customize the Rule Pack

If the default PII rules do not meet your requirements, you can create a customized rule pack for an activity.

To customize the rule pack:

1. Right-click Robility Designer and select Open file location.
2. Navigate to the following path:
SupportFiles\Packages\Features\(choose the project)\Robility.CoreActivities\2.5.7\PiiRulePack.default.json

3. Copy PiiRulePack.default.json to a location of your choice.
4. Modify the copied PiiRulePack.default.json file according to your requirements.
5. Specify the path of the customized rule pack in the PII Rule Pack File Path

For example:
“C:\Users\” + Environment.UserName + “\Documents\PiiRulePack.default.json”

Note: Always customize the copied PiiRulePack.default.json file. Do not modify the original file in the Robility Designer installation folder, as it may be overwritten during installation or updates.

MISC

Display Name: Specifies the display name of the activity. You can customize this value to improve workflow readability and simplify troubleshooting.

Skip On Error: Specifies whether the workflow should continue execution when the activity encounters an error.
True: Continues workflow execution when the activity encounters an error.
False: Stops workflow execution when the activity encounters an error.
Default: The default value is set to False.

Version: Displays the current version of the activity.

Output

Execution Log: Displays the execution details and status of the activity.

Result: Returns a Boolean value indicating the execution status of the activity.
True: The activity executed successfully.
False: The activity failed due to an unexpected error.

Task

Prompt: Displays the prompt specified in the activity canvas. You can use the value directly or provide a String variable or argument.

Prompt Design

For better AI execution, structure the prompt using the following format:

Action → Target → Details → Constraints → Expected Outcome

For example: Open the customer portal → search for customer ID 10245 → open the matching customer record → do not modify existing information → verify that the customer details page is displayed.

This structure provides the AI with the context required to identify the action, target, constraints, and expected result.

Viewing Activity Execution Reports

After the activity is completed, an HTML execution report is automatically generated.

Report Location

The generated HTML report is saved in a folder created at the activity’s configured file path.

To quickly access the execution folder, press Win + R, paste %LOCALAPPDATA%\Temp\RobilityScreenPlay (or C:\Users\<Username>\AppData\Local\Temp\RobilityScreenPlay), and press Enter.

Report Summary

The report provides the following execution details:

a. Result: Displays the overall execution status.
b. Steps: Displays the number of steps completed.
c. Actions: Displays the number of actions performed.
d. Duration: Displays the total execution time.
e. Model: Displays the AI model used for execution.
f. Tokens: Displays token usage, including input and output tokens.

Step Details

Each step can be expanded to view detailed execution information:

a. Action Performed: Displays the action performed during the step.
b. Observation: Displays the result or information captured after the action.
c. Raw JSON Data: Displays the underlying technical data for the step for debugging and troubleshooting.

Execution Configuration Log

Displays the execution configuration and activity details, including the original goal, AI model, execution start time, actions performed, and task completion verification.