URL
The URL component fetches and processes content from one or more web addresses, returning the data in various formats. It can recursively follow links up to a specified depth and supports output in plain text or raw HTML.
URL Parameters
| Parameter | Description |
|---|---|
| URLs | One or more web addresses to crawl recursively. Click the + icon to add multiple URLs. |
| Depth |
Controls how many links deep the crawler navigates (a value of 1.00 crawls only the main page). Adjust using the slider.
|
| Output Format | Sets the extracted content format to either plain Text or raw HTML. |
| Timeout | Sets the maximum time (in seconds) to wait for a request to respond. |
| Headers | Custom HTTP headers sent with the request. Click Open table to edit key-value pairs. |
| Extracted Pages / Raw Content | Selected via the dropdown near the output port to choose the output format (Table or Message). |
Controlling Component Output
The output of the URL component is configured at two distinct stages:
1. Extracted Content Format
Set via the Output Format input parameter:
Text (default): Extracts clean plain text from the HTML of crawled pages.
HTML: Extracts the complete raw HTML code of crawled pages.
2. Output Data Type
Set using the selection menu near the component’s output port:
Extracted Pages: Outputs a DataFrame with separate columns for content and metadata (such as url and title).
Raw Content: Outputs a Message containing all extracted text or HTML combined with metadata in a single text block.
Integration Guidelines
Standard Flow Execution
The URL component must be connected to a node that accepts its selected data structure (DataFrame or Message).
Use a Type Convert component if downstream nodes require a different data structure.
Because the URL component often yields large payloads, using a Parser component before sending data downstream allows you to extract and pass only specific fields.
Agent Tool Mode
Connect directly to an Agent component’s Tools port with no data conversion required.
The agent autonomously decides when to trigger the URL component based on user input and natively parses both DataFrame and Message formats.
