Token Optimizer
Token Optimizer reduces token usage by removing unnecessary words, formatting, and repetition while preserving information relevant to AI processing. It can be used to optimize documentation, prompts, knowledge-base articles, and other large text inputs.
Key Features
1. Reduce text size: Makes large inputs more efficient to process.
2. Optimize only when needed: Runs optimization when the input exceeds the configured token threshold.
3. Control compression levels: Adjusts the amount of content retained using a configurable retention percentage.
4. Preserve essential information: Removes redundant or low-value content while retaining information required for downstream AI tasks.
5. Measure token savings: Displays the original and optimized token counts, total tokens saved, and percentage reduction.
Compression Levels
The Target Content to Keep setting defines the approximate percentage of the original information retained after optimization. Lower values provide greater compression, while higher values retain more content.
| Content to Keep* | Compression Level | Best For |
|---|---|---|
| 75–90% | Light | Documentation and prompts where context and detail are important. |
| 40–60% | Balanced | General documentation, prompts, and knowledge-base articles. |
| 10–30% | Aggressive | Scenarios where minimizing token usage is the highest priority. |
*Percentages indicate the approximate amount of original information retained after optimization, not the exact number of tokens retained.
Expected Outcomes
Using Token Optimizer can help:
1. Lower AI processing costs by reducing the number of tokens sent to AI models.
2. Improve response times by reducing input size and processing overhead.
3. Improve prompt efficiency by focusing AI models on relevant information.
4. Optimize large documentation sets for knowledge retrieval and AI-powered workflows.
5. Improve scalability when processing large volumes of text.
Tip: Start with Balanced compression for general use. Use Light when preserving more context is important and Aggressive when maximum token reduction is the priority.
