Back
Technology

Tutorial: Four LLM cost optimization techniques reduce API costs up to 63%

Generated on: Last updated: 1 sources
View source

Here is the formatted article.

A technical guide outlines four techniques to reduce LLM API costs: prompt compression, semantic caching, chain-of-thought pruning, and output length constraints. Combined, these methods can achieve up to 63% cost savings.

Cost Saving Techniques

1. Prompt Compression
Remove unnecessary tokens from system prompts while preserving essential instructions. Manual strategies include eliminating hedge language, consolidating instructions into structured formats (e.g., JSON schema), and using reference tokens. Programmatic compression via LLMLingua can reduce input tokens by about 40%.

2. Semantic Caching
Store responses based on embedding similarity rather than exact match. Application-layer caching using embedding models and vector databases can skip redundant API calls. Provider-native caching (OpenAI, Anthropic, Google) also offers discounts for repeated prompt prefixes.

3. Chain-of-Thought Pruning
In production, limiting the model to output only the final answer instead of intermediate reasoning steps can reduce output tokens by 3x to 5x. CoT can be enabled via feature flag for debugging.

4. Output Length Constraints
Setting max_completion_tokens (OpenAI) or max_tokens (Anthropic) prevents runaway generation. Using structured output (JSON schemas or function calling) constrains response shape and reduces token count by 2x to 4x.

Cost Comparison

For a standardized entity extraction task run 1,000 times, the guide estimates cost savings across models:

Model Savings Baseline Cost Optimized Cost GPT-4o 62% $4.95 $1.90 GPT-4o mini 63% $0.30 $0.11 Claude 3.5 Sonnet 62% $6.99 $2.64 Claude 3.5 Haiku 62% $1.86 $0.70 Gemini 1.5 Flash 60% $0.15 $0.06

Recommended Implementation Order

  1. Compress prompts (easiest, highest input savings)
  2. Constrain outputs (targets most expensive tokens)
  3. Prune chain-of-thought (for production)
  4. Add semantic caching (most infrastructure, highest long-term savings)

Savings formula: (baseline_cost - optimized_cost) / baseline_cost.
Prompt compression saves 20-40% on input tokens; output constraints save 30-50% on output tokens; caching eliminates calls proportionally to hit rate. The guide advises instrumenting token logging first to establish a baseline.