The three taxes: cost, latency, and trust

Why this matters

Adding AI to a product is never free — every AI feature levies three taxes that a plain feature doesn't. There's a money tax (you pay per token, every call, forever), a latency tax (model calls are slow, often seconds, breaking the snappy interactions users expect), and a trust tax (the feature can be confidently wrong, and one bad answer poisons the whole feature's credibility). You must price all three into the decision up front, because a feature that's 'cool' but adds a 4-second wait, costs a dollar per use, and is wrong 10% of the time is a net negative. Naming these taxes turns vague unease into a checklist.

Demo

Tally the three taxes for a candidate feature and compare against the value it delivers. The exercise is forcing yourself to put a number on latency and an error rate on trust, not just cost.

Try it yourself

1. Compute the monthly per-user cost of your top feature at your real call volume.
2. Put a number on p95 latency and decide if it breaks the interaction's flow.
3. Estimate the feature's error rate and what one wrong answer costs in user trust.
4. Find the break-even: how much value per use must the feature deliver to be net-positive?

References

• OpenAI — Pricing
• Nielsen Norman Group — Response times: the 3 important limits

Working within free-tier limits. Free / low-tier provider keys rate-limit aggressively, and eval or agent loops that fan out calls will hit 429 Too Many Requests fast. Survive it: read Retry-After and the x-ratelimit-* headers and back off (exponential backoff with jitter + a max-retry cap) instead of hammering; cap in-flight requests with a small concurrency limiter so you stay under the RPM/TPM ceiling; cache identical requests so retries don't re-spend quota; downshift to a smaller/cheaper model for practice runs; use the provider Batch API for non-interactive jobs; or sidestep hosted limits entirely by running a small model locally (Ollama / llama.cpp) or on a free Colab/Kaggle GPU while you learn.