I wondered, "What if we could reuse tokens based on their underlying priors: context, evidence, and reasoning?"
I have been asked how I am able to optimise my Claude token usage and ship fast?
I adopted the quick‑setup indexing guide (https://t.co/XIbdX2NYjv) as the baseline for Claude token optimisation, planning to later extend it with Procedural Graphs.
developer_workflow.quick_setup.resource = https://t.co/XIbdX2NYjv
I expanded my thinking on inference economics, emphasizing that the biggest cost driver is the number of output tokens generated. I propose building a cache of prior‑aware reusable tokens—objects that encode the underlying context, evidence, and reasoning of a generation—so that future queries can splice these tokens instead of regenerating them, potentially cutting inference expenses dramatically.
I explored how inference cost in large‑language‑model calls is driven by both input and output tokens and wondered whether we could cut that cost by reusing output tokens when the underlying priors match.
Price = f(tokens_in, tokens_out)
Providers cache input tokens to reduce inference costs.
But output tokens are still generated again, even when similar problems have been solved before.
What if we could reuse tokens based on their underlying priors: context, evidence, and reasoning?
- Open questions:
- How to formally define and extract priors from a request?
- Which similarity metrics reliably indicate shared priors?
- How to ensure cached output tokens remain fresh and correct?
- What security and privacy risks arise from storing generated tokens?
- What measurable cost reduction could be achieved?
What would prior-aware, reusable tokens look like?
Imagine shifting inference economics from repeatedly generating intelligence to accumulating and reusing it.
What if the biggest optimization isn't cheaper tokens, but fewer tokens generated?