Skip to content
TokenShift

How TokenShift works

TokenShift sits between a coding agent and the shell. When the agent runs a command — git diff, npm test, kubectl get pods — TokenShift intercepts the call, runs the real command, rewrites the output into a shorter form, and hands the trimmed text back to the agent. The full original is kept locally so the agent can fetch it back at any time.

The loop, in plain language

Section titled “The loop, in plain language”
agent runs:  git diff
     ↓
TokenShift recognizes the command
     ↓
runs the real command
     ↓
keeps the changes, trims the noise
     ↓
caches the full output on disk
     ↓
returns the short version to the agent

That’s all that happens. Five steps, every invocation. None of it requires configuration from the user — the rules ship in the binary.

What gets compressed

Section titled “What gets compressed”

Source control

git diff, git log, git show, git status. Keeps every change but trims unchanged context, omits binary blobs, and collapses long history.

Test output

pytest, jest, vitest, go test, cargo test. Keeps failing tests with stack traces and assertion details; drops the wall of pass lines.

Build & lint

npm, cargo, go build, eslint, tsc. Keeps errors and warnings; drops the noise around them.

Search

rg, grep, find. Groups results by file, trims duplicate matches.

Listings

ls, tree, git ls-files. Bounds depth, summarizes large directories.

Infrastructure

kubectl, docker, gh, aws, gcloud. Keeps the structure of resources; drops boilerplate fields the model already knows.

What never gets compressed

Section titled “What never gets compressed”
  • Anything the agent writes (prompts, file edits, code) — TokenShift only touches command output, never input.
  • Commands TokenShift doesn’t recognize — they pass through untouched.
  • Local files — TokenShift never reads the filesystem outside of running commands you’ve asked the agent to run.

Two ways it can be installed

Section titled “Two ways it can be installed”

Transparent, the default. A one-time install command writes hooks into the AI agent’s settings — currently Claude Code, Cursor, and GitHub Copilot CLI are supported. From then on, every command the agent runs flows through TokenShift automatically. The agent has no idea compression is happening; it just sees shorter output.

Explicit, for scripts and shells. You can also call TokenShift directly — tokenshift git diff, tokenshift npm test. Useful for shell aliases or quick checks. Rarely needed in normal use.

Nothing is permanently hidden

Section titled “Nothing is permanently hidden”

Every compressed response ends with one extra line:

… tokenshift recover <hash>

That hash points to the full, uncompressed output, stored on the local machine. If the agent ever needs the original, it asks for it by hash and gets it back verbatim. The compressed view is a summary, not a truncation.

What stays local, what goes off-device

Section titled “What stays local, what goes off-device”
  • Command output, full and compressed: local only. Never leaves the device.
  • Local files: never touched. TokenShift only sees what commands print.
  • Telemetry: opt-in, encrypted, and contains no command output — just what compressed and by how much. See Security overview for the full breakdown.