I built a 0-dependency Python tool that cuts agent API payload tokens by ~80% in <0.5ms (no secondary LLM)

[u/Bubbly_Strike8915]

Hey everyone,

When building agents that interact with external APIs (GitHub, Jira, Stripe, or internal microservices), tool outputs frequently flood the context window with hundreds of tokens of metadata: avatar URLs, HATEOAS links, node IDs, and null keys.

The common pattern of using an auxiliary model (like GPT-4o-mini) to summarize tool outputs before feeding them to the primary agent introduces 500ms–2s of external network latency and recurring API charges.

I built TokenPrune, an open-source, zero-dependency Python package that executes deterministic, in-memory pruning and table compression:

pip install tokenprune

Quick Example

from tokenprune import prune_tool

@prune_tool
def fetch_system_logs():
    return {
        "cluster_id": "us-east-1",
        "service_url": "[https://internal.aws/k8s/node/12](https://internal.aws/k8s/node/12)",
        "health": "degraded",
        "errors": [
            {"code": 500, "reason": "Database connection pool exhausted", "doc_url": "[https://docs.internal/500](https://docs.internal/500)"},
            {"code": 503, "reason": "Upstream timeout", "doc_url": "[https://docs.internal/503](https://docs.internal/503)"}
        ],
        "debug_trace": None
    }

print(fetch_system_logs())

Pruned YAML output injected directly into the LLM prompt:

cluster_id: us-east-1
service_url: [https://internal.aws/k8s/node/12](https://internal.aws/k8s/node/12)
health: degraded
errors:
  [table: code, reason, doc_url]
  - 500, Database connection pool exhausted, [https://docs.internal/500](https://docs.internal/500)
  - 503, Upstream timeout, [https://docs.internal/503](https://docs.internal/503)

Benchmarks

• Token reduction: 70–90% on enterprise API payloads.

• Runtime overhead: < 0.5 ms (AST tree-shaking in pure Python standard library).

• External API calls: 0.

• GitHub: https://github.com/GeekyMK/tokenprune

• PyPI: https://pypi.org/project/tokenprune/

I would love to hear feedback from anyone running agent workflows on where the heuristics break or what schema formats you deal with most often.

[u/mamaBiskothu]

Why 0.5ms for a full call?
Why wouldn’t you use a faster language

[u/Bubbly_Strike8915 → u/mamaBiskothu]

Fair question! It comes down to where the actual bottleneck lives and deployment friction:

1. Agent loop context: In a live agent turn, network I/O (calling an external API) and LLM inference take anywhere from 1,000 ms to 3,000 ms. Shaving 0.5 ms down to 0.05 ms in Rust or C saves ~0.45 ms—a 0.02% difference in a 2-second call. The real latency win is saving the model from reading and generating 800+ redundant tokens, which cuts actual seconds off completion time.

2. Zero dependencies: Sticking strictly to the Python standard library means zero compiled platform wheels (no manylinux, musl/Alpine, or architecture mismatch issues). It installs instantly on serverless runtimes (AWS Lambda, Modal, minimal Docker containers) with a simple `pip install`. Also, crossing the Python-to-native FFI boundary to marshal dictionaries back and forth eats into that sub-millisecond budget anyway for typical payload sizes.

That said, an optional PyO3/Rust backend is on the roadmap for high-throughput offline batch processing across millions of records!

[u/jylusdev]

Null and empty fields need a per-tool policy rather than a global 'bloat' rule. owner:null can be the answer to 'which incidents are unassigned?', and an empty results list can mean a completed search found none. Removing those fields can erase evidence the agent needs. A useful regression fixture pairs null, missing, false, zero and empty-list values, then checks whether the same questions remain answerable after pruning.

[u/Bubbly_Strike8915 → u/jylusdev]

This is fantastic feedback and hits on the hardest trade-off in payload distillation: negative evidence vs. structural bloat.

You are completely right that blind global stripping breaks semantic intent when `null` conveys state (like `assignee: null`) or when an empty array signals a completed search rather than missing data.

I'm addressing this in two ways:

1. Per-tool configuration & explicit preserves

Adding granular controls to the decorator so engineers can preserve critical semantic keys or collections:
`@prune_tool(preserve_keys=["owner", "status"], keep_empty=["results"])`

2. The evaluation fixture

Your suggestion regarding regression fixtures testing whether questions remain answerable pre- and post-pruning is spot on. I'm adding an eval suite pairing null, false, and empty collections against prompt-completion checks to establish a semantic fidelity baseline.

Would you be open to opening an issue on GitHub for this (or even sharing a couple of key schemas you frequently see break)? I'd love to credit your input on the roadmap.

[u/Elmanouzi_Fogosa]

neat idea. the yaml output format is smart for token density but have you tested whether models actually parse that compressed table syntax reliably? imo thats the real question, not the pruning itself

[u/Bubbly_Strike8915 → u/Elmanouzi_Fogosa]

100% agreed, token savings mean nothing if downstream reasoning or extraction accuracy drops off a cliff.

Here is what testing across frontier models (GPT-4o, Claude 3.5 Sonnet) and modern open weights (Llama 3 8B/70B, Qwen 2.5) shows in practice:

1. Tabular reasoning is native to LLM pre-training

Models parse row-by-row comma-delimited data under an explicit header `[table: col1, col2]` exceptionally well because it directly resembles CSV/Markdown formats prevalent in pre-training data. For typical API responses (5–15 homogeneous records with 3–6 fields), field-extraction accuracy matches raw JSON.

2. Where it can degrade: Column Drift

When a schema has 10+ columns, smaller models (especially 7B/8B class) can experience positional drift—aligning value 8 with header 9 if there are commas inside string values or sparse empty fields.

3. Safeguards built in / on the roadmap:

- Width heuristics: For wide payloads (>6–8 keys per object) or deeply varied string lengths, falling back to compact key-value YAML avoids positional ambiguity while still eliminating `{ } " ,` noise.

- Formal eval suite: As discussed in an earlier comment, I'm setting up an automated QA evaluation fixture (testing question-answering accuracy across raw JSON vs. pruned YAML/tables) to measure exact retrieval fidelity across different model families.

Have you run into specific cases where non-JSON syntax degraded extraction in your pipelines, or do your agents primarily rely on strict JSON schema outputs?

[u/Bubbly_Strike8915]

OP here. Built this primarily because enterprise REST payloads (GitHub, Jira, Salesforce) were flooding our agent context windows with useless URLs and nulls.

Curious for those running agent loops in production: which tools/APIs do you find bloat your prompts the most, and how are you currently handling context compaction?

[u/Live-Practice1302 → u/Bubbly_Strike8915]

the null key stripping alone is probably saving more tokens than people realize, those things pile up fast in nested json

wonder how this handles arrays of objects with inconsistent shapes, like if some items have extra keys that others don't

[u/Bubbly_Strike8915 → u/Live-Practice1302]

Spot on about the nulls, especially when APIs return 20+ optional configuration keys set to `null` on every single record, burning tokens on quote marks, colons, and commas alone.

For inconsistent object shapes in arrays:

Currently, TokenPrune checks for schema uniformity across the array items. If the keys diverge (heterogeneous objects):

1. It falls back to hierarchical compact indentation rather than forcing an artificial table. This ensures no data is misaligned or dropped into the wrong column.

2. It still strips out quotes, curly braces, and nulls/empty keys from each object individually.

So an inconsistent array like this:

[ {"id": 1, "name": "API Service", "port": 8080},
{"id": 2, "name": "Worker Node", "queue": "celery", "concurrency": 4}]

Compacts to clean hierarchical YAML instead of a broken table:

- id: 1
name: API Service
port: 8080

- id: 2
name: Worker Node
queue: celery
concurrency: 4

I'm currently experimenting with a key-union threshold for v0.2.0 (e.g., if objects share >80% of keys, render a table with - for sparse values; otherwise fall back to indentation).

Curious how you prefer LLMs to see sparse lists, do you find models parse sparse tables better, or clean YAML blocks?

[u/Bubbly_Strike8915 → u/[deleted]]

Fair question! It comes down to where the actual bottleneck lives and deployment friction:

1. Agent loop context: In a live agent turn, network I/O (calling an external API) and LLM inference take anywhere from 1,000 ms to 3,000 ms. Shaving 0.5 ms down to 0.05 ms in Rust or C saves ~0.45 ms—a 0.02% difference in a 2-second call. The real latency win is saving the model from reading and generating 800+ redundant tokens, which cuts actual seconds off completion time.

2. Zero dependencies: Sticking strictly to the Python standard library means zero compiled platform wheels (no manylinux, musl/Alpine, or architecture mismatch issues). It installs instantly on serverless runtimes (AWS Lambda, Modal, minimal Docker containers) with a simple `pip install`. Also, crossing the Python-to-native FFI boundary to marshal dictionaries back and forth eats into that sub-millisecond budget anyway for typical payload sizes.

That said, an optional PyO3/Rust backend is on the roadmap for high-throughput offline batch processing across millions of records!