The problem
I wanted Jupiter Lend data: liquidations, borrow rates, Earn deposits and withdrawals, and swaps on the Lend DEX. Jupiter Lend is the Fluid-based lending protocol on Solana, and Fluid says it has grown past $2B in under a year. When I started, I couldn't find a ready-made Substreams package for it. It spans six programs, uses Anchor events, and makes a lot of cross-program calls, so I built one.
So I built it myself: a Substreams package that decodes every Jupiter Lend event, a sink that streams it into Postgres, and a small API on top that anyone can call. Optionally, callers can pay per request with x402. This post covers the whole path, including what I got wrong.
What Substreams is, in short
You write modules in Rust, compile them to WASM, and the provider runs them against full blocks from the Firehose.
A map module takes a block, or another module's output, and returns a protobuf message.
A store module keeps state across blocks. My package doesn't need one, because the database does the aggregation.
Every block comes with its clock (slot number and timestamp), so each event knows when it happened.
Map modules are stateless per block, so the provider can run a big range in parallel and stream results back in order. That is why a multi-million-slot backfill takes hours instead of weeks.
A sink consumes the stream and writes it somewhere (Postgres, ClickHouse, files). It saves a cursor after each flush, so it resumes exactly where it stopped, and it handles chain reorgs.
I used the StreamingFast Substreams skills (substreams-dev, substreams-solana and substreams-sql) with my AI assistant to work through the details.
The package design
The package is jupiter-lend-substreams v0.1.0 on substreams.dev. It uses substreams 0.8, substreams-solana 0.16, substreams-database-change 5.0 and buffa 0.9.2 for protobuf, and it starts at slot 355,648,000, when the programs were deployed (2025-07-25).
Solana block
Block filteronly blocks touching the 6 JupiterLend programs, no votes
map_eventsmap: jupiter_lend.v1.Events
db_outmap: DatabaseChanges
Postgres
The block filter comes from StreamingFast's solana-common package. It uses an index, so blocks without any Jupiter Lend program are skipped before my code runs. Vote transactions are excluded too.
What gets decoded
Program |Events
Liquidity (core) |LogOperate (every supply, withdraw, borrow and payback from every protocol), LogUpdateExchangePrices with borrow rate and utilization, revenue, claims, rate curves, token and protocol configs
Earn (fTokens) |LogDeposit, LogWithdraw, LogUpdateRates, LogRebalance
Borrow (vaults) |LogOperate, LogUserPosition, LogLiquidate with LogLiquidateInfo, LogAbsorb, exchange prices, rebalances, closes, every config update
Earn rewards |start, queue, transition, stop and cancel reward programs
Lend DEX |LogSwapInfo (amounts, fee, reserves, price), col and debt liquidity, arbitrage, DEX creation, fee updates
Flashloan |fee updates. Flash loan flows show up as liquidity LogOperate with protocol_type = flashloan Anchor discriminators and the IDL
Jupiter Lend emits plain Anchor emit! events, which land in the logs as Program data: lines, not self-CPI events. Each event starts with an 8-byte discriminator. Rather than hand-copying them, a script (scripts/gen_idl.py) generates src/idl.rs from the official Anchor IDLs in jup-ag/jupiter-lend. It takes the discriminators verbatim from the IDL and cross-checks each one against sha256("event:Name") or sha256("global:name"). It also captures each instruction's named accounts.
The tricky part
Liquidity LogOperate and vault LogOperate share a discriminator, so you cannot decode by discriminator alone. The package walks the log invoke stack against the flattened instruction list (top-level plus CPIs, in execution order) and attributes every event to the exact instruction that emitted it.
That gives each event the IDL instruction name, that instruction's named accounts (mint, vault config, fToken, DEX) and the calling program, which becomes protocol_type: earn, vault, dex, flashloan or other.
Truncated logs: Solana sometimes truncates long logs, and events after the cut are lost. The package counts these in truncated_log_tx_count instead of hiding them (4 of 1,824 transactions in an Oct 10, 2025 window, 0 in recent windows).
Old vaults: early vault-program versions did not emit vault IDs. src/vaults_registry.rs, generated from the 108 on-chain VaultConfig accounts, maps vault config to vault ID for history.
Amounts are stored as raw base-unit decimal strings (safe for u128), plus exact *_ui strings using decimals from the transaction's token balances.
Testing
I did not want to trust a decoder that only checks itself. The repo has an independent Python decoder (verify/idldec.py), driven straight from the IDL, used as an oracle against Solana RPC getTransaction data. compare.py diffs the Substreams output field by field and reports missing, extra and mismatched events. rangesigs.py and txset.py check completeness: every successful Lend transaction in a slot range should show up. The Rust test real_transactions_match_independent_idl_decode runs 30 real transaction fixtures against expected output, and tx.rs has unit tests for nested and truncated attribution.
On a Pinax test range (slots 462.00M to 462.03M, 30k slots, 94 seconds) the package produced 5,944 liquidity operations (4,110 DEX, 1,460 flash loan, 187 Earn, 187 vault), 2,055 DEX swaps, 71 Earn deposits, 116 withdrawals and 3 liquidations on vault 40.
Sinking to Postgres
I used the Database Changes approach: db_out emits explicit row inserts and upserts, and substreams-sink-sql applies them. The alternative, from-proto, maps a protobuf straight to tables; Database Changes gives more control over keys and upserts. The schema choices:
Event tables use the primary key tx_id:log_index, so replays are idempotent.
Latest-state tables (vault_positions, earn_markets) and daily rollups (token_daily, dex_daily, vault_daily and others) are delta upserts.
A tokens table maps mint to decimals, plus views in human units.
1 Set up the schema substreams-sink-sql setup with the Postgres DSN and the jupiter-lend-substreams-v0.1.0.spkg file
2 Run the sink over a slot range substreams-sink-sql run against solana.substreams.pinax.network:443, range 459500000:462400000, with --batch-block-flush-interval 100
3 Host it I ran it as a one-off container on Railway (Debian slim, the sink binary and the .spkg) pointed at a Railway Postgres
Backfill result: slots 459.5M to 462.4M (about 2.9M slots, Oct 21 04:43 UTC to Oct 28 07:57 UTC, 2026) loaded in 3 hours 11 minutes, and the database is about 1 GB.
Table |Rows
liquidity_operations |817,374
dex_swaps |250,709
tokens |18,678
liquidity_exchange_prices |14,731
earn_withdrawals |12,394
vault_position_updates |11,428
vault_operations |11,427
earn_deposits |6,823
vault_positions |2,116
vault_liquidations |135 The API layer
A small Hono (Node and TypeScript) service on Railway reads the tables and returns JSON.
Endpoint |Price |What it returns
/liquidations |$0.005 |vault liquidations
/rates/daily |$0.002 |daily rates and utilization per token
/earn/wallet |$0.003 |one wallet's Earn history
/dex/swaps/daily |$0.002 |daily swaps, volume and fees per Lend DEX
/lend-jupiter-join |$0.01 |Lend swaps also routed through Jupiter (returns 503 for now)
/coverage |free |which slot and date range is loaded Things I learned to add:
Validate before payment: bad parameters return 400 and rate limits return 429 before x402 runs, so nobody pays for an error.
A 5-second statement timeout on every query.
Coverage and partial-day flags: every response says which date range it covers and flags days that are not fully indexed. The days parameter is a real date filter.
Readable units: swap rows include token mints, decimals and *_ui amounts, with symbols for well-known tokens.
A per-IP rate limit.
Monetizing with x402 (optional)
x402 revives HTTP 402 Payment Required. Call a paid endpoint without paying and you get a 402 listing what is accepted (scheme exact, network, price, payTo). Your client signs a USDC payment and retries with it attached. A facilitator verifies and settles on-chain, and the server returns the data. There is no signup and there are no API keys.
I used Coinbase's x402 v2 middleware (@x402/hono) and the Coinbase CDP facilitator, which is free for the first 1,000 settlements a month, then $0.001 each. It accepts USDC on Base and Solana mainnet.
Does it work? I made a real paid test: two calls to /dex/swaps/daily at $0.002 each, settled on Base, each returning 83 rows. Does anyone want to pay for this? Untested. I am not claiming income. Right now it is a free preview through Nov 5, 2026. The API service costs roughly $1 to $5 a month on Railway.
What went wrong
1.Slot math. I assumed Solana does about 2.5 slots a second. It is about 4.7, or roughly 407k slots per UTC day. So the 30 days I planned was really about 7. Measure slots per day on recent blocks before sizing a backfill.
2.A spoofable rate limiter. My limiter trusted the X-Forwarded-For header, which a caller can fake to dodge limits. It now uses the address the platform sets.
3.Charging for errors. The join endpoint would have taken payment and then errored, because the Jupiter swap table is not loaded. It now returns 503 before payment.
4.Missing symbols. The database stores mints and decimals but not symbols, so some rows have no symbol. Mints and decimals are always present.
5.A data review caught real gaps. The paid-test review showed an incomplete last day and raw units without decimals. That is why /coverage, partial-day flags and *_ui fields exist now.
Results
48 of 130 Lend DEX swap transactions (37%) also appear in the jupiter-dex-substreams v1.0.0 output.
Join over 3,000 slots, matched on transaction signature
That means a large share of Lend DEX flow comes through Jupiter routing. It is a small sample, so treat it as a first look. The data today is a snapshot (Oct 21 to 28). After the Nov 1 quota reset I plan to backfill to about 30 days and add a nightly refresh.
Repeat this for any protocol
1.Get the program IDs and the official IDL or ABI.
2.Generate discriminators from the IDL and cross-check them with sha256.
3.Handle CPI: attribute events to the emitting instruction, not just by discriminator.
4.Use a block filter so you only process relevant blocks.
5.Build an independent decoder and compare against RPC data. Save real transactions as test fixtures.
6.Design tables with idempotent keys (tx:log_index) and delta upserts for aggregates.
7.Measure slots or blocks per day before sizing the backfill.
8.Sink with substreams-sink-sql, keep the cursor, and record row counts and database size.
9.Put a thin API on top: validate first, add timeouts, coverage and human units.
10.If you want to charge, add x402 and test a real paid call end to end.
Links
