Execution

Alchemy free vs paid against our real call counts, where the seconds go today, what the rest of the stack is worth, and the elite target.

Lane report · REPORT-execution.md · updated 06/09/2026 07:42:15

Worker analysis-execution, 07:26 to 09:30 Beirut, against BRIEF-execution.md. Everything below was measured this morning against the live files and endpoints unless a line says otherwise. Read-only: nothing under executor/, watchtower/, data/ was touched. Builds on EXEC-SPEED-DEEP-DIVE-2026-09-06.md and FOLLOWER-ONE-PLAN-REPORT-2026-09-06.md; what those two already establish is not repeated.

0. TLDR

  1. Spend nothing on Alchemy. The free tier is 30M compute units a month at 300 CU/s. The push-path design needs about 17.1M CU/month and peaks around 500 CU inside a busy second against a 3,000 CU per 10 s bucket. About 1.75x headroom on volume, 6x on burst.
  2. The paid tier buys exactly one measurable thing: eth_getLogs with no 10-block range cap. That is worth having as an insurance switch, not as a spend. Alchemy Pay As You Go has no base fee ($0.45 per 1M CU), so enabling it costs $0 until something calls it.
  3. RPC is not the remaining bottleneck and Alchemy fixes none of the seconds. Wallet buy to our live fill overnight was p50 21.4 s (n=6, an anecdote). Split: 0.4 s chain and push, 4.2 s our poll, 1.4 s pricing, 15 to 17 s the follower. The follower is about 70% of the budget and it is all our own code.
  4. The sequencer-feed detector (D2) is structurally dead and should be retired. Every proven-wallet buy on this chain is submitted by a relayer, not by the wallet: 21 of 21 sampled transactions had tx.from different from the buying wallet. The feed matches on tx.from, so it found 2 proven hits in 3,054,061 transactions overnight.
  5. The Alchemy websocket logs subscription is the working push path and it is already running. It saw 119 of 119 of the night's feed rows, at a median 4.02 s earlier than our poller wrote the row. That is the D2 cutover, retargeted. Detection 4.2 s to under 1 s.
  6. The Relay 429s were self-inflicted. 508 of them between 04:00 and 05:54 UTC, all inside our own exit-retry storm on three rugged lots, against a documented free limit of 50 /quote per minute. A free self-serve key plus elevated limits on request fixes it, and backing off the retry cadence fixes it better. No money.
  7. The 87-minute detection outage was invisible because nothing watches it. No sentinel on the crontab reads shadow-state.json watch.lastAt; the process never crashed, so the flap alerter had nothing to see. Cheapest reliability win on this list, zero dollars.

The one-line answer to Thomas: free tier plus the push path is enough, spend nothing on infrastructure. The seconds that are left are in our own follower code, and they are already half-built.


1. Alchemy, live facts

Fetched 2026-09-06 from alchemy.com/pricing, alchemy.com/docs/reference/pricing-plans, alchemy.com/docs/reference/throughput, alchemy.com/docs/reference/compute-unit-costs.

Free Pay As You Go Enterprise
price / month $0 no base fee, $0.45 per 1M CU to 300M, then $0.40 custom
CU included 30,000,000 none (usage billed) custom
throughput 300 CU/s (docs) / "500 CU/s (25 rps)" (pricing page) 10,000 CU/s (300 rps) from 1,000 rps
eth_getLogs block range 10 blocks unlimited unlimited
apps / webhooks 5 / 5 30 / 100 200 / 500
websockets, archive yes yes yes
Robinhood Chain yes, robinhood-mainnet.g.alchemy.com yes yes

Three things to know about the shape of the limit:

CU costs of the methods we actually call: eth_blockNumber 10, eth_getBlockByNumber 20, eth_getTransactionReceipt 20, eth_call 26, eth_getLogs 60, eth_estimateGas 20, eth_getBalance 20, eth_sendRawTransaction 40. Websocket subscriptions are billed by bandwidth, about 40 CU per 1,000-byte event.

1.1 What we actually consume

Measured this morning from two live reads of data/watchtower/shadow-state.json, 451 s apart (04:33:14Z and 04:40:45Z):

counter delta over 451 s per day
watch.passes 66 12,640 (8.78 passes/min)
watch.calls 138 (2 per pass plus 6 block-timestamp lookups) 25,280 + ~5,560
pricing.receipts 3 575
pricing.poolChecks 8 (eth_call) 1,533
Alchemy ws log events (feed-watch.log heartbeat) 7 in 300 s, 6,750/day at the overnight rate 6,750

Enrichment cycles: 1,410 in 24 h (/var/tmp/crypto-watchtower.log). Live buys: 5, 10 and 4 on Sep 4, 5 and 6 (data/executor/follower-fills.jsonl). The morning is quieter than the night, so the tables below use the overnight websocket rate, not the 04:40 one.

design CU / day CU / month verdict on free tier
A. today (logs on public RPC, Alchemy for blockNumber + block ts + receipts + eth_call + ws) ~0.56M ~16.8M fits, 1.8x headroom
B. log polling through Alchemy (the 01:03 to 07:08 config that died) ~10.6M ~317M 10x over. dies, and it did
C. push path, poller demoted to fallback (recommended) ~0.57M ~17.1M fits, 1.75x headroom

The design B arithmetic is worth writing out, because it is the whole story of last night's outage:

Design A, itemised: eth_blockNumber 12,640 x 10 = 126k CU; block timestamps 5,560 x 20 = 111k; ws log events 6,750 x 40 = 270k; receipts 575 x 20 = 12k; pool eth_call 1,533 x 26 = 40k. Total ~559k CU/day.

Design C, itemised: ws log events 270k; one receipt per hit so the cutover can decode the buy, 6,750 x 20 = 135k; block timestamps 60k; fallback eth_blockNumber at 30 s, 29k; pool and decimals eth_call 45k; the follower's whole buy and sell path if it moves to Alchemy, 30k. Total **569k CU/day, 17.1M/month**.

Peak CU/s under C: the worst block seen overnight put 4 proven wallets in one transaction (ws-detect.jsonl, tx 0x4fb4f063…, three rows plus one). Four ws events (160 CU) plus four receipts (80 CU) plus decimals lands near 500 CU inside a second, against a 3,000 CU bucket.

1.2 So does the upgrade buy anything?

One thing, and it is real: eth_getLogs with no block-range cap. That is the difference between design B being impossible and being trivial (2 calls x 60 CU = 120 CU per pass, 17 CU/s). It matters for exactly two jobs: the fallback poller when the socket is down, and backfilling a gap after an outage. Both are insurance.

The upgrade buys no latency, because the latency win is already ours for free. Measured this morning, five eth_blockNumber samples each, alternating: Alchemy p50 67 ms, public p50 150 ms. Alchemy is now the faster endpoint by about 83 ms per call, a change since the deep dive's 01:05 addendum, which measured no HTTP win. The free tier delivers that; a paid tier would not deliver more of it. The catch is that the process still paying the 150 ms is the follower (section 2.4).


2. Where the seconds go today

2.1 Detection: fixed, and about to be fixed again

measure Sep 6 deep dive (00:25) this morning source
wallet buy to watch tick, last 100 p50 10.5 s, p90 25 s, max 36 s p50 3 s, p90 6 s, max 84 s shadow-state.json watch.lags
wallet buy to feed row, overnight p50 12 s, p90 34 s p50 4 s, p90 6 s, max 46 s (n=141) feed.jsonl lagSec
same, since 00:00 UTC p50 4 s, p90 6 s, max 10 s (n=64) feed.jsonl lagSec
cycle duration p50 27.2 s p50 24.3 s, p90 59.0 s, max 86.3 s (n=1410) watchtower log

D1 worked. The lag distribution is now 37% under 2 s, 44% at 3 to 5 s, 18% at 6 to 10 s, one outlier at 84 s (watch.lags, n=100).

2.2 The push path: 119 of 119, and 4 s earlier

Joining ws-detect.jsonl (wsSeenAt) to feed.jsonl on transaction hash, over the window the socket has been up (2026-09-05T21:58Z onward):

measure value
ws rows / distinct transactions 1,874 / 1,170
distinct proven wallets / tokens seen 52 / 176
transactions with more than one proven receiver 373 of 1,170
feed rows in the window that the socket also saw 119 of 119
feed row written minus wsSeenAt min 0.99 s, p50 4.02 s, p90 6.41 s, max 10.04 s
socket reconnects overnight 0 (feed-watch.log heartbeat)

Zero misses and a 4 s median head start. The 0.99 s floor is the collector's own pricing work, not the socket, so the socket itself is arriving essentially at block time.

2.3 The sequencer feed is structurally dead

The deep dive's D2 plan was to make feed-watch.mjs (the Nitro broadcast feed at wss://feed.mainnet.chain.robinhood.com) the buy source. Overnight it processed 3,054,061 transactions at 132/s and produced 2 proven hits (feed-detect.jsonl has exactly 2 rows).

The reason, measured this morning: feed-watch.mjs:177 matches on tx.from, and on this chain that is never the trading wallet. Twenty-one transactions sampled from ws-detect.jsonl across the whole night, from read from eth_getTransactionReceipt via Alchemy:

sample result
8 most recent distinct transactions 8 of 8 relayed, all by the same address 0x6793ce0fab38a57ab2b4c308fb3981161e2da005
13 spread evenly across 21:58Z to 04:28Z 13 of 13 relayed, 12 distinct relayer addresses, one receipt not found
self-submitted (tx.from == the buying wallet) 0 of 21

Robinhood app swaps are gas-sponsored through a rotating relayer pool. A tx.from filter cannot see them. The 2 hits it did get (both AvgJoesCrypto, 01:18Z) are the exception, not the rule.

This does not make the feed useless, it makes it expensive: to use it you would have to decode the swap calldata per venue to find the beneficiary, which the deep dive already declined in section 4 for a 0.34 to 1.96 s prize. That call still stands, and it now has a second reason behind it.

2.4 The follower: 15 of the 21 seconds

End-to-end for every live buy since 18:00Z, joining feed.jsonl (buyTs, ts, tx), ws-detect.jsonl (wsSeenAt) and follower-fills.jsonl. Seconds after the wallet's own buy:

token ws saw it feed row signal id our fill
hoodrat (before socket) 13.2 10.2 27.9
AJC 0.2 5.5 4.0 21.4
ZEP 0.4 6.1 4.6 16.8
LIQLORACLE 0.9 4.4 3.4 21.3
PEZZED 0.4 6.0 4.7 19.0
PEZ 0.4 6.9 5.9 23.8
p50 (n=6) 0.4 6.0 4.6 21.4

The budget at the medians (percentiles do not add, so read these as the shape of the typical trade, not as an exact sum): 0.4 s chain and push · 4.2 s our detection poll · 1.4 s collector pricing · 15 to 17 s the follower. Signal id to fill measured directly is 12.2 to 18.0 s, p50 17.7 s, n=6, matching the deep dive's 17.4 s p50 over 14 buys. n=6 is an anecdote; the shape is not, because it matches the larger sample.

The one-plan build is deployed. executor/follower.mjs (mtime 05:51) contains reusePlanMs and fileWaker, and pid 1927592 started 05:53. There has been no live buy since, so planReused and roundTripSource appear on zero fill rows and the 9.00 s to 1.82 s A/B has no live confirmation yet. Every number in the table above is pre-deployment.

The follower is still on the public RPC. executor/routes.mjs:22 hard-codes RPC_URL = 'https://rpc.mainnet.chain.robinhood.com' and exec.mjs:92 builds its provider from it. The D3 follower cutover in the deep dive's section 3 has not happened. At 83 ms per call of measured difference and roughly ten sequential provider calls in a buy, that is about 0.8 s left on the table for free.

2.5 The exit path was never optimised

F1 and F4 land on sells (they live in planTrade/measureImpact). F2 (plan reuse) and F3 are buy-only, and the exit ladder re-plans on every rung.

measure value source
live exit fills, all time 18 follower-fills.jsonl
requote lines, all time 121 follower log
of which "estimateGas reverted" 117 follower log
LIQLORACLE, the night's one live winner all 3 clips needed a requote, 2 plans each follower log lines 3387 to 3511
gas per fill p50 $0.51, max $3.86, $28.48 total over 37 follower-fills.jsonl gasUsd

The three LIQLORACLE exits cost $6.45 of gas on a $185 exit, 3.5%, and each one paid for a discarded plan first.


3. The rest of the stack, one verdict each

A paid or dedicated RPC. Not yet, and probably never. Alchemy free covers design C with 1.75x headroom (section 1.1) and is already the faster of the two endpoints (67 vs 150 ms). QuickNode's Build tier is $49/month for 80M credits at 50 rps and is also a documented Robinhood Chain provider (docs.robinhood.com/chain/connecting), but it buys the same thing Alchemy PAYG buys, for a fixed fee instead of a usage fee. A self-hosted node is out: the docs ask for 8+ cores, 64 to 128 GB RAM, several TB of NVMe, plus an L1 execution endpoint and an L1 beacon endpoint (docs.robinhood.com/chain/run-a-full-node), which is a $300 to $600 a month box and an operational job, to save 67 ms on a path where we are losing 15 s in our own code. The honest recommendation is to enable Pay As You Go and leave it unused, because it has no base fee and it removes the 10-block cap the moment the poller needs to backfill.

Kyber. Nothing to buy, already done right. routes.mjs:58 and :81 already send x-client-id, which the docs name as the thing that avoids the stricter unauthenticated limits. Kyber's error profile in our logs is 915 HTTP 400 "route not found" and zero 429s across the whole follower log: those are dead tokens, not rate limits. An X-Api-Key through api.kyberswap.com/swap/ exists for higher limits via their BD team; we have no evidence we need it.

Relay. Get the free key, then fix our own cadence. Documented self-serve limits are 50 /quote per minute; elevated limits (10 /quote per second) are "available on request, applied per key", with no price stated (docs.relay.link/references/api/api-keys). We send no key at all (routes.mjs:121 sets only content-type). Our 686 lifetime relay 429s cluster in exactly two windows: 98 on Sep 4 19:00 to 21:00 (the CYBERDOG exit storm) and 508 on Sep 6 04:00 to 05:54 (the ZEP/PEZZED/PEZ storm), where the follower ran 623 exit attempts in one hour, about 10 a minute across three lots that had no route to build. That is our retry loop walking into a 50/minute limit, not Relay being stingy. Both fixes are free; the backoff is the one that matters.

DexScreener paid API or an alternative. No. DexScreener failed 3 times in 24 h in the watchtower feed against 1,084 Blockscout failures, and publishes no paid tier. The real weakness is not rate limits, it is staleness on a drained pool, which is what cost us ZEP, PEZZED and PEZ, and which the 05:53 build already fixes on-chain by asking the PoolManager. Codex.io does index Robinhood Chain, but its free tier is 10,000 requests a month with no websockets, and real-time starts at $350/month (codex.io/pricing). That is not worth it for a second opinion we can read from the pool for 26 CU. GeckoTerminal is already wired in and failed 20 times in 24 h. Birdeye and Mobula were not checked because the Codex number already sets the ceiling on what this category is worth.

A second executable mark source. Worth building, costs nothing. shadow.mjs:846 builds markSources as {kyber, relay: null, univ4: null, print}: two of the four slots are hard-coded null, so the only "second opinion" is the DexScreener print, which disagrees with an executable quote by design. 146 disagreements on 4,644 marks (3.1%) and markRefuseOnDisagree cannot be turned on until a real second source exists. The univ4 pool read is already implemented for poolChecks and would fill that slot for 26 CU a mark. This is the price-source answer, and it is a build, not a purchase.

Blockscout API key. No. 1,084 failures in 24 h on /api/v2/tokens, of which 826 are HTTP 500, 85 are 503, 54 are 502 and 116 are timeouts. Zero are 429. A key does not fix a server returning 500.

Mempool or pre-confirmation view. It exists, and it is the sequencer feed, and it is not worth the decode. Single sequencer, no public mempool, no builder auction. The feed is the pre-confirmation channel and it does give transactions 0.34 to 1.96 s before the receipt (deep dive section 2.3, 8 samples), but section 2.3 above shows the cheap tx.from match cannot see our wallets. Buying that 1 s means writing a per-venue calldata decoder. Against a post-build budget of 5 to 6 s, it is the last 20%, not the first.

Transaction submission tricks. No, unchanged. 0.10 s blocks, single sequencer, first-come ordering, no priority auction. Chain gas price read this morning: 0.41 gwei. Direct sequencer submission saves one HTTP hop and was already measured pointless. Private transactions and anti-MEV have nothing to protect against here.

Monitoring. Free, and the biggest reliability gap on the list. Nothing on the crontab reads shadow-state.json watch.lastAt. service-flap-alerter.sh watches systemd, and during the 05:41 to 07:08 outage crypto-watchtower.service stayed active running the whole time with a dead watch loop inside it, so there was nothing to flap. crypto-follower-v2-status.sh exists in scripts/sentinels/ but is not scheduled and does not read lastAt. A Grafana or a paid alert service would not have caught this either: the missing thing is a liveness assertion on the data, and our existing tg-alert.sh sentinel pattern already does that shape for a dozen other jobs.


4. Elite, defined

What a top bot achieves on a fast EVM chain, from DEGEN-BOT-ANALYSIS-2026-09-04.md plus a web check this morning:

stage elite ours today ours after the deployed build + ws cutover
detect 0.15 to 0.3 s (websocket is 150 to 300 ms behind chain; gRPC streams under 300 ms; Solana Yellowstone 10 to 40 ms) 4.2 s (poll) ~0.4 s (measured wsSeenAt)
decide and price ~0 s. They do not quote. Expected output is computed locally from pool state, slippage is the only guard 1.4 s collector + a quote round trip ~1.4 s
quote to ready-to-send ~0 s 9.0 s (A/B before arm) 1.8 s (A/B after arm, not yet live-confirmed)
send and mine 0.15 to 0.4 s ~2.5 s ~2.5 s, ~1.7 s if the follower moves to Alchemy
wallet buy to fill under 2 s 21.4 s ~6 s

The gap after both changes is about 4 s against an elite 2 s, and it is no longer network, it is three things in order of size: the send and receipt block (2.5 s), the collector's serial pricing (1.4 s), and the exit path, which has had none of this work. The single item that closes the most of today's gap is the websocket cutover: 3.6 s for zero dollars. The one already paid for is the one-plan build: ~7 s, sitting deployed and unproven.

The deeper structural gap is the one the degen analysis named on Sep 4 and this pass did not change: they do not quote before they trade. Every second we still spend is spent asking an aggregator what a price is. That is a design change, not a purchase.


5. Recommendations

Ship now (config only, reversible)

  1. Leave WATCH_LOGS_VIA_ALCHEMY = false. The 07:08 fix is correct and section 1.1 is the arithmetic proving it can never be turned back on under a 10-block cap.
  2. Nothing else. Every remaining item needs code or an account.

Build (code, with the test that proves it)

  1. Retire the tx.from sequencer-feed detector; cut shadow.mjs over to the Alchemy websocket logs subscription as the buy source, with pollWallets as the fallback. Test: replay the 1,874 ws-detect.jsonl rows through the buy-shaped filter and assert it reproduces the 119 feed.jsonl rows in the window and nothing else. Saves 3.6 s of the 21.4 s, and takes the log poller off the hot path. This is the highest-value item on the page.
  2. A watch.lastAt staleness sentinel on the crontab, in the existing scripts/sentinels/ shape: alert if shadow-state.json watch.lastAt is older than 3 minutes, or if the ws socket's heartbeat hit counter has not moved in 15 minutes. Test: point it at a frozen copy of the file and assert it fires. Would have caught the 87-minute outage in 3 minutes. Costs $0.
  3. Move routes.mjs:22 and exec.mjs:92 onto the shared rpc-url.mjs resolver, with redact() on the follower's log and alert sinks first (the deep dive's D3, still not done). Test: the existing dry buy and dry sell on a live token, plus an assertion that the key appears nowhere in the log after a failed send. Buys ~0.8 s and removes the public endpoint's 429s from the trading path.
  4. Back the exit retry off. 623 attempts in an hour against a 50/minute Relay limit is our bug. An exponential backoff per lot, and a hard stop once v4PoolDrained has already written the lot off (that check shipped 05:53 and the storm still ran). Test: a stubbed engine with a permanently unroutable lot, asserting attempt count over a simulated hour stays under 50.
  5. Apply the F2/F3 treatment to the sell path. 121 requotes for 18 exits, and the night's one live winner paid a discarded plan on all three clips.
  6. Fill the univ4 slot in shadow.mjs:846 markSources from the pool read that poolChecks already does, so markRefuseOnDisagree has a real second opinion to work with. This is the "alternative price source" answer, and it costs 26 CU instead of $350 a month.
  7. Confirm the one-plan build live. First live buy after 05:53, check the fill row for planReused: true and roundTripSource: 'plan'. If either reads the slow way the 9.0 to 1.82 s win is not actually firing.
  8. Fix test-univ4.mjs. It still throws on the CYBERDOG fixture, whose LP was pulled Sep 4. It has been reported as pre-existing twice now; that is how a broken test becomes permanent.

Needs Thomas (money, accounts, or a call only he makes)

item price the single measurable thing it buys
Alchemy Pay As You Go (upgrade the existing account) $0 base, $0.45 per 1M CU. At design C's 17.1M CU/month it would bill about $8/month if it billed at all, and $0 while we stay under the free 30M eth_getLogs with no 10-block range cap, so the fallback poller and any post-outage backfill cannot spiral. Nothing else. Not needed for the push path
Relay API key (free, self-serve at the Relay dashboard) + email them for elevated limits $0 /quote from 50/minute to 10/second. Removes the 508-per-storm 429 class entirely, and makes the exit ladder's cadence a design choice instead of a wall
sysctl fs.inotify.max_user_instances=512 on Bot-PC (root, not a worker's call) $0 takes F5's waker off its 250 ms polling fallback. Worth about 0.25 s

Nothing on this page needs a credit card. If Thomas wants to spend anyway, the honest order is: Alchemy PAYG first (because it is free until used), and nothing second.


6. Open questions

7. Sources

data/watchtower/shadow-state.json (watch.lags, watch.passes, watch.calls, pricing.*, markSources) · data/watchtower/feed.jsonl (ts, buyTs, lagSec, tx, signalIds) · data/watchtower/ws-detect.jsonl (wsSeenAt, tx, wallet, block) · data/watchtower/feed-detect.jsonl (2 rows) · data/executor/follower-fills.jsonl (ts, side, signalId, gasUsd, route) · /var/tmp/crypto-watchtower.log (429 counts by host and hour, cycle durations, feed failures) · /var/tmp/crypto-feed-watch.log (heartbeat counters, reconnects) · /var/tmp/crypto-follower.log (relay/kyber HTTP status histograms, requote lines, EXIT FAILED cadence) · watchtower/watchtower.mjs (lines 132 to 152, 846 to 870) · watchtower/shadow.mjs (pollWallets 566, markSources 846) · watchtower/feed-watch.mjs (line 177) · executor/rpc-url.mjs, executor/routes.mjs:22, executor/exec.mjs:92 · executor/follower.json · live eth_getTransactionReceipt reads via Alchemy (21 transactions) and eth_blockNumber latency samples (5 per endpoint), 2026-09-06 04:33 to 04:40Z · alchemy.com/pricing, alchemy.com/docs/reference/pricing-plans, alchemy.com/docs/reference/throughput, alchemy.com/docs/reference/compute-unit-costs, alchemy.com/docs/chains/*/eth-get-logs · docs.relay.link/references/api/api-keys · docs.kyberswap.com/developer-guide/start-here · docs.dexscreener.com/api/reference · codex.io/pricing, docs.codex.io/networks · quicknode.com/pricing · docs.robinhood.com/chain/connecting, docs.robinhood.com/chain/run-a-full-node · chainstack.com and rpcfast.com copy-trade latency write-ups (2026) for the elite detection band · research/execution/EXEC-SPEED-DEEP-DIVE-2026-09-06.md, FOLLOWER-ONE-PLAN-REPORT-2026-09-06.md, DEGEN-BOT-ANALYSIS-2026-09-04.md · memory/2026-09-06.md 07:06 block.

Built 06/09/2026 22:00:01 Beirut. Source files under projects/crypto-2026-09/analysis-2026-09-06/.