Developer guide

How the code is put together, the rules it keeps, how to test it and the exact shape of every data file. Read Honesty rules first: several things that look like limitations are the point.

Repository layout#

index.html                       landing page (reads board.json, tape.json, config/public.json)
config/public.json               public operator settings, written by the admin (contract: config/README.md)
.github/workflows/refresh.yml    the data job
scanner/
  src/run.js                     Scanner job entry point
  src/category-filter.js         category filter for the job and the committed files
  src/venues/                    kalshi.js, polymarket.js, http.js (rate limit, retry, cache), types.js (interface)
  src/match/                     normalise.js (extraction), matcher.js (scoring, vetoes), pair-db.js (JSONL store)
  src/math/spread.js             cost, spread and annualisation
  src/fees/                      kalshi-fees.js, polymarket-fees.js
  test/                          subject, crossvenue, polymarket-adapter, category-filter (+ fixtures/)
  data/pairs.jsonl               the pair database (committed by the job)
  public/                        index.html, board.json, crossvenue-diag.json, assets/
terminal/
  src/onchain.js                 Polygon logs → public/data/tape.json
  src/titles.js                  Gamma titles → tape.json "markets", then the category filter
  src/category-filter.js         category filter for tape.json (also runs on its own)
  test/                          rpc, category-filter
  src/keccak.js                  keccak-256 with a self-test
  src/exchangeV2.abi.ts          exchange ABI (reference only, not imported)
  public/                        index.html, data/tape.json, assets/
admin/                           operator admin on the MIKODES Admin Kit (admin/packages/admin-kit)
design-system/                   mikodes-pm.css (base) + theme-*.css, csp-style.js, categories.js (the category rule), build.sh
docs/                            this documentation, build.mjs

Architecture#

• Static JSON, no server, no database. A scheduled job fetches each venue server-side, computes, and writes static files the browser reads. Zero running cost; the operator brings the hosting. It also answers two hard constraints: Kalshi refuses browser origins, and the spreads in question last hours to days.
• No keys. Every endpoint used is public. The code contains no credential and no default for one.
• Read-only everywhere. Venue adapters must not place orders or sign (types.js). The Terminal decodes public logs.
• The admin is outside the user path. It runs on the operator's machine and communicates with the public site only through config/public.json (and pairs.jsonl via git).

The Scanner job (run.js)#

1. Ingest both venues in parallel. Each adapter returns {venue, ok, proven, markets, error, fetchedAt, provenance}. A failed venue returns ok: false with no markets.
2. Cross-venue match: blocking pass (candidatePairs, at least 3 shared title tokens of length ≥ 3, up to 20,000 pairs), then scorePair on each. Every pair is offered to the pair database. Only pairs a person confirmed (and not flagged for re-review) are priced.
3. Intra-Kalshi exercise: the matcher runs Kalshi against Kalshi over up to 900 markets with volume (4 shared tokens, 40,000 pairs max, same event skipped). Not a spread; it exercises the matcher on real prose and fills the review queue.
4. Single-venue coherence: YES + NO on the same binary market, and all legs of a mutually exclusive event (2 to 40 legs). An event is exhaustive only when a leg matches the catch-all pattern (other, none, field, no one, …); only exhaustive rows can count as positive.
5. Write board.json, crossvenue-diag.json (top 150 cross-venue pairs by confidence, with vetoes) and the sorted pairs.jsonl.
6. Category filter. Right after the fetch, markets whose label (design-system/categories.js) is in categories.hidden of config/public.json are dropped from both venues' lists, so they are never matched, priced, stored or written. venues[].hiddenByCategory counts them; board.category_filter records the list used. node src/category-filter.js applies the current setting to the committed board, diag and pair files without fetching (counts measured during the run are kept and the block says so).

Options: --contracts N (position size, default 1000), --max-events N (Kalshi, default 1200), --no-depth, --out path. Environment: POLYMARKET_CLOB_HOST. The HTTP helper caches responses under data/.cache/ per call site (ttlMs), rate-limits with a token bucket and retries a bounded number of times.

The matcher (0.3.0)#

Claiming that two contracts on two venues are the same contract is the hard part; if the claim is wrong, the user holds an unhedged position without knowing it. So text similarity only filters, and the verdict comes from structured dimensions. buildProfile extracts tokens, predicates (a lexicon of lifecycle groups such as nominated / wins / certified / inaugurated), numeric thresholds, deadline kind, resolution-source kinds, subjects, persons, years, parties and headline actions.

Confidence is the weighted mean over the dimensions that could be evaluated; a missing dimension dilutes rather than scores zero. Status: any veto → rejected; confidence ≥ 0.78 with at most one unevaluated weighted dimension → candidate; else review. confirmed is set only by a person, only in the pair database. Partial subject overlap ("Newsom v. Vance" against "Newsom v. Rubio") goes to review, not veto: a false veto costs a glance, a missed one ships a false match.

Last runner result (2026-09-23): 23,509 cross-venue pairs, all rejected; vetoes fired: settlement 23,509, year 12,888, source 12,838, subject 6,355, predicate 3,763, threshold 3,417, party 1,976. The strongest semantic matches differ in settlement because Kalshi's contract has no deadline and Polymarket's has one. Candidate generation is dominated by Kalshi's long numeric ladders. If you change the matcher, bump MATCHER_VERSION in matcher.js: it flags every human decision for re-review.

The pair database#

scanner/data/pairs.jsonl, one JSON object per line, sorted by pair_id so diffs stay reviewable.

• pair_id = first 20 hex characters of SHA-256 over the two sorted venue:id strings; independent of argument order.
• A new pair is stored only if the matcher did not reject it. A stored pair is always updated, including a demotion, so an improved matcher can retire a pair it used to like.
• A human decision (review.decision) is never changed by the machine. If a market's title, outcome, rules, sources or expiry change (a 16-hex fingerprint), or the matcher version changes, the decision is flagged needs_rereview with a reason.
• Each row keeps provenance: endpoints, whether each venue was proven, and the last 50 run ids.
• A corrupt line is skipped on load; it does not destroy the rest.
• The admin writes decisions through the same PairDb class (admin/src/review.ts), so the format cannot drift.

Spread math#

• Two-leg pair: buy YES on A and NO on B; exactly one pays $1. grossSpread = 1 − (yesAsk + noAsk). Size is capped by the smaller visible ask size when both are known.
• netEdge = grossSpread × n − fees; capitalLocked = cost × n + fees; returnOnCapital = netEdge / capitalLocked.
• annualisedSimple = returnOnCapital × 365 / days (primary). annualisedCompounded is secondary and null when the exponent exceeds 400.
• Any unknown fee makes the row unpriceable with a stated reason; it is never priced at zero fee.

On-chain decoding (Terminal)#

• keccak.js is a small keccak-256. Node's sha3-256 uses different padding and gives wrong Ethereum topics. The self-test checks "", "abc" and Transfer(address,address,uint256).
• Topics are computed from the event signatures: OrderFilled(bytes32,address,address,uint8,uint256,uint256,uint256,uint256,bytes32,bytes32), OrdersMatched(bytes32,address,uint8,uint256,uint256,uint256), FeeCharged(address,uint256).
• OrderFilled data words: side, tokenId, makerAmount, takerAmount, fee, builder. Side 0 is BUY and the maker pays collateral. Price = collateral ÷ tokens (both 6 decimals); a price outside 0..1 becomes null. A zero builder word means no builder.
• The window is pinned at both ends (blocks argument, default 40) and labelled with the chain's own timestamps.
• onchain.js writes every token and up to 5,000 fills; titles.js resolves titles and calls filterTape() in src/category-filter.js, which drops tokens and fills of hidden markets and keeps the 60 tokens with the most notional and the latest 300 fills (newest first). The file also keeps the latest 100 fee events and every builder; totals (kpis, counts, series, builders, per-token builderFills) are computed over every fill in the window, hidden markets included.

The admin#

• src/manifest.ts: sections and checks. Standard Kit modules (branding, legal, access minus blocked countries, integrations, notifications minus email) plus Venues and Categories. No money section and no revenue source: the product is read-only.
• src/public-config.ts: builds and atomically writes config/public.json; the hidden list is limited to the labels the pages know.
• src/checks.ts: the Status checks and the Integrations test buttons.
• src/app.ts: createAdmin() mounts the Kit and the product routes under /admin-product/ (session required; deciding needs manager or owner). No revenue callback: the product has no revenue of its own. src/server.ts only reads .env, opens the SQLite store and listens on loopback.
• The Admin Kit lives in admin/packages/admin-kit/. You may change it (see its LICENSE.md); keep your changes in a separate commit so a kit update from the author is easy to merge.

Design system and docs build#

sh design-system/build.sh   # base + theme → each app's public/assets/app.css
node docs/build.mjs         # base + docs theme → docs/assets/docs.css; sidebar, contents, pager, search index

Edit design-system/mikodes-pm.css (structure, tokens) or a theme-*.css (identity), never an app's assets/app.css. The Scanner (teal, round mark, filled nav pill) and the Terminal (electric blue, square mark, underline nav) must not look like one app. The docs use the base's neutral accent. design-system/README.md lists every token and component. The committed app.css files match a fresh build (checked 2026-09-25).

Tests#

cd scanner
node test/subject.test.js              # subject veto cases
node test/crossvenue.test.js           # 37 real false cross-venue pairs stay rejected + conservative cases
node test/polymarket-adapter.test.js   # 13 offline cases: keyset paging, books batching, bisection
node test/category-filter.test.js      # board, diag and pair database filtering

cd ../terminal
node src/keccak.js                     # 3 keccak-256 vectors
node --test test/*.test.js             # 11 offline cases: own RPC order, fallback, URL never leaked; category rule and tape filter

cd ../admin
npm test                               # 19 tests: manifest, read-only settings, category filter, policies, product routes
npm run typecheck

The Scanner and Terminal tests need no network. The Polymarket adapter is tested only offline, against fixtures shaped like the SDKs' own test payloads; live behaviour is proven only by runner runs.

Data contracts#

board.json (schema_version: 1)#

crossvenue-diag.json#

generated_at, run_id, matcher_version, pairsScored, verdicts, vetoes (count per dimension), category_filter, top[] (up to 150 highest-confidence pairs, hidden categories removed: confidence, status, vetoes, and a / b with id, title, outcome, expiry). Diagnostics only; never shown as a spread.

tape.json#

config/public.json#

Documented field by field in Admin and configuration and in config/README.md. To add a field: add it to admin/src/public-config.ts (and the manifest if it is a setting) with an "off / empty" default and keep /1. To rename, remove or retype a field: bump the schema to /2 and update every reader in the same change.

Adding a venue#

Write a new adapter in scanner/src/venues/ that implements the interface in types.js (read-only, null for anything unpublished, provenance on every market), add its fee model in src/fees/ returning {known:false} for anything unverified, and a legFee branch in spread.js. The matcher and the board do not need to know which venue they are looking at.