# Torrance Watch: A Technical Case Study ## Building a nonpartisan voter guide whose every number can be checked by the reader A build record for torrancewatch.org, April 1 to August 19, 2026. One person, one laptop, 498 merged pull requests. 2,543 campaign donors matched against 6.4 million voter records, and the work of proving the matches right. > Every figure in this document is generated from the database and the accuracy report > by `scripts/build-record/derive-figures.mjs`, and a build fails if this file and > `figures.json` disagree. --- ## What it is A nonpartisan voter guide for a California city of about 96,000 registered voters. It reports where campaign money comes from. It matches each donor to a record in the county's list of registered voters, so a reader can see who is backing each candidate. It covered the June 2 2026 election and the three-week count that followed, and every number on it is footnoted to a public filing. I moved to Torrance and wanted to know who was running for city council. That turned out to be hard, and partly by design. California city races are nonpartisan by law, so nobody runs as a Democrat or a Republican and the ballot says nothing about party. The politics don't stop. They just stop being written down. What is left is scattered across county voter files behind paywalls, donor spreadsheets and council minutes buried in PDFs, and the local paper that used to read all of that for you has thinned to almost nothing. Mailers fill the gap. I got them: a party's colors, a party leader's photo beside a candidate's, an endorsement implied and never stated. > "City coverage has thinned, the public records are scattered, and mailers are built to > persuade. Torrance Watch is the page in the middle." — torrancewatch.org/about So the site had to be useful to a reader who does not trust it. Every claim on it should be checkable somewhere else. Rather than labelling a candidate as leaning one way, it publishes five separate party-related facts about each one, every fact traced to a public document. The fifth is the only one the site computes itself: the party mix of that candidate's donors, published as a group total and never as a list of people. Three editorial rules are enforced by the software rather than remembered by me. - If one candidate in a race has coverage another lacks, the site will not publish at all. That check gets tested every so often by breaking it on purpose, to confirm it still fires, because *a safeguard that fails the first time you need it just gets switched off.* - Party color appears in exactly one place, the donor group total, and never next to a candidate. - "No record found" is displayed at the same size and weight as a filled-in answer, so a blank never looks like a discovery. The donor party mix could point at one person if the group were tiny, so two rules apply. A candidate's mix is published only once enough donors have been matched that no single person can be picked out of the total, and the smallest party groups are reported together instead of separately. ### Through the election LA County counts for three weeks after election day and no wire service covers city races, so explaining the count became the most-read work the site did. Explaining it was only possible because of one estimate. "Leads by 400" means nothing until you know whether 500 or 15,000 ballots are still to be counted, so the site had to be able to say how many were left. The estimator starts from the county's own published turnout projection, subtracts the ballots already counted, and publishes a range for how many are left. It never publishes a vote count, a margin, or a winner, and when the inputs are missing the estimate does not appear on the page at all. *A forecast that cannot say "I don't know" is not a forecast.* Declaring a winner followed the same discipline, and it worked differently in two places. The results page never declared one. Publishing a call there takes a deliberate editorial step, and the name gets checked against the official list of people in that race, so a typo stops the call from going out instead of putting a wrong name on the page. The newsletter did call it, and said what the call rested on. On [June 5](https://torrancewatch.org/newsletter/2026-06-05-kalani-favored), Chen was ahead by 359, and the last three batches of results the county had released were moving toward Kalani. The newsletter said Kalani was favored to overtake him, and labeled that a read of public data rather than an official count. On [June 7](https://torrancewatch.org/newsletter/2026-06-07-kalani-leads) it reported that she had passed him, and said the three council races were settled. It kept the treasurer's race open as the only one still in doubt, with the leader ahead by 286 and mailed ballots still to count. That race was certified on June 26, decided by **309** votes. *A read you are willing to publish early is worth more than a verdict you publish once it is safe, provided you show what it rests on.* On election night the county results feed broke. The page fell back to showing zeros, which looked exactly like the normal page before the polls close, and it stayed that way for hours. *A failure that looks exactly like health needs its own detector.* This is the build record, not the site. Three things were hard. Four government systems describe the same people and none of them spell the names the same way. Thousands of judgment calls had to be handed to an AI language model that was never allowed to make the final call. And accuracy had to be measured against an answer key nobody had ever written down, which then turned out to be half the system's own output. **Stack:** Astro 7 static · Cloudflare Workers · Postgres on the laptop, never at the edge · Node pipeline scripts · Claude Code as build partner and in-session reviewer. --- ## The matcher, and where the model is **498** merged PRs and **653** commits, Apr 1 to Aug 19, one person. How many donors the system could match was settled in a day, on May 7. Everything after that was about knowing whether those matches were right, and the two biggest changes that came out of it both made the site publish *less*. **First make it exist, then make it deserve belief.** In August that discipline got its final test: the scoring settings themselves were put on trial against the evidence, and kept their job. That story is at the end of the accuracy section. Four government systems describe the same people without agreeing on spelling. "Jim Smith, donated $200" and "James Smith, registered on Maple Ave": one human or two? That question has to be answered defensibly for **2,543** donors against **6,411,869** voter registrations, and every confident answer has to show which facts it agreed on. The software that does this is called the *matcher* throughout. Version 1 ran a list of rules in order and took the first hit. It was scrapped within a day, for three reasons that turned out to be the whole problem: it could not say how sure it was, it never looked at the second-best possible match, and a near-miss had nowhere to go except silently into the bin. Version 2 scored the evidence, required a clear lead over the runner-up, and sent close calls to a queue for review. *Doubt became a thing the system could express and route to a human.* Every version after that bought better measurement rather than more matches. Facts that agree earn points, weighted by how much each one narrows the field, because a shared zip code means little in a city of 96,000 and a shared uncommon first name means a great deal. Treating every agreement as an equal yes-or-no vote produces confident nonsense. The last name earns nothing at all: it is only how a record gets onto the shortlist in the first place, not evidence that the match is right. **The requirement it took three months to arrive at.** A link needs a high enough total score, and a clear lead over the second-best possible match, and at least one fact beyond the name to agree. The lead test alone only asks whether anyone else came close, and for an unusual name nobody does, so a single candidate could clear it on the name alone. Being the only Emily Wang in the county is not evidence that you are the right Emily Wang. Adding that third requirement moved the links the matcher makes on its own from 13 wrong in 134 to none wrong in 79, at a cost of about 100 links, which now go to a person instead. *A name by itself is never enough to publish a link.* **Where the 2,543 donors stand today:** **1,182** linked by the matcher on its own · **177** waiting on review · **797** with no match found · **387** settled by review, by hand, or by copying the answer from another record that turned out to be the same donor. Some cannot match at all: **305** are organizations, and a few hundred live outside LA County, which is the only voter file this matches against. **Coverage, measured against the donors who could match.** An organization has no voter registration, so counting companies in the total guarantees they show up as failures. Against the **2,238** individual donors, **1,569 of 2,238** are linked, and **1,182** of those were linked by the software alone, with no human reviewer. The accuracy chart's second line divides by the whole pool instead, organizations included, which is why it sits lower. That line tracks one specific thing over time, how much the matcher settled on its own, and it is not a coverage figure. **Where the model is, and isn't.** Across **3,283** decided queue rows: - **1,944 by rule**, a standing policy written before the row existed - **1,014 by a model**, under written instructions, per row - **80 by hand**, the edge cases nothing else could settle - **245 with no stamp**, meaning nothing recorded who or what decided A standing rule decides more than the model does. The rules handle the kinds of case that can be written down in advance. The model handles what is left over, working through the queue 50 rows at a time, with each row arriving as its own short brief of the evidence. It proposes and it explains, and it **can never approve**. *The model is rented. The rules around it are the thing I actually built.* **Where the panel fits.** A *panel* here means several model runs on the same question that have to reach the same answer before it counts, which turns one opinion into a small vote. Panels appear at two points, doing different jobs. Inside this queue, a panel settled **25** donor-match rows the ordinary single runs had left undecided, and those sit within the 1,014 above. Separately, a panel ran the blind re-check described in the accuracy section, marking 200 already-published links without being shown what the matcher had picked. That second one is not a decision path at all. It is an audit. **That split was backwards on this page until Aug 17.** It credited the model with about 1,700 decisions and standing rules with about 460, near enough to inverted. The cause is worth naming, because it is exactly the failure this page is about: a number that looks fine from the inside and is wrong. Every decision in the queue is stamped with a marker naming what decided it, and the largest marker in the queue is `claude-heuristic-donor-voter-match`, at 1,149 rows. So any count that just looks for the word "claude" in the marker counts all 1,149 of those rows as the model's. The name says heuristic, which means a fixed rule of thumb, and a rule of thumb is a rule. The split above now comes from the same sorting code the accuracy report uses. That code decides what each marker means by reading the documentation of the code that writes it, rather than by the marker's name. And an automatic check compares the result against this page every time the site is built, so the two cannot drift apart again. *A label that names a tool does not tell you what did the work.* ### The privacy wall The site is built from files packed with personal data, and a conventional site with a live database puts all of it one breach away from the internet. So the data lives in one Postgres database on one laptop. The public site is a set of plain files with **no database connection at all**. Those files are written out on the laptop before anything is uploaded, and only from tables that have had the personal details stripped out. Once the site is live, there is no database behind it to break into. How it got there, in order. The split was in the design from day one. Then came written rules about what may ever be published. Then reports that listed individual people were kept out of the code repository. Then database permissions were taken back in the places where new tables had been handing them out automatically. Last came an audit, which found a route into voter data that nothing was using but nothing was blocking either, closed it, and then proved the block worked by trying the same thing again on purpose. --- ## How accurate is it, and how would you know An *answer key* is a set of cases where somebody already knows the right answer, so a system's output can be marked against it. Nobody has ever recorded which Torrance donors are which of 6.4 million registered voters, and no such key exists to buy. So for the first three months the system was graded against its own earlier output. That is a genuinely useful test, and it catches a system drifting away from what it used to do. What it cannot do is tell you whether what it used to do was right, because a system compared to itself can only ever agree. Building a real key meant deciding cases by hand, cheapest method first: - fixed rules for the unambiguous ones - then three separate model runs that all had to reach the same answer - then me by hand for the rest The key has grown as reviews come in, and it now holds **464** donors, every one decided by a person. None of those labels came from the matcher's own output, which is the entire point. The old key held 159 rows the matcher itself had produced, filed as truth, and those were half of its 313 rows. They scored 100% in every run they ever appeared in, which is what it looks like when a test only ever asks a system whether it agrees with itself. Removing them dropped the measured accuracy on the spot, and the drop tripped the automatic check that blocks the site from publishing when accuracy falls. That was the correct outcome, and the first number the project had that meant anything. Every version of the scoring code is still in the project's history, so each one can be re-run against the new key and marked the same way. That is why the accuracy chart can show numbers for May, months before anything was measuring accuracy at all. Where it stands now. *Unattended* links are the ones the matcher publishes on its own, with nobody looking; they are the ones that need proving. Those came back **104 of 104** right against the answer key. Read that as no errors found in 104 checks rather than no errors anywhere, because a sample that size can only narrow the true rate to a range, and that range runs **between 96 and 100 out of 100**. Most of those labels were written with the matcher's pick on screen, which measures agreement rather than tests it, so a second panel re-checked 200 published links without being shown what the matcher had chosen, and found **195 of 195** correct as of Aug 16. That is the stronger of the two numbers, and it is the reason the blind version was run. Two more angles on the same question. The matcher used to publish a second, lower-confidence class of link on its own as well. That class was retired in August, and it scored **70 of 74** on the same key. And every number above only grades the links the matcher actually made. None of them can see the person it never pulled up for consideration in the first place. That gap gets measured too: **394 of 401** is how often the right voter registration made it onto the shortlist of possible matches the matcher chooses from. The seven misses: two compound first names, one first-name variant, three surname spellings, and one where the verified answer itself looks wrong, which is its own kind of finding. **Checks that read what the matcher cannot.** A second pass hunts for links that cannot be true, using only facts the scoring code never looks at. An impossible age. An address that contradicts every address the donor filed. A registration that has never once voted. Two donors pointing at one registration. The principle is simple and worth stealing: a check is only evidence if it reads something the thing it grades never saw. Contradicted addresses among unattended links stand at **34 in 1,182**, down from 147 in 1,310 before a second agreeing fact was required. Two separate tests pointing the same way, and neither one was told what the other found. **The trap, and what it actually measures.** Take a link whose answer is known, delete the correct person from the list of possible matches, and make the matcher choose again. Anything it still commits to has to be wrong, because the right answer is no longer on the table. That is the appeal: it needs no labels at all, so it works long before an answer key exists. It now does that **12 of 1,182** times, down from roughly one in nine. Read it precisely, because it answers a narrower question than it appears to: how often the matcher commits *when the right person is missing entirely*. It is not the share of published links that are wrong, and the two get confused easily enough that this section exists mostly to keep them apart. **The settings stood trial, and kept their job (Aug 19).** The matcher's settings, meaning how many points each agreeing fact earns and the bar a total has to clear, were set by judgment in May and never checked against real evidence, because until August there was no honest evidence to check them against. With a real answer key in hand, they could finally be tested. The rules of the contest were written down and saved before anything ran: what would count as better, how ties would break, and which verified answers would stay sealed. The sealed ones were kept out of the search entirely, to be used once at the end. An automated search then tried **28,532** alternative settings. Thousands beat the current ones on the cases they could study; the best raised correct automatic matches on the key from **100 to 147** without a single error there. Then the sealed answers were opened. Each of the two winning settings made one confident wrong match on them. Each one picked out a voter registration for a donor that a human reviewer had already decided was not on the voter file at all. The current settings made none. So the current settings stayed, which was the pre-written rule. The winners had gained ground by tuning right up against the hardest mistake anyone had already found, and one sealed case sitting just past that line was enough to tip them into a wrong answer. *A sealed test that can veto your winner is what makes it tuning instead of wishful thinking.* Then the grader itself got checked. An independent code review found that it was counting 33 rows nobody had ever verified as if they were confirmed answers. After the fix, everything was re-graded: same verdicts, on clean evidence. The settings the site runs on today are the ones it ran on in May, but "unchanged" now means measured and confirmed rather than never examined. --- ## The numbers | Number | What it is | | --- | --- | | 498 / 653 | merged PRs / commits, Apr 1 to Aug 19, one person | | 6,411,869 / 54.5M | people in the voter file / rows of participation history, all of LA County | | 2,543 | donors matched against it: 1,182 linked by the matcher alone · 177 waiting on review · 797 no match found · 387 settled by review, by hand, or by copying the answer from another record that turned out to be the same donor | | 6,209 of 6,210 | reported contributions that are tied to a single cleaned-up donor record, after different spellings of the same name were merged into one | | 464 | donors in the answer key, every one decided by a person and none of them the matcher's own output | | 104 of 104 | links the matcher made on its own, checked correct against that key. Read as "no errors found", not "no errors exist": a sample this size narrows the true rate to between 96 and 100 out of 100, and the key leans toward hard cases on purpose | | 195 of 195 | a blind re-check of 200 already-published links, the reviewer never shown which registration the matcher had picked. The strongest accuracy figure here, as of Aug 16 | | 394 of 401 | how often the right person even reaches the shortlist the matcher chooses from, which no figure above can see. The seven misses: two compound first names, a first-name variant, three surname spellings, and one verified answer that itself looks wrong | | 70 of 74 | the lower-confidence class of link, which stopped publishing automatically in August. A second sample, drawn to represent all links rather than the easy ones, got it wrong 3 times in 13, and that is the number that retired it | | 34 in 1,182 | links the matcher made on its own whose address contradicts every address the donor filed, down from 147 in 1,310 before a second agreeing fact was required | | 12 of 1,182 | how often the matcher still commits to an answer after the correct person is deleted from its list of possible matches. Not the share of published links that are wrong | | 28,532 | alternative settings tried in the Aug 19 trial. The best raised correct automatic matches on the key from 100 to 147, then made one wrong match on the sealed answers the search never saw. The shipped settings made none there, and stayed | | ~26,000 | estimated human visitors, lifetime (range 23.6-26.9K, as of Aug 5). The raw 33,058 figure counts internet addresses rather than people, automated crawlers included, and was retired as the headline | | 2,522 / ~227K | internet addresses seen on the site's biggest day, June 8, mid-count / social views across 29 posts, both as of Aug 5 | | 15,200 | estimated humans in the peak 30 days, May 30 to Jun 28, as of Aug 5. This is the figure behind "about 1 in 3 of the 43,600 ballots cast", not the 26,000 above: that figure spans 110 days from April, and setting a four-month audience against the people who voted on one day is a different and weaker claim | | 309 / 96,606 | the margin, in votes, that decided the treasurer's race (44.8% turnout) / registered voters, the denominator behind every turnout figure | --- ## What transfers **One. Check what your answer key is made of before you trust it.** For three months this system was graded against an answer key that was half its own output. It scored 100% every time. Of course it did: it was agreeing with itself. And nothing about that looks broken from the inside. The tests pass, the number is high, everyone is happy. So before you trust a score, ask where the answers behind it came from. If your system produced any of them, throw those out, and add a check in code that keeps them out. This mistake sneaks back in dressed as good news, and memory is not a defense against it. If you have no answer key at all yet, there is a cheap stopgap: delete the true answer, run the system anyway, and count how often it still says it found something. Every one of those is a mistake you can count without knowing any right answers. **Two. If a model does judgment work for you, put it inside a contract.** The model here reads one case file at a time and proposes an answer. It cannot approve anything. Its allowed answers are spelled out in advance, and every decision is stamped with who or what made it. That stamp earned its keep: months in, the stamps showed a plain rule was doing most of the work everyone had credited to the model. Then measure the parts you assumed were working. I asked the model to rate its own confidence on every row. When I finally checked those ratings against real outcomes, "medium" did worse than "low". The rating meant nothing, so I deleted it. **Three. Forecast the thing you can actually estimate.** During the three-week vote count, the site never predicted a winner. It published the one number it could defend: how many ballots were probably left to count. And when the inputs were missing, it published nothing at all. Picking the smaller, defensible forecast is what made publishing anything possible. **Four. Write the contest rules before the contest, and keep a sealed test.** When the matcher's settings finally got tuned, three things were written down before anything ran: what "better" would mean, how ties would break, and which verified answers would stay locked away from the search, to be used once at the end. The search tried 28,532 alternative settings and found ones that looked clearly better. Then the sealed answers came out. Both winners got one wrong. The old settings got none wrong. So the old settings stayed. That was the rule, written in advance, and it is the sealed test doing its one job: telling you your improvement is a mirage before your readers find out. One more habit worth stealing: have someone check the grader too. An outside review found mine counting unverified guesses as verified answers. The verdicts held up after the fix, but I only know that because somebody looked. One person and Claude Code on a nine-week deadline. That was possible not because rigor was skipped but because rigor got cheap. You do not rebuild trust by asking for it. You rebuild it with systems willing to show their sources and publish their own error rate.