Research report · 24 September 2026
Is Jev a better gate for research findings than an LLM judge?
Not on its own. But Jev first, with an LLM second opinion when Jev is unsure, beat either judge alone. It judged all 30 test findings correctly, let no bad finding through, and cost about a tenth of a cent.
What we found
A quick definition first. The gate is the check every finding goes through before it's posted to the board. Jev is the judge inside it, but not the whole gate. Today the gate has three steps: code checks that the citations exist, Jev judges the claim against the cited text, and an LLM gives a second opinion when Jev is unsure. The original gate was one step: Jev reading only the citation text.
The original gate judged how a citation looked, not what it said.
It sent Jev the claim and the citation text, never the cited page or file. A made-up URL that looked right would pass. run 1 · R1-6, R2-8
Give Jev the source text, and do the rest in code.
Code checks that every cited file, line, and page exists, and does any arithmetic. Jev then judges the claim against the real passages. TypeSafe's own citation-check example works the same way. run 1 · R2-6 · run 3 · R2-4
Jev and an LLM tie on accuracy, but fail on different findings.
Given the same passages, Jev agreed with 28 of 30 labels and GPT-6 Luna with 27. Jev can't check arithmetic. Luna is sometimes stricter than the claim needs. eval
Jev knows when it's unsure, which makes a second opinion cheap.
Both arithmetic claims landed in Jev's "unsure" band instead of passing quietly. Sending only those to Luna fixed them, and Luna explained why, which Jev never does. eval · run 3 · R2-9
Jev is about 3 times cheaper and 6 times faster than the cheapest LLM judge.
For one check on a 2,000-token finding, Jev costs $0.000084, GPT-6 Luna $0.00025, and Claude Haiku 4.5 $0.0025. In the eval, Jev took 0.3 seconds per finding and Luna 2.0. run 3 · R2-2 · eval
Keep the current cut-offs for now.
Jev rejects below 0.5 support and asks for a second opinion below 0.8. On this set the cut-offs separate good findings from bad, but 30 findings aren't enough to call them tuned. run 3 · R1-6, R2-9
The evidence
We built 30 findings with known answers. Ten are correct. Twenty are bad in five ways: they claim more than their source says, get a number wrong, contradict their source, cite a source that doesn't mention the claim, or cite a file or line that doesn't exist. Four judges ruled on every finding.
| Old gate | Jev on sources | Luna on sources | Jev, then Luna | |
|---|---|---|---|---|
| Agrees with the label | 14 / 30 | 28 / 30 | 27 / 30 | 30 / 30 |
| Bad findings let through | 14 | 2, both flagged | 1 | 0 |
| Good findings failed | 2 | 0 | 2 | 0 |
| Sent to Luna for a second opinion | – | – | – | 5 |
| Cost for all 30 | $0.00059 | $0.00067 | $0.00194 | $0.00120 |
| Time per finding | 0.36 s | 0.34 s | 2.03 s | 0.79 s |
The old gate saw only the citation text. The other three got the same checks and the same cited passages. The labels were written by Claude, who built the tool, and haven't been reviewed independently. Differences of one or two findings are within run-to-run noise: one finding scored 0.46, 0.52, and 0.48 across three Jev calls, on either side of the 0.5 line. The combined gate's perfect score includes that finding: it scored 0.48 in that run and was rejected. At 0.52 it would have gone to Luna, which passed it when judging alone. Read the result as 29 or 30 out of 30.
How the answer took shape
- Run 1
The gate can't see the sources
The researchers read the gate's own code and found it judged citations by their wording alone. The run also exposed four bugs in the tool: prices broke in the Summary, a released thread was skipped, a blocked thread was closed with a non-answer, and every thread went to the mid-priced model.
- Changes
The gate learns to read
Code now checks references and fetches the cited lines and pages before Jev judges them. All four bugs were fixed.
- Run 2
A much stricter gate
The new gate rejected 4 of 6 findings, even after retries. The cheap-model fix worked, sending the two lookup threads to the fast tier. A blocked researcher now handed its thread back and asked for help. Was the gate right to be this strict? That needed a test.
- Eval
30 findings with known answers
The first eval run caught a bug of ours: cited table rows arrived without their headers, which made the LLM look worse than it was. With that fixed, Jev and Luna tied, but failed on different findings. That pointed to combining them.
- Run 3
Jev first, LLM second
With the second opinion switched on, Luna confirmed 5 findings Jev was unsure about, and 2 were rejected. The researchers read the eval results in the repo and used them for the final recommendation.
How simple-agent-collabs helped
simple-agent-collabs is a small research loop. It has a goal file, a shared board in plain markdown, and several LLM researchers working in parallel, with a human leading. Jev makes the quick decisions around them.
board.md.Every turn follows the same eight steps, and every finding goes through the same gate before it reaches the board:
Two researchers, no overlap
Each run had two researchers working at once, taking 7 or 8 claims between them. A researcher claims a thread before starting, so no two ever worked the same one, except T6, which was taken again after being handed back.
The right model for each thread
Jev picked a model tier for every thread. By run 3, the lookups T1 and T2 went to GPT-6 Luna, T3 to T5 to GPT-6 Sol, and the final recommendation T6 to GPT-6 Astra.
Researchers asking each other for help
In run 3, R2 took T6, the recommendation, before T5 was done. Instead of guessing, it handed T6 back and asked for T5's wording, and R1 answered. R2 took T6 again in the same round, so it worked from the code rather than R1's answer. Nobody had to step in, but the handoff shows a limit: researchers don't see posts made during their own turn.
A human in charge, with everything in plain files
The board is markdown you can read, search, and edit. The Summary is yours. Each run's board is kept, so the three runs can be compared side by side.
A tool that researched itself
Run 1 found a flaw in the tool's own gate. The fix was tested with the eval, and run 3 used those eval results to reach its answer. Each round of research improved the tool doing the research.
How the two researchers worked in this research
| Run 1 | Run 2 | Run 3 | |
|---|---|---|---|
| Gate | Citation text only | Reads sources | Reads sources, then Luna if unsure |
| Posts on the board | 17 | 15 | 17 |
| Findings | 6 | 6 | 7 |
| Rejected by the gate | 0 | 4 | 2 |
| Confirmed by Luna | – | – | 5 |
| Lookup threads on the cheap model | 0 of 2 | 2 of 2 | 2 of 2 |
Limits and open questions
- The test set is small and unreviewed. Thirty findings, labeled by the tool's author. The source files are short summaries written for the test, not copies of the real pages.
- On real research, Luna did most of the judging. It confirmed 5 of 7 findings in run 3, against 5 of 30 in the eval. Real findings are longer and cite more, so Jev is less sure of them.
- The gate treats recommendations as facts. It rejected a finding that proposed new cut-offs, because no source can back a new idea.
- Findings with unreadable sources still get posted. They're flagged, but the research recommends holding them for the human lead.
- No controlled speed comparison. The timings come from the eval runs only.
Sources
- Jev models: limits and pricing
- Jev 1.13 model jaggedness
- Confidence
- Choice primitive
- Noul primitive
- Citation check cookbook
- Jev for coding agents
- OpenAI API pricing
- OpenAI models
- Claude API pricing
- Research boards: run 1, run 2, run 3, and the goal
- Eval: the 30 labeled findings and the four-judge results