| | jev-citations | jev-sources | llm:gpt-6-luna | jev+gpt-6-luna |
| --- | --- | --- | --- | --- |
| Agrees with label | 14/30 (47%) | 28/30 (93%) | 27/30 (90%) | 30/30 (100%) |
| Bad finding passed | 14 | 2 | 1 | 0 |
| …of those, flagged for review | 0 | 2 | 0 | 0 |
| Flagged for review, all findings | 0 | 6 | 0 | 0 |
| Sent to the LLM for a second opinion | 0 | 0 | 0 | 5 |
| Good finding failed | 2 | 0 | 2 | 0 |
| Decided by code, no judge | 0 | 3 | 3 | 3 |
| Errors | 0 | 0 | 0 | 0 |
| Cost for the set | $0.00059 | $0.00067 | $0.00194 | $0.00120 |
| Average time per case | 357 ms | 335 ms | 2029 ms | 786 ms |

| Kind | jev-citations | jev-sources | llm:gpt-6-luna | jev+gpt-6-luna |
| --- | --- | --- | --- | --- |
| correct | 8/10 | 10/10 | 8/10 | 10/10 |
| overreach | 1/6 | 5/6 | 5/6 | 6/6 |
| wrong-number | 1/5 | 4/5 | 5/5 | 5/5 |
| contradicted | 1/3 | 3/3 | 3/3 | 3/3 |
| unsupported | 2/3 | 3/3 | 3/3 | 3/3 |
| fake-citation | 1/3 | 3/3 | 3/3 | 3/3 |

Disagreements with your labels:

- jev-citations O01 (overreach, label fail): said pass. supported (0.65)
- jev-citations O02 (overreach, label fail): said pass. supported (0.78)
- jev-citations O04 (overreach, label fail): said pass. supported (0.73)
- jev-citations O05 (overreach, label fail): said pass. supported (0.82)
- jev-citations O06 (overreach, label fail): said pass. supported (0.53)
- jev-citations W01 (wrong-number, label fail): said pass. supported (0.91)
- jev-citations W02 (wrong-number, label fail): said pass. supported (0.89)
- jev-citations W03 (wrong-number, label fail): said pass. supported (0.50)
- jev-citations W04 (wrong-number, label fail): said pass. supported (0.90)
- jev-citations X01 (contradicted, label fail): said pass. supported (0.66)
- jev-citations X02 (contradicted, label fail): said pass. supported (0.64)
- jev-citations U03 (unsupported, label fail): said pass. supported (0.52)
- jev-citations F01 (fake-citation, label fail): said pass. supported (0.76)
- jev-citations F02 (fake-citation, label fail): said pass. supported (0.54)
- jev-citations C01 (correct, label pass): said fail. not supported (0.43)
- jev-citations C08 (correct, label pass): said fail. not supported (0.47)
- jev-sources O02 (overreach, label fail): said pass with review. gate unsure the sources support the claim (0.52); gate unsure of the confidence level (0.52)
- jev-sources W05 (wrong-number, label fail): said pass with review. gate unsure the sources support the claim (0.64); gate unsure of the confidence level (0.71)
- llm:gpt-6-luna O02 (overreach, label fail): said pass. supported
- llm:gpt-6-luna C05 (correct, label pass): said fail. The passage advises doing arithmetic in code rather than asking Jev, but it does not establish that TypeSafe gives this advice.
- llm:gpt-6-luna C08 (correct, label pass): said fail. The code establishes that the finding passes when both `answers.concrete.noul` and `answers.supports.noul` are at least 0.5, but it does not establish that these values are yes-or-no answers.

Jev's numbers per case. Support under 0.50 rejects; under 0.80 is flagged for review. Certainty under 0.80 is flagged too.

| Case | Kind | Label | jev-citations support | jev-sources support | jev+gpt-6-luna support | jev-citations certainty | jev-sources certainty | jev+gpt-6-luna certainty |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C01 | correct | pass | 0.43 | 0.72 | 0.68 | – | 0.81 | 0.80 |
| C02 | correct | pass | 0.70 | 0.92 | 0.93 | – | 0.99 | 1.00 |
| C03 | correct | pass | 0.56 | 0.91 | 0.89 | – | 0.96 | 0.96 |
| C04 | correct | pass | 0.66 | 0.90 | 0.90 | – | 0.99 | 0.99 |
| C05 | correct | pass | 0.58 | 0.86 | 0.86 | – | 0.87 | 0.81 |
| C06 | correct | pass | 0.50 | 0.90 | 0.89 | – | 0.97 | 0.98 |
| C07 | correct | pass | 0.62 | 0.90 | 0.89 | – | 1.00 | 0.99 |
| C08 | correct | pass | 0.47 | 0.76 | 0.72 | – | 0.55 | 0.67 |
| C09 | correct | pass | 0.68 | 0.59 | 0.58 | – | 0.35 | 0.37 |
| C10 | correct | pass | 0.53 | 0.66 | 0.67 | – | 0.68 | 0.66 |
| O01 | overreach | fail | 0.65 | 0.09 | 0.10 | – | 0.88 | 0.87 |
| O02 | overreach | fail | 0.78 | 0.52 | 0.48 | – | 0.52 | 0.44 |
| O03 | overreach | fail | 0.24 | 0.02 | 0.02 | – | 1.00 | 1.00 |
| O04 | overreach | fail | 0.73 | 0.24 | 0.27 | – | 0.27 | 0.33 |
| O05 | overreach | fail | 0.82 | 0.06 | 0.05 | – | 0.96 | 0.97 |
| O06 | overreach | fail | 0.53 | 0.21 | 0.22 | – | 0.37 | 0.30 |
| W01 | wrong-number | fail | 0.91 | 0.04 | 0.04 | – | 0.97 | 0.97 |
| W02 | wrong-number | fail | 0.89 | 0.15 | 0.19 | – | 0.75 | 0.70 |
| W03 | wrong-number | fail | 0.50 | 0.03 | 0.03 | – | 0.98 | 0.99 |
| W04 | wrong-number | fail | 0.90 | 0.04 | 0.05 | – | 0.98 | 0.99 |
| W05 | wrong-number | fail | 0.31 | 0.64 | 0.61 | – | 0.71 | 0.56 |
| X01 | contradicted | fail | 0.66 | 0.04 | 0.05 | – | 1.00 | 1.00 |
| X02 | contradicted | fail | 0.64 | 0.08 | 0.08 | – | 0.99 | 0.99 |
| X03 | contradicted | fail | 0.26 | 0.03 | 0.03 | – | 1.00 | 1.00 |
| U01 | unsupported | fail | 0.37 | 0.03 | 0.03 | – | 1.00 | 1.00 |
| U02 | unsupported | fail | 0.48 | 0.04 | 0.04 | – | 1.00 | 1.00 |
| U03 | unsupported | fail | 0.52 | 0.06 | 0.06 | – | 1.00 | 1.00 |
| F01 | fake-citation | fail | 0.76 | – | – | – | – | – |
| F02 | fake-citation | fail | 0.54 | – | – | – | – | – |
| F03 | fake-citation | fail | 0.21 | – | – | – | – | – |