SubQ Snap is a decision model of the same kind as Jev: give it some state, ask a choice, score or yes/no question, and get an answer with probabilities in well under a second. Its launch note compares it to Jev on size, benchmark score and price. We compared it on the property that matters to us in regulated work: does the same input produce the same answer, every time? We used exactly the test we ran on Jev on 1 October, so the two can be read side by side.
What SubQ says
SubQ's documentation does not address determinism, consistency or reproducibility, and the Decisions API has no temperature, seed or deterministic setting. The only decision model offered is subq-snap-preview, in public preview, with no version-pinned identifier; the documentation notes that model behaviour may change. So, as with Jev, we are not testing a claim the vendor makes. We wanted to know what a user gets today.
Setup
- Data. ContractNLI (Koreeda & Manning, 2021; CC BY 4.0). First 10 contracts of the dev split × 2 hypotheses (nda-2, nda-7) = 20 questions, the same 20 as in the Jev test.
- Request. The full contract as
state; onechoicequestion with three options (Entailment, Contradiction, NotMentioned), each with the same one-line definition as before. 1,037 to 6,171 input tokens per request. - Model.
subq-snap-preview, the only id returned by the model list. All 600 responses reported it. - Calls. For each question we built the request body once and sent the identical bytes 30 times: 15 sequential calls, then 15 concurrent calls. One day (2026-10-13), from a VM in us-central1. No failures, no retries.
Results
| Identical across all 30 calls | SubQ Snap | Jev (1 Oct) |
|---|---|---|
| Chosen label | 20 / 20 | 18 / 20 |
| Probabilities, as returned | 15 / 20 | 6 / 20 |
| Probabilities, sequential calls only | 18 / 20 | 6 / 20 |
| Probabilities, concurrent calls only | 17 / 20 | 6 / 20 |
| Response bytes | 15 / 20 | — |
- No label flipped. Every question received the same label in all 30 calls.
- Probabilities moved on 5 of 20 questions. The spread of a single option's probability across 30 calls had a median of 0.00 and a maximum of 0.07 (Jev: 0.04 and 0.15). SubQ returns probabilities at full floating-point precision, so this comparison is stricter than for Jev, which rounds to two decimals; rounding SubQ's values to two decimals leaves the count at 15 of 20.
- Two answers per question, split by how the calls were sent. On each of the 5 questions that moved, exactly two distinct probability vectors appeared, and the split followed the sending mode. On contract 14 / nda-2, all 15 sequential calls returned one vector and 13 of the 15 concurrent calls returned the other; contracts 14 / nda-7, 15 / nda-2 and 15 / nda-7 show the same pattern, and contract 10 / nda-7 differed on a single sequential call. We measure from outside and cannot say why; the pattern is what one would expect if the result depends on which other requests are processed with it, or on which of two replicas serves it.
- Fast. Median latency was 0.27 s for sequential calls and 0.83 s for 15 concurrent calls.
Why it matters
We did not measure accuracy, so nothing here says SubQ Snap is right or wrong. On this sample it is closer to deterministic than Jev was: no label changed, and three quarters of the questions returned identical bytes 30 times. But the remaining quarter show that the output of an identical request still depends on circumstances the caller does not control. A system that gates on probabilities, for example by routing low-confidence items to a reviewer, can route the same item differently from one run to the next; and a label that holds on 20 questions in one session is not evidence that it holds on the next 20, or next week under a preview identifier that may change. In regulated assessment, a recorded result has to be reproducible exactly, for an auditor, on demand. That is a property of how a model is served, not of how good it is.
Determinism is engineerable, end to end
At GFLOPS we are building AskDona Assessment, a system that is deterministic end to end: not only the language model, but every stage from the source documents to the final assessment. Our preprint on byte-level reproducibility of the whole pipeline, with measurements of thirteen commercial LLM deployments, is in preparation.
Limitations
- A small sample: 20 questions, one model in public preview with no pinned version, one day, one region.
- Only one question per request. Requests with several questions, images or longer contexts were not tested.
GFLOPS, 2026-10-13. Data: ContractNLI (Koreeda & Manning, Findings of EMNLP 2021), CC BY 4.0. Our script and the 600 raw responses are available on request.