Does Jev Play Dice?

Conclusion

We previously studied GPT-4’s ability to generate random numbers (Does GPT-4 Play Dice? [1]). This time we apply a similar approach to Jev. Jev is a decision model from TypeSafe [2]. Instead of generating text, it takes a context and a set of options and returns the selected option along with a probability for each option. Kanta Hayashi has also tested Jev with a fair die and found that its probabilities deviate sharply from 1/6 [4]; this report builds on that and focuses on how option position and the words themselves affect the result. We ran six experiments with jev-1.13.0. When all six outcomes should be equally likely and Jev is asked to pick one, it does not produce anything close to a uniform choice or probability distribution. Instead, it clearly favors the option listed first. Beyond position, the words themselves matter too: the number 1 and dog receive more probability than the other options.

This is not just about dice. It points to a broader risk: in non-deterministic decisions — situations with no single correct answer, where several options could all be reasonable — Jev’s output may be shaped by option position and wording rather than reflecting only the preference the task is meant to express. Recommendation, ranking, routing, candidate selection, and tie-breaking can all pick up subtle systematic bias as a result.

In this report, a non-deterministic decision means a task with no objectively unique answer, where several options could all be reasonable. “Non-deterministic” describes the task, not how the model produces output: Jev does not sample an answer at random. It assigns each option a probability and picks the one with the highest probability.

What we did

The study has six steps:

1. Number die: Ask Jev for the outcome of a roll among six equally likely faces (Sections 1 and 2).
2. Animal die: Replace the numbers with six animals to see whether the bias persists (Sections 1 and 2).
3. Moving the number 1: Change where the number 1 appears, to separate the effect of position from the effect of the word (Section 3).
4. Guess the animal: Have Jev read 72 different animal descriptions and name the animal. These questions have a single correct answer, making them deterministic decisions. They test whether the bias from option order also affects this kind of judgment (Section 4).
5. Favorite animal: Try all 720 orderings of the six animals to see how subjective preference and option order jointly shape the result (Section 5).
6. Noul calibration: The first five steps all have Jev pick one option from a set (Choice). This step instead asks a separate yes/no question for each face (Noul), to test whether Jev’s probability estimates are accurate for fair dice with 2 to 20 faces (Section 6).

Main findings

1. Jev cannot produce fair dice rolls

   All six faces should be equally likely, yet across 2,000 requests of the same prompt, Jev picked the first-listed number every time.

2. The first-option bias is not limited to numbers

   With the six numbers replaced by animals, Jev still favors the first option: it picks cat in the natural order and switches to fish once the order is shuffled.

3. The words themselves also affect probability

   When the number 1 is placed 4th or 5th, the probability at that position rises noticeably. When the prompt does not say which animals are on the die, dog, listed second, still receives 0.23, whereas the number 2, listed second in the number prompt, receives only 0.01.

4. With a single correct answer, order barely matters

   When Jev guesses the animal from 72 different descriptions, 71 answers stay the same after the options are reordered. Every description with clear or medium clues is answered correctly.

5. Subjective choice reflects both existing preference and position bias

   Across 720 orderings, Jev chose dog 599 times; but whenever cat was listed first, it chose cat all 120 times. Other animals listed first usually could not beat dog.

6. With separate yes/no judgments, probability no longer piles onto one option, but it is still inaccurate

   When Noul judges each face of a fair die separately, the faces of the same die get similar probabilities. But each face of a 2-sided die gets only 0.43 (should be 0.5), and each face of a 20-sided die gets 0.095 (should be 0.05): large probabilities are underestimated and small ones overestimated.

7. This bias can carry into real applications

   In recommendation, ranking, routing, or candidate selection, options listed earlier and more appealing words may be overrated, especially when there is little evidence to decide by and several options are plausible.

So Jev’s choices should not be treated as random numbers, nor should we assume they are unaffected by order when there is no single right answer. Fair sampling should happen outside the model. For non-deterministic decisions, rotate the option order, compare multiple orderings, and calibrate for position and wording effects on real data.

Method

The study looks at single rolls only. Each request has three parts: the context says “a fair die was rolled once,” the question asks which face landed up, and the options give a fully symmetric description of each of the six faces.

The model is fixed at jev-1.13.0. The number die uses the options 1–6; the animal die uses cat, dog, bird, fish, horse, cow. The “animal die” is still a fair six-sided die; each of the six numbers is simply replaced by an animal name, one animal per face. This lets us check whether the bias comes from the numbers themselves or from where each option is listed.

The first five experiments use Choice questions. Besides an answer, Jev returns a probability for each option, and the probabilities sum to 1. The dice charts show the average of these probabilities over 2,000 identical requests.

How the prompts differ

The question stays the same. We only change whether the context lists the six faces, and in what order.

Number die · Which face landed up?

Animal die · Which animal is the outcome?

To keep the wording consistent, all six options use the same sentence pattern, changing only the number or animal name.

1. Number die and animal die: probability by die face

Sections 1 and 2 use the same data — three prompts each for the number die and the animal die, 2,000 requests per prompt — viewed by die face and by option position respectively.

This chart compares the average probability each number or animal receives. In the natural order, the number 1 and cat receive the highest probability. In the shuffled order [4, 2, 6, 1, 5, 3], the number 4 is listed first and the number 1 is fourth. The highest probability moves to the first option, 4, but the number 1 still keeps a clear second peak.

The animal die uses the matching order, with fish first and cat fourth; here only fish stands out. In addition, in the base prompt the context does not say which animals are on the die, but the options are still listed starting with cat and dog. There, cat gets only 0.44 and dog, listed second, still gets 0.23, possibly because “dog” is a more common word. Under the same conditions for the number die, the first option, 1, gets 0.87 and the second option, 2, gets only 0.01. Red bars mark the highest-probability option in each panel.

Summary: Each face of a fair die should get 1/6, yet whether the faces are numbers or animals, Jev gives most of the probability to a single option, and that option changes with the order: whichever is listed first gets the highest probability. On top of that, the words “1” and “dog” receive some extra probability of their own.

2. Probability by option position

When the x-axis shows where each option is listed, the difference between numbers and animals largely disappears: every curve peaks at position 1, and most other positions fall below 1/6. The number die has a smaller second peak at position 4, because that is where the number 1 sits. In the animal die the same position holds cat, and no similar peak appears there: cat gets only about 0.06. The first-option bias is the strongest effect; the words themselves have a smaller one.

Summary: Viewed by position, the number die and animal die curves have almost the same shape, so what mainly determines the result is where an option is listed, not what is written on the face. The difference at position 4 (a peak for the number 1, none for cat) comes from the word itself, and it is much smaller than the first-option bias.

3. Move the number 1: does the peak follow it?

In the original shuffled order [4, 2, 6, 1, 5, 3], the number 1 is fourth. The new order [5, 3, 2, 6, 1, 4] moves 1 to fifth place and makes 5 the first option. Every answer is now 5; the 4 that used to be first moves to last and is no longer chosen.

The second peak moves with the number 1. Look at position 5: in the original order it holds the number 5, with a probability of only 0.04; in the new order it holds the number 1, and the probability rises to 0.15, about 3.5 times higher. Position 4 changes from the number 1 to the number 6, and its probability drops from 0.22 to 0.16.

Summary: After the number 1 moves, the highest probability still follows the first option, while the second peak follows the number 1. Position bias and word bias come from different sources, and both can appear in the same question.

4. Guess the animal from a description (deterministic decision)

The dice questions have no correct answer. This section switches to questions with a single correct answer, to see whether the position bias still appears. Jev reads an English description and answers Which animal does this sentence describe?, choosing from the same six animals.

We wrote 72 distinct descriptions, 12 per animal, none of which names the animal. They fall into three levels by how many clues they contain — clear, medium, and vague — with 4 sentences per level for each animal. For example:

• Clear (dog): This loyal pet barks at strangers and wags its tail when excited.
• Medium (cat): A small agile pet silently climbs furniture and lands on its feet.
• Vague (cow): A large farm animal grazes slowly in a green pasture. (This could also be a horse.)

Each sentence was tested once with each of two option orders: the natural order starting with cat, and a shuffled order starting with fish — 144 requests in total.

Clear and medium sentences were all answered correctly under both orders, and clear sentences gave the correct animal a probability of 100%. On vague sentences, accuracy dropped to 75% in the natural order and 79% in the shuffled order, with most errors between similar animals such as cat/dog and horse/cow.

Only 1 of the 72 sentences changed its answer after reordering: This soft-coated companion spends much of the afternoon asleep. The correct answer is cat. In the natural order, with cat listed first, Jev answered dog; after shuffling, with cat in fourth place, Jev got it right. It did not switch to the new first option, fish.

Summary: When a question has a single correct answer and the sentence contains enough clues, Jev answers based on content. Option order barely affects the result, and the “always pick the first option” pattern from the dice questions does not appear. With vague clues Jev makes mistakes, but mostly by confusing similar animals, not by favoring the first-listed option. Position bias mainly shows up in non-deterministic decisions, where there is little basis for judgment.

5. Let Jev pick its favorite animal

The question is Which animal do you like best? The context and the answer options list the animals in the same order, and every option uses the same sentence pattern. Each of the 720 orderings of the six animals was requested once, so every animal appears at every position 120 times.

Dog was chosen 599 times. Even when listed 2nd through 6th, it still won about 80% of the time, and its average probability always stayed above 0.5. In the 120 orderings with cat first, Jev chose cat every time; when cat was not first, it never won. Bird, fish, and cow never won even when listed first. Horse won only once, with a probability of 0.37 against dog’s 0.35.

Summary: “Which animal do you like best?” also has no single right answer, but Jev clearly prefers dog, so position bias alone does not decide the result: the first-listed animal wins only if it is also well liked (like cat). In other words, Jev’s subjective choice is determined jointly by its existing preference and a boost from position; simply reordering the same options can change the answer.

6. Does Jev flip coins? Judging each face with Noul

TypeSafe offers several question types, set by the type field of the request [3]. The first five sections all use Choice: every option goes into a single question, Jev picks one, and it gives each option a probability, with all probabilities summing to 1. Because the options share one budget of probability, whatever one option gains the others must lose; the position and wording effects seen earlier all arise in this comparison.

Noul is a yes/no question: each question judges whether a single condition holds, and Jev returns only the probability of “yes,” without choosing anything. One request can contain several Noul questions. Each is judged independently and cannot see the others’ answers, so their probabilities need not sum to 1. To ask about a die with Noul, each face gets its own question, such as Did the die land on 3?, so the options no longer compete with one another. If Choice is like rolling a die once to pick one of the options, Noul is more like flipping a separate coin for each face, with only two outcomes: “yes” or “no.” Jev never actually flips the coin, though; it only reports the probability of “yes,” that is, how far the coin is weighted toward “yes.”

This section uses Noul to test whether Jev estimates the probability of equally likely events accurately. The context reads “a fair N-sided die with faces numbered 1 to N is rolled once,” with N = 2, 3, 4, 6, 8, 10, 12, and 20. Each face is asked separately, and each die is requested 100 times. For a fair die, every face should have probability 1/N, and all faces together should sum to 1.

On any one die, the faces get similar probabilities. On the 6-sided die, for example, every face falls between 0.17 and 0.19; nothing like Choice’s habit of giving most of the probability to one option appears. But the size of the probability is off. The 6- and 8-sided dice come closest to the true value. With few faces, each face is underestimated: the true value for a 2-sided die is 0.5, and Jev gives 0.43. With many faces, each face is overestimated: the true value for a 20-sided die is 0.05, and Jev gives 0.095, nearly double, so all faces together sum to 1.90.

Summary: With separate Noul judgments, probability no longer piles onto one option, but the estimates are pulled toward the middle: true probabilities of 0.25 and above are underestimated by 13%–17%, and those of 0.1 and below are overestimated by 26%–90%, more so the smaller the probability. This shape resembles human probability judgment: prospect theory finds that people also overweight small probabilities and underweight moderate to large ones [5]. This is an analogy only; it does not explain why Jev behaves this way. So Noul outputs cannot be used directly as true probabilities either; they need to be calibrated on real data first. Noul also does not “roll” a result for us, so fair sampling still has to happen outside the model.

Appendix

References

[1] Qiang Liu. Does GPT-4 Play Dice? https://qiangliu.net/publications/Does_GPT_Play_Dice.pdf
[2] TypeSafe. Introduction to Jev. TypeSafe documentation. https://docs.typesafe.ai/introduction
[3] TypeSafe. Choice; Noul. TypeSafe documentation. https://docs.typesafe.ai/primitives/choice; https://docs.typesafe.ai/primitives/noul
[4] Kanta Hayashi. Jev Does Not Play Dice. September 20, 2026. https://kantahayashiai.github.io/posts/jev-does-not-play-dice/
[5] A. Tversky, D. Kahneman. Advances in Prospect Theory: Cumulative Representation of Uncertainty. Journal of Risk and Uncertainty, 5(4): 297–323, 1992. https://doi.org/10.1007/BF00122574
[6] E. T. Jaynes. Information Theory and Statistical Mechanics. Physical Review, 106(4): 620–630, 1957. https://doi.org/10.1103/PhysRev.106.620

The following sources relate to the experiments in this report, grouped by topic, for further reading.

Tests and discussion of Jev

• TypeSafe. Jev 1.13 jaggedness. TypeSafe documentation. The official list of known weaknesses. It notes that values from the same question asked as a Noul and as a Choice are not directly comparable, and that a question and its negation asked as two Nouls can sum to 1.19; users should not expect arithmetic identities to hold across separate questions.
• Alex Molas. Jev can't be calibrated. September 23, 2026. Argues that calibration is a joint property of the model and the data distribution, so Jev cannot be calibrated on every user’s data; recommends treating its outputs as ranking scores and recalibrating on a few hundred labeled examples of your own.
• KITE: Scaling Jev Population Experiments with Sparse Flagship Calibration. arXiv:2609.27535. Using jev-1.13.0, asking one Noul per option and renormalizing gave a calibration error of 0.027, against 0.23–0.25 for Choice.
• Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures. arXiv:2609.29429. Tests Noul on 31 benchmarks: close to calibrated when pooled, but the median per-benchmark calibration error is 0.168; ranking is good (median AUROC 0.905), and the error comes mainly from an overall shift in probability.
• Leonard Grazian. Is Jev Calibrated? A contrasting view: after tuning the prompt for each dataset, Jev’s calibration error was about 0.03 on two real datasets.

Related research on large language models

• C. Zheng et al. Large Language Models Are Not Robust Multiple Choice Selectors. ICLR 2024. Finds “selection bias” in multiple-choice questions across 20 models, arising from a preference for particular option labels and for particular positions; proposes estimating the bias by permuting options and then removing it (PriDe).
• P. Pezeshkpour, E. Hruschka. Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions. 2023. Reordering options alone changes accuracy by 13%–75%; models are most affected by position when torn between the top two or three options.

How to cite

Qiang Liu. Does Jev Play Dice? Systematic Bias in Non-deterministic Decisions. Technical report, September 2026. https://qiangliu.net/jev-dice/

Code and data: https://github.com/pobooo/jev-dice

@techreport{liu2026jevdice,
  author      = {Qiang Liu},
  title       = {Does {Jev} Play Dice? Systematic Bias in Non-deterministic Decisions},
  institution = {qiangliu.net},
  type        = {Technical report},
  year        = {2026},
  month       = sep,
  url         = {https://qiangliu.net/jev-dice/},
  note        = {Code and data: \url{https://github.com/pobooo/jev-dice}}
}