Does Jev Play Dice?

Conclusion

Jev cannot roll a fair die. More generally, when a choice has no single correct answer, its output is influenced by the position of the options and by the words themselves, producing systematic bias.

We previously studied GPT-4’s ability to generate random numbers (Does GPT-4 Play Dice? [1]). This time we apply a similar approach to Jev. We ran five experiments with jev-1.13.0. When all six outcomes should be equally likely, Jev does not produce anything close to a uniform choice or probability distribution. Instead, it clearly favors the option listed first. Beyond position, the words themselves matter too: the number 1 and dog receive more probability than the other options.

This is not just about dice. It points to a broader risk: in non-deterministic decisions — situations with no single correct answer, where several options could all be reasonable — Jev’s output may be shaped by option position and wording rather than reflecting only the preference the task is meant to express. Recommendation, ranking, routing, candidate selection, and tie-breaking can all pick up subtle systematic bias as a result.

In this report, a non-deterministic decision means a task with no objectively unique answer, where several options could all be reasonable. “Non-deterministic” describes the task, not how the model produces output: Jev does not sample an answer at random. It assigns each option a probability and picks the one with the highest probability.

What we did

The study has five steps:

1. Number die: Ask Jev for the outcome of a roll among six equally likely faces (Sections 1 and 2).
2. Animal die: Replace the numbers with six animals to see whether the bias persists (Sections 1 and 2).
3. Moving the number 1: Change where the number 1 appears, to separate the effect of position from the effect of the word (Section 3).
4. Guess the animal: Have Jev read 72 different animal descriptions and name the animal. These questions have a single correct answer, making them deterministic decisions. They test whether the bias from option order also affects this kind of judgment (Section 4).
5. Favorite animal: Try all 720 orderings of the six animals to see how subjective preference and option order jointly shape the result (Section 5).

Main findings

1. Jev cannot produce fair dice rolls

   All six faces should be equally likely, yet across 2,000 requests of the same prompt, Jev picked the first-listed number every time.

2. The first-option bias is not limited to numbers

   With the six numbers replaced by animals, Jev still favors the first option: it picks cat in the natural order and switches to fish once the order is shuffled.

3. The words themselves also affect probability

   When the number 1 is placed 4th or 5th, the probability at that position rises noticeably. When the prompt does not say which animals are on the die, dog, listed second, still receives 0.23, whereas the number 2, listed second in the number prompt, receives only 0.01.

4. With a single correct answer, order barely matters

   When Jev guesses the animal from 72 different descriptions, 71 answers stay the same after the options are reordered. Every description with clear or medium clues is answered correctly.

5. Subjective choice reflects both existing preference and position bias

   Across 720 orderings, Jev chose dog 599 times; but whenever cat was listed first, it chose cat all 120 times. Other animals listed first usually could not beat dog.

6. This bias can carry into real applications

   In recommendation, ranking, routing, or candidate selection, options listed earlier and more appealing words may be overrated. The weaker the evidence and the closer the options, the higher the risk.

So Jev’s choices should not be treated as random numbers, nor should we assume they are unaffected by order when there is no single right answer. Fair sampling should happen outside the model. For non-deterministic decisions, rotate the option order, compare multiple orderings, and calibrate for position and wording effects on real data.

Method

The study looks at single rolls only. Each request has three parts: the context says “a fair die was rolled once,” the question asks which face landed up, and the options give a fully symmetric description of each of the six faces.

The model is fixed at jev-1.13.0. The number die uses the options 1–6; the animal die uses cat, dog, bird, fish, horse, cow. The “animal die” is still a fair six-sided die; each of the six numbers is simply replaced by an animal name, one animal per face. This lets us check whether the bias comes from the numbers themselves or from where each option is listed.

Besides an answer, Jev returns a probability for each option, and the probabilities sum to 1. The dice charts show the average of these probabilities over 2,000 identical requests.

How the prompts differ

The question stays the same. We only change whether the context lists the six faces, and in what order.

Number die · Which face landed up?

Animal die · Which animal is the outcome?

To keep the wording consistent, all six options use the same sentence pattern, changing only the number or animal name.

1. Number die and animal die: probability by die face

Sections 1 and 2 use the same data — three prompts each for the number die and the animal die, 2,000 requests per prompt — viewed by die face and by option position respectively.

This chart compares the average probability each number or animal receives. In the natural order, the number 1 and cat receive the highest probability. In the shuffled order [4, 2, 6, 1, 5, 3], the number 4 is listed first and the number 1 is fourth. The highest probability moves to the first option, 4, but the number 1 still keeps a clear second peak.

The animal die uses the matching order, with fish first and cat fourth; here only fish stands out. In addition, in the base prompt the context does not say which animals are on the die, but the options are still listed starting with cat and dog. There, cat gets only 0.44 and dog, listed second, still gets 0.23, possibly because “dog” is a more common word. Under the same conditions for the number die, the first option, 1, gets 0.87 and the second option, 2, gets only 0.01. Red bars mark the highest-probability option in each panel.

Summary: Each face of a fair die should get 1/6, yet whether the faces are numbers or animals, Jev gives most of the probability to a single option, and that option changes with the order: whichever is listed first gets the highest probability. On top of that, the words “1” and “dog” receive some extra probability of their own.

2. Probability by option position

When the x-axis shows where each option is listed, the difference between numbers and animals largely disappears: every curve peaks at position 1, and most other positions fall below 1/6. The number die has a smaller second peak at position 4, because that is where the number 1 sits. In the animal die the same position holds cat, and no similar peak appears there: cat gets only about 0.06. The first-option bias is the strongest effect; the words themselves have a smaller one.

Summary: Viewed by position, the number die and animal die curves have almost the same shape, so what mainly determines the result is where an option is listed, not what is written on the face. The difference at position 4 (a peak for the number 1, none for cat) comes from the word itself, and it is much smaller than the first-option bias.

3. Move the number 1: does the peak follow it?

In the original shuffled order [4, 2, 6, 1, 5, 3], the number 1 is fourth. The new order [5, 3, 2, 6, 1, 4] moves 1 to fifth place and makes 5 the first option. Every answer is now 5; the 4 that used to be first moves to last and is no longer chosen.

The second peak moves with the number 1. Look at position 5: in the original order it holds the number 5, with a probability of only 0.04; in the new order it holds the number 1, and the probability rises to 0.15, about 3.5 times higher. Position 4 changes from the number 1 to the number 6, and its probability drops from 0.22 to 0.16.

Summary: After the number 1 moves, the highest probability still follows the first option, while the second peak follows the number 1. Position bias and word bias come from different sources, and both can appear in the same question.

4. Guess the animal from a description (deterministic decision)

The dice questions have no correct answer. This section switches to questions with a single correct answer, to see whether the position bias still appears. Jev reads an English description and answers Which animal does this sentence describe?, choosing from the same six animals.

We wrote 72 distinct descriptions, 12 per animal, none of which names the animal. They fall into three levels by how many clues they contain — clear, medium, and vague — with 4 sentences per level for each animal. For example:

• Clear (dog): This loyal pet barks at strangers and wags its tail when excited.
• Medium (cat): A small agile pet silently climbs furniture and lands on its feet.
• Vague (cow): A large farm animal grazes slowly in a green pasture. (This could also be a horse.)

Each sentence was tested once with each of two option orders: the natural order starting with cat, and a shuffled order starting with fish — 144 requests in total.

Clear and medium sentences were all answered correctly under both orders, and clear sentences gave the correct animal a probability of 100%. On vague sentences, accuracy dropped to 75% in the natural order and 79% in the shuffled order, with most errors between similar animals such as cat/dog and horse/cow.

Only 1 of the 72 sentences changed its answer after reordering: This soft-coated companion spends much of the afternoon asleep. The correct answer is cat. In the natural order, with cat listed first, Jev answered dog; after shuffling, with cat in fourth place, Jev got it right. It did not switch to the new first option, fish.

Summary: When a question has a single correct answer and the sentence contains enough clues, Jev answers based on content. Option order barely affects the result, and the “always pick the first option” pattern from the dice questions does not appear. With vague clues Jev makes mistakes, but mostly by confusing similar animals, not by favoring the first-listed option. Position bias mainly shows up in non-deterministic decisions, where there is little basis for judgment.

5. Let Jev pick its favorite animal

The question is Which animal do you like best? The context and the answer options list the animals in the same order, and every option uses the same sentence pattern. Each of the 720 orderings of the six animals was requested once, so every animal appears at every position 120 times.

Dog was chosen 599 times. Even when listed 2nd through 6th, it still won about 80% of the time, and its average probability always stayed above 0.5. In the 120 orderings with cat first, Jev chose cat every time; when cat was not first, it never won. Bird, fish, and cow never won even when listed first. Horse won only once, with a probability of 0.37 against dog’s 0.35.

Summary: “Which animal do you like best?” also has no single right answer, but Jev clearly prefers dog, so position bias alone does not decide the result: the first-listed animal wins only if it is also well liked (like cat). In other words, Jev’s subjective choice is determined jointly by its existing preference and a boost from position; simply reordering the same options can change the answer.