{"notes":[{"id":"model-evaluation/jev-evaluation","category":"model-evaluation","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Jev Evaluation","subtitle":"Ninety Percent Sure, Are We?","author":"Z. Yuan","paperDate":"2026-09-29","publishedOn":"2026-11-05","paperTitle":"Evaluating and Benchmarking the System One Model Jev","paperUrl":"https://arxiv.org/abs/2609.37647","bibliography":{"title":"Evaluating and Benchmarking the System One Model Jev","authors":["Tobias Deußer","Lorenz Sparrenberg","Rafet Sifa"],"url":"https://arxiv.org/abs/2609.37647","pdfUrl":"https://arxiv.org/pdf/2609.37647v1","codeUrl":"https://github.com/AppliedMachineLearning-Lab/jev-benchmarking","preprint":{"id":"2609.37647","submittedOn":"2026-09-29","url":"https://arxiv.org/abs/2609.37647"},"publication":null,"citation":"@misc{modelevaluationjevevaluation2026,\n  title = {{Evaluating and Benchmarking the System One Model Jev}},\n  author = {Tobias Deußer and Lorenz Sparrenberg and Rafet Sifa},\n  year = {2026},\n  eprint = {2609.37647},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2609.37647}\n}\n","citationPath":"/content/papers/citations/model-evaluation/jev-evaluation.e441e60850f7.bib"},"thumbnail":{"src":"/content/papers/assets/c38715617566fd975974709b55c9984cf286a1ff5745dba0cb0f63bbddeaf348.png","width":1308,"height":592},"locale":"en","path":"/content/papers/en/model-evaluation/jev-evaluation.dfc50d3b2429.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"tackling-the-problem","text":"Tackling the Problem","depth":2},{"id":"three-basic-operations","text":"Three Basic Operations","depth":3},{"id":"making-the-comparison-repeatable","text":"Making the Comparison Repeatable","depth":3},{"id":"how-do-the-language-models-join-the-comparison","text":"How Do the Language Models Join the Comparison?","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"classification-results","text":"Classification Results","depth":3},{"id":"are-the-probabilities-reliable","text":"Are the Probabilities Reliable?","depth":3},{"id":"ranking-thresholds-and-f1","text":"Ranking, Thresholds, and F1","depth":3},{"id":"what-do-we-gain-by-answering-only-half","text":"What Do We Gain by Answering Only Half?","depth":3},{"id":"the-gap-in-score-ratings","text":"The Gap in Score Ratings","depth":3},{"id":"mmlu-where-do-the-high-scores-come-from","text":"MMLU: Where Do the High Scores Come From?","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"transformers/bert","relation":"background","topic":"Task-specific classification and decisions without retraining"},{"id":"transformers/gpt_3","relation":"comparison","topic":"Adapting to a task through prompts or structured decisions"}]},{"id":"transformers/bert","category":"transformers","legacyPath":"transformers/1810-bert","legacySlug":"1810-bert","legacyPaths":[],"title":"BERT","subtitle":"Twelve Layers of Encoders","author":"Z. Yuan","paperDate":"2018-10-11","publishedOn":"2024-05-12","paperTitle":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding","paperUrl":"https://arxiv.org/abs/1810.04805","bibliography":{"title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding","authors":["Jacob Devlin","Ming-Wei Chang","Kenton Lee","Kristina Toutanova"],"url":"https://arxiv.org/abs/1810.04805","pdfUrl":"https://arxiv.org/pdf/1810.04805","codeUrl":null,"preprint":{"id":"1810.04805","submittedOn":"2018-10-11","url":"https://arxiv.org/abs/1810.04805"},"publication":{"venue":"NAACL-HLT","year":2019,"kind":"conference","url":"https://doi.org/10.18653/v1/n19-1423"},"citation":"@misc{transformersbert2018,\n  title = {{BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding}},\n  author = {Jacob Devlin and Ming-Wei Chang and Kenton Lee and Kristina Toutanova},\n  year = {2018},\n  eprint = {1810.04805},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1810.04805}\n}\n","citationPath":"/content/papers/citations/transformers/bert.d9bf47a21c5f.bib"},"thumbnail":{"src":"/content/papers/assets/57ed95afec24d2732e290d4fcb5633e14a2bbe74b83c6b6cdee90ef44ac89411.jpg","width":1224,"height":396},"locale":"en","path":"/content/papers/en/transformers/bert.75ec0229da6b.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"unidirectionality","text":"Unidirectionality","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"pre-training-strategy","text":"Pre-training Strategy","depth":3},{"id":"fine-tuning-strategy","text":"Fine-tuning Strategy","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"ablation-study","text":"Ablation Study","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"transformers/transformer","relation":"background","topic":"Pretraining bidirectional language representations"},{"id":"transformers/roberta","relation":"next","topic":"Pretraining bidirectional language representations"},{"id":"transformers/albert","relation":"comparison","topic":"Training and compressing BERT"}]},{"id":"transformers/gpt_3","category":"transformers","legacyPath":"transformers/2005-gpt_3","legacySlug":"2005-gpt_3","legacyPaths":[],"title":"GPT-3","subtitle":"Ninety-Six Layer Decoder","author":"Z. Yuan","paperDate":"2020-05-28","publishedOn":"2024-06-04","paperTitle":"Language Models are Few-Shot Learners","paperUrl":"https://arxiv.org/abs/2005.14165","bibliography":{"title":"Language Models are Few-Shot Learners","authors":["Tom B. Brown","Benjamin Mann","Nick Ryder","Melanie Subbiah","Jared Kaplan","Prafulla Dhariwal","Arvind Neelakantan","Pranav Shyam","Girish Sastry","Amanda Askell","Sandhini Agarwal","Ariel Herbert-Voss","Gretchen Krueger","Tom Henighan","Rewon Child","Aditya Ramesh","Daniel M. Ziegler","Jeffrey Wu","Clemens Winter","Christopher Hesse","Mark Chen","Eric Sigler","Mateusz Litwin","Scott Gray","Benjamin Chess","Jack Clark","Christopher Berner","Sam McCandlish","Alec Radford","Ilya Sutskever","Dario Amodei"],"url":"https://arxiv.org/abs/2005.14165","pdfUrl":"https://arxiv.org/pdf/2005.14165","codeUrl":null,"preprint":{"id":"2005.14165","submittedOn":"2020-05-28","url":"https://arxiv.org/abs/2005.14165"},"publication":{"venue":"NeurIPS","year":2020,"kind":"conference","url":"https://proceedings.neurips.cc/paper_files/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html"},"citation":"@misc{transformersgpt32020,\n  title = {{Language Models are Few-Shot Learners}},\n  author = {Tom B. Brown and Benjamin Mann and Nick Ryder and Melanie Subbiah and Jared Kaplan and Prafulla Dhariwal and Arvind Neelakantan and Pranav Shyam and Girish Sastry and Amanda Askell and Sandhini Agarwal and Ariel Herbert-Voss and Gretchen Krueger and Tom Henighan and Rewon Child and Aditya Ramesh and Daniel M. Ziegler and Jeffrey Wu and Clemens Winter and Christopher Hesse and Mark Chen and Eric Sigler and Mateusz Litwin and Scott Gray and Benjamin Chess and Jack Clark and Christopher Berner and Sam McCandlish and Alec Radford and Ilya Sutskever and Dario Amodei},\n  year = {2020},\n  eprint = {2005.14165},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2005.14165}\n}\n","citationPath":"/content/papers/citations/transformers/gpt_3.e117d4f773ec.bib"},"thumbnail":{"src":"/content/papers/assets/7cf129fba1be78b1b43198edc35c44063ba70cb8fc082136055a265ee022b092.jpg","width":1204,"height":1080},"locale":"en","path":"/content/papers/en/transformers/gpt_3.091cea326188.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"diverse-language-task-requirements","text":"Diverse Language Task Requirements","depth":3},{"id":"dependence-on-large-supervised-datasets","text":"Dependence on Large Supervised Datasets","depth":3},{"id":"over-specialization-and-generalization-issues","text":"Over-Specialization and Generalization Issues","depth":3},{"id":"human-learning-efficiency-vs-model-efficiency","text":"Human Learning Efficiency vs. Model Efficiency","depth":3},{"id":"solution","text":"Solution","depth":2},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"dataset-setup","text":"Dataset Setup","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"lambada","text":"LAMBADA","depth":3},{"id":"closed-book-question-answering","text":"Closed Book Question Answering","depth":3},{"id":"conclusion","text":"Conclusion","depth":2},{"id":"1-gender-bias","text":"1. Gender Bias","depth":3},{"id":"2-racial-bias","text":"2. Racial Bias","depth":3},{"id":"3-religious-bias","text":"3. Religious Bias","depth":3}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"transformers/gpt_2","relation":"background","topic":"From language pretraining to in-context learning"},{"id":"transformers/scaling_laws","relation":"next","topic":"Trading model size against training data"},{"id":"model-evaluation/jev-evaluation","relation":"comparison","topic":"Adapting to a task through prompts or structured decisions"}]}],"text":"Large language models (LLMs) are good at answering questions. Getting those answers into a program often takes a bit more work.\n\nSuppose we just want to route customer emails to billing, technical support, or another department. The model reads a complaint about a duplicate charge and gives a perfectly coherent explanation. All the program needs, though, is `billing`. Generating the extra explanation takes time, and the returned text still has to be turned into a value the program can use.\n\nThe formatting problem already has solutions. Tools for serving language models, such as vLLM, support [**structured outputs**](https://docs.vllm.ai/en/latest/features/structured_outputs/) that constrain an answer to specified choices or a predefined JSON format.\n\nThe harder next step is to decide **whether we can act on this classification automatically**.\n\nIf we ask the model to add “I'm ninety percent sure,” we will usually just get a very confident wrong answer. That does not help much. Even if we read the probabilities assigned to each option directly, we still need to test whether those numbers are reliable.\n\nAnd if a person has to check every case, the automation starts to feel a lot less automatic.\n\n**Jev**, from TypeSafe AI, aims to handle exactly these decisions with a clearly defined scope. It is an AI decision model that takes data and a question, along with a choice of classification, scoring, or a yes/no judgment, and returns results and probabilities that a program can use directly. The version tested in this paper gives up free-form text generation. The company emphasizes speed, low cost, and confidence information that can help decide when to hand a case over to a person.\n\nWe originally wanted to see how the model works. But while the [**official launch post**](https://typesafe.ai/blog/introducing-system-one-models-and-jev) mentions a new architecture and new training methods, no paper came with it. As of October 17, 2026, we still have not found an official model paper explaining the architecture and training.\n\nThat part will have to wait.\n\n\n> [!TIP]\n> \n> **Who is TypeSafe AI?**\n> \n> According to its [**launch post**](https://typesafe.ai/blog/introducing-system-one-models-and-jev), the team spent two years developing in private before releasing Jev, its first public model.\n> \n> Founder Diogo Almeida worked on InstructGPT at OpenAI and is one of the [**paper's authors**](https://arxiv.org/abs/2203.02155). InstructGPT used human demonstrations and preferences over responses to train language models to follow instructions. Jev focuses instead on outputs such as classifications and scores that can feed directly into programs.\n\n\n## Defining the Problem\n\nTypeSafe AI calls models like Jev “System One” models, emphasizing fast decisions within a clearly defined scope. The `jev-1.13.0` version evaluated here generates neither free-form text nor a step-by-step reasoning trace.\n\n\n> [!TIP]\n> \n> **What does System One mean?**\n> \n> The name borrows the distinction in Daniel Kahneman's *Thinking, Fast and Slow*, where System 1 responds quickly and intuitively, while System 2 slows down and works through a problem step by step. The [**official FAQ**](https://typesafe.ai/blog/introducing-system-one-models-and-jev) explains this origin too.\n\n\nThis positioning has also drawn criticism. In a [**LocalLLaMA thread on Reddit**](https://www.reddit.com/r/LocalLLaMA/comments/1woe70t/jev_isnt_new_tech_its_marketing_targets_people/), people asked why, if classifiers already return options and probabilities, it should get a new name and be compared only with LLMs. Researcher [**Sebastian Raschka's analysis**](https://magazine.sebastianraschka.com/p/classifier-history-and-jev) notes that it is convenient to use, since switching to a new set of classification requirements spares you from first preparing training data and fine-tuning the model for the new task.\n\nThat convenience is appealing. But if we already have labeled data, how does Jev compare with a dedicated classifier?\n\nAnother claim, “0% hallucination,” is even easier to misread. A hallucination usually means that a model produces unsupported content. “Zero hallucinations” can easily sound like a promise that Jev never answers incorrectly. Looking at [**TypeSafe's launch explanation**](https://typesafe.ai/blog/introducing-system-one-models-and-jev), however, the claim is actually about **schema matching**, meaning compliance with an agreed output format. The 0% is not a measured error rate.\n\nTake that email again. Jev guarantees a choice from `billing`, `technical`, and `other`, but it could still choose `technical`. The program runs just fine; the refund request simply ends up with technical support. That guarantee is useful for API integration, but classification accuracy still needs a separate test.\n\nTo understand the system's capabilities more fully, the authors examine the model's confidence from several angles. Are answers the model considers ninety percent certain actually correct about ninety percent of the time? If low-confidence answers are more likely to be wrong, how much more reliable do the remaining answers become once we exclude them?\n\n## Tackling the Problem\n\n### Three Basic Operations\n\nJev calls the data to be judged the **state**.\n\nThis can be text, a JSON object, or an array—for example, the email content, order status, and customer information bundled together.\n\nAttach a question to that data, and the interface offers three **primitives**, or basic operations. Using the same email, here is the difference:\n\n| Operation | Example question | Returned values |\n| --- | --- | --- |\n| **Choice** | Should this email go to billing, technical support, or another department? | One option, probabilities for every option, and a confidence score |\n| **Noul** | Does this email need priority handling? | The probability of “yes,” $P(\\text{yes})$ |\n| **Score** | How urgent is this email on a predefined five-level scale? | Probabilities for each level, a probability-weighted score, and a confidence score |\n\nChoice supports up to 255 options. Score uses 2 to 10 ordered levels, each with a description. If an option's name is not clear enough, a description can help define the boundary between categories.\n\nThe name Noul does not tell us much. For now, just remember: ask a yes/no question and get back the probability of “yes.” The confidence score in the table is a separate, transformed value. We will unpack its formula when we use it to filter answers later.\n\nThe score returned by Score needs a closer look.\n\nSuppose we encode urgency as $0,1,2,3,4$, from not urgent to extremely urgent. The model assigns the following probabilities:\n\n$$\np=[0.05,\\ 0.05,\\ 0.20,\\ 0.45,\\ 0.25]\n$$\n\nLevel 3 has the highest probability, but the weighted score is:\n\n$$\ns=\\sum_{i=0}^{4}i\\,p_i\n=0\\times0.05+1\\times0.05+2\\times0.20+3\\times0.45+4\\times0.25\n=2.8\n$$\n\nSo **the most likely level and the returned continuous score need not be the same**. The former suits tasks that require a single label. The latter preserves uncertainty between adjacent levels and is more convenient for ranking.\n\nThese numbers are a worked example, not an actual Jev API call. The calculation treats adjacent levels as equally spaced, so moving from 0 to 1 adds one point, just like moving from 3 to 4. If your rubric defines only an order, without specifying what a one-level difference means, keep that in mind when interpreting the weighted score.\n\nYou can attach several questions to the same state, but each is answered independently. If a message can express both “joy” and “surprise,” two Noul questions are a good fit. Their two “yes” probabilities do not have to sum to 1. If exactly one of three departments must be chosen, use Choice instead.\n\n### Making the Comparison Repeatable\n\nThe authors organize 37 public datasets into seven groups: text classification; intent and topic routing; natural language inference and evidence support; reading comprehension and knowledge; commonsense reasoning; safety and legal judgments; and rubric-based scoring.\n\nFor each dataset, they first write a question template and try it on 20 training or validation examples. Once the wording works, they freeze the template. They do not keep adjusting the prompt against the answers during the main evaluation.\n\n\n> [!TIP]\n> \n> **Does zero-shot mean taking a test without ever having learned anything?**\n> \n> Here, it means that the evaluation provides only task instructions and options, with no demonstrations of how to answer that kind of question and no retraining of Jev for these tasks. This is a zero-shot setting. Adding a few examples of inputs and correct answers to the prompt would make it few-shot.\n> \n> Of course, Jev has already been trained. Zero-shot describes how this evaluation presents the questions.\n\n\nMost datasets use the full designated test split. When test answers are not public, the validation split is used instead.\n\nBIG-bench is a collection of language-model evaluation tasks. This paper uses selected multiple-choice tasks from it, excluding arithmetic, programming, and similar tasks. That leaves 93 subtasks, with at most 500 examples from each.\n\nEach example becomes one request. Multiple label questions are bundled into that same request. Identical requests are sent only once, and their responses are saved for reuse. Recomputing the results with a different metric later does not require paying for the API again.\n\nThe main evaluation obtained responses for 346,009 unique requests. One additional BIG-bench example exceeded the model's input limit and was rejected. Both the [**evaluation code**](https://github.com/AppliedMachineLearning-Lab/jev-benchmarking) and the [**responses from all three models**](https://doi.org/10.5281/zenodo.23039006) are public, providing a starting point for recalculating metrics or adding comparison models.\n\n### How Do the Language Models Join the Comparison?\n\nThe reference models are Qwen3.8-27B and Gemma-4-E4B. Both have open weights, so their parameters are available for running them yourself.\n\nThey receive the same data, questions, and options. The authors still need to turn their outputs into comparable probabilities, though.\n\nLanguage models process text in units called **tokens**. A token might be a character, part of a word, or punctuation; the tool that splits text into these units is a **tokenizer**. The authors encode the answer options using codes that each model can represent with a single token. One model pass can then provide the next-token probability for each answer.\n\nBut the model's vocabulary contains many other tokens that are not valid answers.\n\nThe authors retain only the valid options and rescale their probabilities to sum to 1.\n\nFor example, suppose the valid answers are A, B, and C, with original probabilities $0.60,0.20,0.10$. All other tokens together account for $0.10$.\n\nKeeping only the options gives:\n\n$$\n\\tilde p_A=\\frac{p_A}{p_A+p_B+p_C}=\\frac{0.60}{0.60+0.20+0.10}\\approx0.667\n$$\n\nThis gives a probability for each option, just as Jev does, allowing the same metrics to be computed. The authors also check how much probability mass is discarded. For both reference models, 99% of questions place at least 0.93 probability on the valid codes.\n\nIn this setup, both models **turn thinking off and answer directly**. Thinking here means generating a reasoning trace before giving the answer. The authors do not let either model work through the problem in a long text response.\n\n## Discussion\n\n### Classification Results\n\nLet's start with a few datasets where all models are evaluated using accuracy.\n\n| Dataset | What is being judged? | Jev | Qwen3.8-27B | Gemma-4-E4B |\n| --- | --- | ---: | ---: | ---: |\n| IMDB | Positive or negative sentiment in movie reviews | 96.5% | 96.5% | 95.6% |\n| Banking77 | 77 banking-service intents | 79.7% | 77.4% | 67.0% |\n| CLINC150 | Intent classification, including an out-of-scope option | 89.5% | 88.8% | 68.3% |\n| Belebele | Reading comprehension across 122 language varieties | 86.7% | 77.8% | 68.4% |\n| HellaSwag | Choosing a plausible continuation of a situation | 95.5% | 94.9% | 82.9% |\n| Emotion | Six-way emotion classification | 58.5% | 57.1% | 56.9% |\n\nThese results are excerpted from [**Table 3**](https://arxiv.org/html/2609.37647v1#S4.T3) of the paper. Each row uses the evaluation split specified there, and all numbers come from the same evaluation pipeline.\n\nJev leads on most of these tasks, but performance varies considerably between tasks. Positive/negative movie-review classification reaches 96.5%. The finer categories in banking-service intent classification bring that down to 79.7%. Emotion, whose hashtag-derived labels contain noise, reaches only 58.5%.\n\nThe Belebele average also hides a large gap. The score averages 86.7% across 122 language varieties and reaches 97.2% for English, but just 37.4% for Nigerian Fulfulde. If the latter is the language you need, the overall average is not much help.\n\nAcross all 37 datasets, using each dataset's primary metric, Jev scores higher than Qwen on 27, lower on 9, and the same on 1.\n\n\n> [!TIP]\n> \n> **How does bootstrap estimation produce an interval?**\n> \n> The authors sample the evaluated questions “with replacement,” putting each sampled question back, so the same question can appear more than once. They compute a score for the resampled set, repeat this 500 times, and use the resulting variation to estimate a 95% confidence interval.\n> \n> This examines variation due to the sample of questions, using 500 resamples of existing responses. Non-overlapping intervals are evidence of a clearer gap. Overlapping intervals leave the difference between models uncertain.\n\n\nWhen two scores are close, a different set of questions could reverse their ranking. This is the variability the authors estimate with that procedure.\n\nOf the 27 datasets where Jev leads, 11 have intervals that do not overlap with Qwen's. All nine datasets where Qwen leads have overlapping intervals. Simply tallying 27 wins skips over quite a few results whose differences remain uncertain.\n\nGemma scores below Jev on all 37 datasets, with non-overlapping intervals on 31. This comparison uses the same questions and a direct-answer setting. Jev's parameter count and training data are unknown, so we cannot attribute the gap to a particular architectural design.\n\n### Are the Probabilities Reliable?\n\nTo check a claim of ninety percent certainty, one right or wrong answer is not enough. We need many answers with similar predicted probabilities and must see how often they are actually correct.\n\nSuppose 100 answers each have a highest-option probability of about 0.9, but only 70 are correct. The average prediction and the observed accuracy differ by about 0.2. The model is overconfident on this group of answers.\n\nThe paper summarizes this with **Expected Calibration Error (ECE)**. It divides $[0,1]$ into 15 equal-width bins and groups answers with similar confidence. For each bin, it takes the absolute difference between the average predicted probability and the observed accuracy, then weights that difference by the number of answers.\n\n$$\n\\operatorname{ECE}\n=\\sum_{b=1}^{15}\\frac{|B_b|}{N}\n\\left|\\operatorname{acc}(B_b)-\\operatorname{prob}(B_b)\\right|\n$$\n\nHere, $B_b$ is the set of answers in bin $b$, and $N$ is the total number of answers. For Noul, the comparison on the right is between mean $P(\\text{yes})$ and the observed proportion of “yes” answers. Examples whose true answer is “yes” are **positives**; those whose true answer is “no” are **negatives**.\n\nPlotting this gives the **reliability diagram** below. The horizontal axis shows the predicted probability; the vertical axis shows observed accuracy or the observed frequency of “yes.” Ideally, a group assigned 0.7 lands at 0.7, and a group assigned 0.9 lands at 0.9, forming the gray diagonal. Larger circles represent more answers.\n\n[![Probability calibration for Choice, single-question Noul, and multilabel Noul, comparing Jev, Qwen, and Gemma](/content/papers/assets/c38715617566fd975974709b55c9984cf286a1ff5745dba0cb0f63bbddeaf348.png)](/content/papers/assets/c38715617566fd975974709b55c9984cf286a1ff5745dba0cb0f63bbddeaf348.png)\n\n*Figure 2 from the paper, taken from [**page 9 of the v1 PDF**](https://arxiv.org/pdf/2609.37647v1#page=9). Left to right: Choice, single-question Noul, and multilabel Noul. Click to enlarge.*\n\nIn the left plot, Jev's blue line roughly follows the diagonal. Pooling 279,925 answers across 22 Choice datasets gives an ECE of 0.028, below Qwen's 0.063 and Gemma's 0.208.\n\nThis supports using Choice probabilities for decisions, but large datasets influence the pooled result. Belebele, SIB-200, and IMDB account for 63% of the answers. Giving every dataset equal weight instead yields a mean ECE of 0.061 for Jev. Emotion alone reaches 0.279.\n\nSo 0.028 is the calibration error for this pooled collection. It measures how well probabilities match accuracy. A model can often be wrong yet accurately recognize that it is unsure.\n\nThe middle and right plots show biases in different directions:\n\n- **Single-question Noul**: the overall curve lies mostly above the diagonal. The average predicted “yes” probability is 0.465, versus an observed rate of 0.518, indicating a tendency to underestimate positives.\n- **Multilabel Noul**: the curve lies clearly below the diagonal. The average predicted probability is 0.209, but the observed positive rate is only 0.041, indicating a tendency to overestimate the probability that each label applies.\n\nIn the multilabel setting, each Noul question independently judges whether a label applies. There is no joint constraint expressing that a message usually has only a few labels. This is the authors' explanation for the bias. What the plot itself measures is the gap between predicted probabilities and positive frequencies.\n\nFor multilabel Noul, Qwen's ECE of 0.119 is lower than Jev's 0.168. If we average ECE equally across the 33 Choice or Noul datasets, the values are 0.075 and 0.074, respectively—almost identical. Jev's strong pooled Choice result does not translate into a calibration advantage in every setting.\n\n### Ranking, Thresholds, and F1\n\nConsider a six-example calculation: the first three examples are positive, and the last three are negative.\n\n| True label | Model's $P(\\text{yes})$ |\n| --- | --- |\n| Positive | 0.40, 0.30, 0.20 |\n| Negative | 0.15, 0.10, 0.05 |\n\nEvery positive ranks ahead of every negative, so the ranking is perfect. But if the program predicts positive only when `p >= 0.5`, it selects none of them.\n\nIn this case, AUROC, which measures ranking, can be 1, while F1, which measures the classification result, is 0.\n\n**F1 evaluates the classifications made after choosing a threshold.** It combines precision—the fraction of selected examples that really are positive—and recall—the fraction of all actual positives that were found. In the example, no positives were selected, so recall is 0 and F1 is also 0.\n\nTo examine the effect of thresholds, the authors take another 1,000 training or validation examples per dataset. For each Noul question, they search thresholds in steps of 0.01, choose the one giving the highest F1, and apply it unchanged to the evaluation split.\n\n| Dataset | What needs to be identified? | Metric | Fixed threshold of 0.5 | Threshold chosen on development data |\n| --- | --- | --- | ---: | ---: |\n| UNFAIR-ToS | Unfair provisions in terms of service | micro-F1 | 0.499 | 0.748 |\n| GoEmotions | Emotions expressed in comments | macro-F1 | 0.243 | 0.353 |\n| ToxicChat: jailbreaking | Messages attempting to bypass model restrictions | F1 | 0.717 | 0.833 |\n| AGB-DE | Invalid German consumer-contract clauses | F1 | 0.204 | 0.204 |\n\nThis table summarizes Jev's results in the paper's [**Table 4**](https://arxiv.org/html/2609.37647v1#S4.T4).\n\nMultilabel tasks benefit most. Across GoEmotions' 28 labels, the median selected threshold reaches 0.86, consistent with the overestimated positive probabilities in the previous plot. AGB-DE, however, does not improve at all. For identifying invalid German consumer-contract clauses, the best threshold chosen on the training examples remains 0.5, and AUROC is only 0.784. Here, the problem already includes difficulty separating positives from negatives; moving the threshold alone cannot solve it.\n\n### What Do We Gain by Answering Only Half?\n\nIf the model is allowed to leave some questions unanswered, we can use **selective prediction**: rank answers by confidence, accept only a portion, and pass the rest to a person or another model.\n\nTwo numbers need to be considered together:\n\n- **Coverage**: the fraction of all cases that this model is allowed to answer.\n- **Accuracy on retained answers**: how often the model is correct on the cases it accepts.\n\nStart at the far right of the plot. Coverage of 100% means accepting every answer. Moving left progressively excludes the lowest-confidence answers.\n\n[![The relationship between the fraction of high-confidence answers retained and accuracy for Jev on eight Choice datasets](/content/papers/assets/15581234207a38133b246aa950aa75bb7ec00c54027f78df7888d4120b20a800.png)](/content/papers/assets/15581234207a38133b246aa950aa75bb7ec00c54027f78df7888d4120b20a800.png)\n\n*Figure 3 from the paper, taken from [**page 10 of the v1 PDF**](https://arxiv.org/pdf/2609.37647v1#page=10). The horizontal axis shows coverage; the vertical axis shows accuracy only on retained answers.*\n\nOn Banking77, accuracy is 79.7% when every answer is accepted. Keeping only the most confident 50% raises it to 96.3%. So errors are not spread evenly across all answers; the lower-confidence half is more likely to be wrong, something classification accuracy alone cannot reveal.\n\nBut that 96.3% applies only to the retained half, and 3.7% of those answers are still wrong. Suppose every 1,000 inputs followed these proportions. We would automatically handle 500, still get about 19 wrong, and need someone to take over the other 500. Whether that trade is worthwhile depends on the situation.\n\nEmotion improves from 58.5% when answering everything to 73.3% when answering only half. Giving up the same fraction of questions still leaves accuracy far below Banking77's. An abstention mechanism cannot rescue every task.\n\nAt the same 50% coverage, Qwen reaches 94.5% on Banking77, close to Jev's 96.3%. On ANLI natural language inference, Qwen's 88.0% even exceeds Jev's 86.9%. These results support filtering answers by confidence, but that use is not unique to Jev.\n\n\n> [!TIP]\n> \n> **confidence is not the probability of being correct**\n> \n> For Choice calibration above, we used the highest-option probability, $p_{\\max}$. Here, answer filtering uses the `confidence` returned by Jev. To calculate confidence for the reference models, the authors use the same formula:\n> \n> $$\n> c=\\frac{Kp_{\\max}-1}{K-1}\n> $$\n> \n> $K$ is the number of options. With three options and $p_{\\max}=0.7$, we get $c=0.55$. The original probability of seventy percent becomes 0.55 after the transformation. Do not read that directly as a 55% chance of being correct.\n> \n> This transformation maps the equally likely baseline $1/K$ to 0 and complete certainty, 1, to 1. With a fixed number of options, the ranking remains the same as the ranking by highest probability. When the number of options changes, so does the transformation. The formula can be checked against the authors' [**evaluation implementation**](https://github.com/AppliedMachineLearning-Lab/jev-benchmarking/blob/87562114d23597f610754f057e5e0da8659dbcfb/jev_benchmarking/openmodel/prompts.py#L77-L80).\n\n\n### The Gap in Score Ratings\n\nThe authors use **Spearman's rank correlation coefficient** to see whether the model and human raters put items in a similar order. Values closer to 1 mean more consistent rankings. If people prefer A to B and the model also ranks A ahead of B, that helps with ordering. Whether they assign the same scores is a separate question.\n\nHere are the complete scoring results. The middle three columns compare the models' Spearman correlations: higher is better, and the highest value in each row is bold. The rightmost column shows **mean absolute error (MAE) for Jev only**, measuring the average difference from human ratings in rubric levels. Lower is better.\n\n[![Table 5 from the paper: Spearman correlations for Jev, Qwen, and Gemma on twelve scoring criteria, and Jev's mean absolute error](/content/papers/assets/e6922c4b31453fa2a2f8ade1b3d85d4883bb1909e196effa9009b037d2643bc5.png)](/content/papers/assets/e6922c4b31453fa2a2f8ade1b3d85d4883bb1909e196effa9009b037d2643bc5.png)\n\n*Table 5 from the paper, taken from [**page 11 of the v1 PDF**](https://arxiv.org/pdf/2609.37647v1#page=11). Click to enlarge. SummEval correlations are calculated separately for each source document and then averaged.*\n\nOn STS-B sentence semantic similarity, Jev reaches 0.890. On SST-5, five-way sentiment classification accuracy is only 57.9%, but the Spearman correlation between weighted scores and human labels is 0.851. Classifying “very positive” as “somewhat positive” still counts as wrong in classification, but in a ranking it is closer than calling it negative.\n\nRating the quality of generated content is harder. HelpSteer2 asks people to rate model responses. For helpfulness and correctness, Jev's correlations with human ratings are only 0.399 and 0.369, respectively.\n\nSummEval evaluates summary quality. Its fluency rank correlation is 0.495, but the mean absolute error reaches 1.77 rubric levels. In other words, the absolute gap between each score and the experts' score averages almost two levels, with Jev generally rating lower.\n\n### MMLU: Where Do the High Scores Come From?\n\n[**MMLU**](https://arxiv.org/abs/2009.03300) and [**C-Eval**](https://arxiv.org/abs/2305.08322) are multiple-choice knowledge tests spanning many subjects, primarily in English and Chinese, respectively. An unusual result appears here: on MMLU, Jev performs better in calculation-heavy subjects than in the other subjects.\n\nThe authors divide subjects into calculation-heavy and other groups. Mathematics, physics, chemistry, statistics, accounting, and formal logic go into the calculation-heavy group. This is a subject-level grouping, not a question-by-question check that every problem requires calculation.\n\nIn the plot below, each point is a subject's accuracy, and the box plots summarize the distribution across subjects. Calculation-heavy subjects are weaker on C-Eval, on the left. MMLU, on the right, does not show the same decline.\n\n[![Accuracy distributions for Jev on calculation-heavy and other subjects in C-Eval and MMLU](/content/papers/assets/aa4f91723d35c79b5563a1155d884848c69d49f50d5ec492bb5a6a981566d768.png)](/content/papers/assets/aa4f91723d35c79b5563a1155d884848c69d49f50d5ec492bb5a6a981566d768.png)\n\n*Figure 6 from the paper, taken from [**page 13 of the v1 PDF**](https://arxiv.org/pdf/2609.37647v1#page=13). Each point represents one subject; read the two datasets separately.*\n\nAggregating by question, Jev's MMLU accuracy is 94.3% in calculation-heavy subjects and 91.3% in the others. The two reference models show the opposite pattern: 74.7% versus 83.0% for Qwen, and 54.6% versus 69.8% for Gemma.\n\nThe authors therefore check two possible shortcuts:\n\n1. **Rotate the answer-option positions.** Each answer moves to a different letter, yet Jev's 94.3% remains unchanged. This does not support an explanation based only on memorized answer positions.\n2. **Remove the question, leaving only the subject and options.** Accuracy drops to 31.5%, well below the original result but still above the 25% expected from random guessing with four options. The option contents alone cannot reproduce the original high score either.\n\nBut what if the model memorized whole question-and-answer combinations? Reordering the options would not prevent that, and removing the question would simply take away a cue used to recognize it. This is why **data contamination** is such a headache for evaluation: questions meant to be held back for the test may already have entered the model's training data. The authors leave this possibility open, but without access to Jev's training data, they cannot confirm it.\n\n## Conclusion\n\n**Ninety percent confidence does not always mean ninety percent accuracy.**\n\nThis paper evaluates `jev-1.13.0` on 37 datasets and finds the following:\n\n- **Multiple-choice probabilities are better calibrated overall, but the task matters.** On Emotion, for example, Jev clearly overestimates its certainty. A good overall average does not apply to every task.\n- **Yes/no judgments are biased.** Single-question Noul slightly underestimates “yes,” while multilabel Noul clearly overestimates it: the average predicted probability that a label applies is 20.9%, but the observed rate is only 4.1%. Watch for this when, for example, assigning several emotion labels to one comment.\n- **Confidence scores help select answers, but someone must handle the rest.** For Banking77 banking-service classification, keeping only the more confident half raises accuracy from 79.7% to 96.3%. Applying those proportions to 1,000 cases means automatically handling 500, still getting about 19 wrong, and passing the other 500 elsewhere.\n- **Ratings of generated content still differ substantially from human judgments.** For SummEval summary fluency, for example, Jev's scores differ from the experts' by an average of 1.77 levels and are generally lower. The two sides have yet to agree even on whether a passage reads smoothly.\n\nFinally, thanks to everyone in the community contributing evaluations and research from different angles. Their work gives us more guidance for practical use and helps us avoid quite a few wrong turns.\n","imageSizes":{"/content/papers/assets/c38715617566fd975974709b55c9984cf286a1ff5745dba0cb0f63bbddeaf348.png":{"width":1308,"height":592},"/content/papers/assets/15581234207a38133b246aa950aa75bb7ec00c54027f78df7888d4120b20a800.png":{"width":1308,"height":592},"/content/papers/assets/e6922c4b31453fa2a2f8ade1b3d85d4883bb1909e196effa9009b037d2643bc5.png":{"width":864,"height":448},"/content/papers/assets/aa4f91723d35c79b5563a1155d884848c69d49f50d5ec492bb5a6a981566d768.png":{"width":1111,"height":558}}}
