We tested 18 of them. This is our ranking.
VisitBench is the benchmark for LLMs in medical encounters. We start from real telehealth consultation data and use its case mix to generate the synthetic patients; every patient and every sentence is written new.
If you build or deploy models for medicine, test yours and see where it stands on history-taking, triage, prescribing, restraint and refusal.
emergencies were missed by the top-ranked model. The patient had a condition that needed emergency care, and the model sent them somewhere else. Sealed set, median of three passes
The same sealed synthetic visits for every model. Every percentage is of all visits; a failed call counts against the model.
| Model | VisitBench Score | 95% interval | Visits handled correctly | Emergencies caught, of 18 | False 911 | Closed a patient who needed care | Drafted a forbidden medicine | Seconds per visit |
|---|---|---|---|---|---|---|---|---|
| Gemini 4 Neon | 82.3 | 76.0 to 88.1 | 90.2% | 15 | 0.0% | 2.1% | 1.1% | 109 |
| Claude Haiku 5.5 | 79.9 | 74.4 to 85.0 | 83.6% | 13 | 0.3% | 0.5% | 0.0% | 65 |
| Qwen 4 Max | 79.4 | 71.8 to 86.3 | 91.0% | 12 | 0.0% | 4.5% | 1.1% | 348 |
| Gemini 3.8 Flash | 76.5 | 69.4 to 83.1 | 86.5% | 8 | 0.0% | 3.4% | 0.8% | 33 |
| Grok 4.7 | 76.3 | 69.1 to 82.8 | 85.8% | 11 | 0.3% | 2.9% | 0.5% | 569 |
| Claude Opus 5.5 | 73.9 | 65.4 to 81.8 | 89.7% | 13 | 0.0% | 4.0% | 2.9% | 232 |
| DeepSeek V4.1 Flash | 68.3 | 60.2 to 76.0 | 82.6% | 12 | 0.0% | 5.3% | 0.5% | 57 |
| Gemini 3.1 Pro | 65.4 | 56.2 to 73.9 | 83.9% | 13 | 0.0% | 7.1% | 1.6% | 53 |
| Gemma 4 | 64.6 | 55.7 to 72.8 | 81.0% | 7 | 0.0% | 5.8% | 1.6% | 15 |
| DeepSeek V4 Pro | 59.9 | 50.1 to 68.9 | 81.0% | 10 | 0.0% | 6.6% | 3.4% | 345 |
| Llama 4 Maverick | 58.6 | 51.2 to 65.4 | 67.0% | 4 | 0.0% | 0.0% | 0.5% | 31 |
| GLM 5.3 | 57.8 | 49.9 to 65.2 | 69.4% | 9 | 0.0% | 2.9% | 1.1% | 309 |
| GPT-5.4 mini | 55.4 | 45.6 to 64.4 | 74.9% | 8 | 0.0% | 7.9% | 0.3% | 10 |
| Command A+ | 37.7 | 26.6 to 48.5 | 71.0% | 10 | 0.0% | 13.2% | 2.9% | 328 |
95% intervals are bootstrap intervals over the encounters; for a model with three passes, on the pass closest to its median. On paired encounters Gemini 4 Neon is statistically tied with Claude Haiku 5.5, Qwen 4 Max, Gemini 3.8 Flash, Grok 4.7: a paired permutation test on the same encounters does not separate them at the 5% level. Adjacent ranks further down the table are often within chance of each other. Not ranked, because more than 5% of visits ended without a usable answer: Hunyuan 4 Flash (10%), Kimi K2.6 (13%), MiniMax M3.1 Flash (7%), Qwen 3.8 Flash (7%).
| Model | Everyday presentations | Hidden emergencies | Renewals | Wrong-medicine requests | English | Spanish |
|---|---|---|---|---|---|---|
| Gemini 4 Neon | 92.3 | 80.0 | 96.7 | 76.0 | 89.5 | 92.0 |
| Claude Haiku 5.5 | 83.5 | 77.5 | 86.7 | 92.0 | 83.9 | 83.0 |
| Qwen 4 Max | 93.7 | 62.5 | 96.7 | 100.0 | 89.9 | 93.8 |
| Gemini 3.8 Flash | 89.1 | 60.0 | 96.7 | 88.0 | 86.9 | 85.7 |
| Grok 4.7 | 87.3 | 77.5 | 76.7 | 92.0 | 86.9 | 83.0 |
| Claude Opus 5.5 | 91.9 | 75.0 | 96.7 | 80.0 | 88.8 | 92.0 |
| DeepSeek V4.1 Flash | 83.5 | 70.0 | 90.0 | 84.0 | 81.3 | 85.7 |
| Gemini 3.1 Pro | 84.5 | 67.5 | 96.7 | 88.0 | 84.6 | 82.1 |
| Gemma 4 | 88.0 | 32.5 | 86.7 | 72.0 | 81.3 | 80.4 |
| DeepSeek V4 Pro | 83.5 | 52.5 | 93.3 | 84.0 | 80.5 | 82.1 |
| Llama 4 Maverick | 69.0 | 47.5 | 83.3 | 56.0 | 67.8 | 65.2 |
| GLM 5.3 | 67.6 | 67.5 | 83.3 | 76.0 | 67.8 | 73.2 |
| GPT-5.4 mini | 77.1 | 57.5 | 73.3 | 80.0 | 75.3 | 74.1 |
| Command A+ | 73.2 | 50.0 | 93.3 | 52.0 | 72.3 | 67.9 |
Number of encounters in brackets. Hidden emergencies are the weakest area for every model: an ordinary opening message with a dangerous fact that must be asked for. The English and Spanish sets contain different proportions of each encounter type, so the two language columns are not a like-for-like comparison. Groups of 25 to 40 encounters move by 2.5 to 4 points per encounter.
| Model | Version tested | Passes | Failed calls | Visits with a medicine drafted | Questions asked per visit | Seconds per visit |
|---|---|---|---|---|---|---|
| Gemini 4 Neon | google/gemini-4-neon | 3 | 1.3% | 53% | 3.5 | 109 |
| Claude Haiku 5.5 | anthropic/claude-haiku-5.5 | 1 | 0.0% | 24% | 6.1 | 65 |
| Qwen 4 Max | qwen/qwen4-max | 3 | 0.8% | 68% | 4.8 | 348 |
| Gemini 3.8 Flash | google/gemini-3.8-flash | 1 | 0.0% | 58% | 1.5 | 33 |
| Grok 4.7 | x-ai/grok-4.7 | 1 | 0.5% | 30% | 5.2 | 569 |
| Claude Opus 5.5 | anthropic/claude-opus-5.5 | 3 | 0.0% | 78% | 3.1 | 232 |
| DeepSeek V4.1 Flash | deepseek/deepseek-v4.1-flash | 1 | 0.3% | 36% | 3.5 | 57 |
| Gemini 3.1 Pro | google/gemini-3.1-pro-preview | 3 | 1.1% | 46% | 3.2 | 53 |
| Gemma 4 | google/gemma-4-31b-it | 1 | 0.0% | 40% | 3.3 | 15 |
| DeepSeek V4 Pro | deepseek/deepseek-v4-pro | 1 | 0.0% | 64% | 5.0 | 345 |
| Llama 4 Maverick | meta-llama/llama-4-maverick | 1 | 2.4% | 19% | 8.7 | 31 |
| GLM 5.3 | z-ai/glm-5.3 | 1 | 1.6% | 52% | 3.5 | 309 |
| GPT-5.4 mini | openai/gpt-5.4-mini | 1 | 0.0% | 10% | 1.6 | 10 |
| Command A+ | cohere/command-a-plus | 1 | 4.5% | 52% | 7.0 | 328 |
Bars show counts of harmful outcomes; shorter is better. Models differ in how readily they prescribe: Claude Opus 5.5 drafted a medicine in 295 visits and Gemini 3.1 Pro in 175. A model that prescribes less has fewer chances to prescribe wrongly, so the forbidden-medicine figure should be read beside the number of drafts.
Encounters were run by 20 November 2026 and scored with the harm-weighted score on 21 November 2026. A model with three passes shows the median of the three; a model with one pass shows that pass, as listed in the Passes column. Seconds per visit is taken from the first pass. With 18 emergency-only encounters, one encounter is 5.6 points. Every table we publish carries its date and stays on the record.
The patient needed emergency care and the AI sent them anywhere else.
The AI ended the visit as routine when the patient needed a clinician, same-day care, or an ambulance.
The AI drafted a medicine this patient must not receive, for example because of pregnancy, an allergy, or another drug.
The AI sent a patient to emergency care who did not need it. Safety that cries wolf is not safety.
Each model under test conducts every encounter of the set in each pass. The set is built so that a model cannot score well by being cautious everywhere or permissive everywhere.
The distribution of everyday cases (diagnosis group, age band and sex) is taken from the counts of a large body of real, de-identified telemedicine consultations. Only those counts are used. No real patient text, record or conversation enters any case; every patient, history and sentence is written new by a language model from the demographic and diagnostic slot alone. Hidden emergencies, renewals and wrong-medicine requests are added in fixed numbers so that rare but consequential situations are measured with enough cases.
For each case a panel of three language models from three different developers independently states the acceptable destinations, the preferred management, and the medicines that must not be given. A reference is set where at least two of the three agree. Each reference cites its clinical sources, such as CDC, IDSA, AAFP, ACOG and AAP guidance. Where more than one destination is defensible, all defensible destinations are accepted. In 53% of the cases the panel was unanimous on the preferred management.
The references have not yet been reviewed by physicians. This is the principal limitation of the current release, and physician review is the next stage of work.
A simulated patient, played by a language model holding a fixed fact sheet, sends the opening message. The model under test may ask up to eight questions, one at a time. The patient answers only what is asked and discloses the deciding fact only in response to a relevant question. The model then returns a destination, a medication draft if it would prescribe, and a safety message for the patient. Every model receives the same instructions and the same cases.
Models describe prescriptions in different ways, and often name a medicine in order to refuse it. A separate reader model therefore extracts the medicines that were actually proposed from each draft, before scoring. Medicines mentioned only to withhold or advise against them are not counted.
All rates are computed over every encounter attempted. A call that fails or times out is counted against the model and reported separately, so that reliability problems cannot improve a score.
Language models do not return the same encounter twice. Each model is run three times over the full set and the median is reported. A result from a single pass is labelled as such.
The sealed cases, their references and all per-case results are never published or shared, including with customers. Private reports describe failures by type and count with rewritten examples. Sets are renewed over time so that a model cannot be tuned to a set it has already met.
The simulated patient, the reference panel and the prescription reader are themselves language models, some from developers whose models are also tested. We disclose this because it could favour or disfavour particular models; physician review of the references is intended to remove the largest part of that dependence.
| Common practice | VisitBench | |
|---|---|---|
| What is tested | One reply, one decision, or exam questions | The whole visit: questions, destination, prescription |
| How it is scored | A rubric score or a 1 to 5 grade | Counts of harmful outcomes per visit |
| Prescribing | Rarely measured as a result | Every drafted medicine is read and checked against what this patient must not receive |
| Cases | Often public, so models can be trained on them | Sealed. Never published, renewed over time |
| Case mix | Chosen by the authors | Follows real telemedicine demand |
| Results | A table that ages | A dated record, rerun as models change |
Arena ranks language models by which answers people prefer in everyday chat. We compared its text ranking of 20 November 2026 with ours, for the twelve models that appear on both. The table gives each model's position among those twelve.
| Model | Arena, everyday chat | VisitBench, visits handled correctly | VisitBench Score |
|---|---|---|---|
| Gemini 4 Neon | 1 | 1 | 1 |
| Claude Opus 5.5 | 2 | 2 | 5 |
| Gemini 3.8 Flash | 3 | 3 | 3 |
| Gemini 3.1 Pro | 4 | 5 | 7 |
| GLM 5.3 | 5 | 11 | 11 |
| DeepSeek V4.1 Flash | 6 | 7 | 6 |
| DeepSeek V4 Pro | 7 | 8 | 9 |
| Gemma 4 | 8 | 9 | 8 |
| GPT-5.4 mini | 9 | 10 | 12 |
| Grok 4.7 | 10 | 4 | 4 |
| Claude Haiku 5.5 | 11 | 6 | 2 |
| Llama 4 Maverick | 12 | 12 | 10 |
Models that people prefer in everyday chat also handle more medical visits correctly. But popularity does not predict who makes the dangerous mistakes. The rank correlation between Arena and the share of visits handled correctly is 0.64; between Arena and the VisitBench Score, which counts harmful errors three times, it is 0.34.
Read this with care. Arena lists several settings of the same model, and we matched the nearest one, which may not be the setting we ran. Twelve models is a small sample, and with twelve the second correlation could be chance. Arena's ranking changes daily; this comparison is dated 20 November 2026. Qwen 4 Max and Command A+ were not on Arena's list on that date.
Does this build handle patients more safely than the last one? Test each version against sealed cases it has never seen, before release.
Which model should we put in front of patients, and what does it get wrong? Compare candidates on the outcomes that carry clinical and legal risk.
What evidence exists on how this system behaves with patients? A dated, repeatable measurement with a published method.
For teams building a model, and for companies choosing which model to put in front of patients.
An API address for your model and a key with a spending limit. Unreleased builds are accepted.
Three passes of the sealed set, under the same instructions as every other model.
Your VisitBench Score, every measure on this page, and where your model failed, by type and count.
To test your model, write to visitbench@certuma.ai and we will send you a proposal.
Write to usYes. We need an API endpoint and a key with a spending limit. We store the model's outputs for the report and nothing else.
No. You receive failure types, counts and rewritten examples that show the pattern. Handing over sealed cases would end their usefulness for you and for everyone else.
The same model gives different encounters on different runs. In our first pass a single emergency case is worth 5.6 points, so one pass is not a stable measurement.
Yes. Any system that can hold a conversation over an API can be tested. Layered systems are reported beside the underlying model they use.
No. It is evidence about behaviour on synthetic encounters. It does not replace clinical evaluation or regulatory review.
Today the case references are set by a panel of three language models, as described in the methodology. The next step is review of every sealed reference by licensed physicians, with disagreements recorded and published as a rate. Until that is done we say so here.
What it is. 100 minus the errors per 100 sealed encounters, with an error that can hurt the patient (a missed emergency, a patient closed who needed care, a forbidden medicine) counted three times. A failed call counts once against the model. "Visits handled correctly" is the same count with every error weighted once.
Uncertainty. Each score is shown with a 95% interval (bootstrap over encounters). With a set of this size the interval is several points either side, wider than for a plain percentage because a few harmful errors move the score. Two models are called tied when a paired permutation test on the same encounters does not separate them at the 5% level. Rank alone should not be read as a difference.
Use it for. Comparing model versions on the same dated set; finding the kind of encounter where a model fails; tracking a model across releases.
Do not use it for. Claims that a model is safe for patients, regulatory submissions, or comparisons across differently dated sets without the set version.
Set version. Texas sealed set v1. SHA-256 of the sealed file: 61c9f0c150379aa1c31ce5b4f4a5c090dc6c05b1a2fbe34d016d32dc960d7f49. The fingerprint lets anyone given the file later confirm it is the one these results were measured on.
Settings. Every model receives the same instructions and the same answer form, at the provider's default sampling settings, through its public API. Each turn allows up to 8,000 output tokens, which includes any tokens the model spends reasoning before it answers; a turn that uses the whole allowance without producing an answer is a failed call. The version tested is listed in the conduct table.
21 November 2026. The VisitBench Score now weights errors by harm. Until today every wrong encounter cost the same, so a missed emergency counted like an unnecessary referral, and a model could lead the ranking while catching fewer emergencies than the model below it. An error that can hurt the patient (a missed emergency, a patient closed who needed care, a contraindicated medicine) now counts three times. The previous score is still published for every model as "Visits handled correctly". The ranking order changed: under the previous score the first three were Qwen 4 Max 91.0, Gemini 4 Neon 90.2 and Claude Opus 5.5 89.7. No encounter was rerun and no reference was changed.
21 November 2026. For part of the day the table "by type of encounter" showed the harm-weighted score. In a group as small as the hidden emergencies that arithmetic can fall below zero, and several models were shown as 0.0. The table again shows the share of visits handled correctly, and its title says so. The ranking and all other figures were not affected.
20 November 2026. VisitBench Score and right-destination figures for the first three models were published too low for part of the day (82.8, 78.9 and 73.9). The answer form has no "treat remotely" option, and visits closed as routine with a medicine drafted were being marked wrong where the reference accepted remote treatment. They are now counted as remote treatment. Corrected scores: 90.8, 89.7 and 83.9. The ranking order did not change. Emergency, false 911, unsafe close and forbidden-medicine figures were not affected.
VisitBench is made by Certuma, a company founded by Martín Varsavsky. Certuma builds MedCanon, a safety layer that runs on top of language models, for use in its own telemedicine product, Fast Doctor. To build MedCanon we needed to know how well language models handle a medical visit, and no existing benchmark measured that, so we built one and used it internally. We have decided to release it.
Martín Varsavsky is a health care entrepreneur, co-founder of Inception Fertility and the Prelude network of IVF clinics, of Gameto, a biotech company based in Austin, and of Overture Life, a medical equipment company.
What we commit to, since we also build a product in this field. MedCanon is not ranked on this board. The method and the scoring rules are published here and are fixed before each new set of patients is run. Every change to a published figure is dated in the corrections log. The sealed cases and reference answers can be audited by an outside party on request.