Best AI Models for Medical Diagnosis

Why Abductive Reasoning Matters for Diagnostic AI

Abductive reasoning is critical in cases where patient data is incomplete or ambiguous. It involves ranking plausible diagnoses and suggesting further tests or treatments. This approach helps clinicians manage uncertainties in patient symptoms and complex decision-making, ultimately improving accuracy and integrating seamlessly into clinical workflows.

How We Compared AI Models for Medical Diagnosis

We evaluated models like o1 mini, GTP-4o mini, Copilot, and Gemini by analysing their performance across 10 detailed patient cases. These cases included symptoms, medical histories, risk factors, and test results. For the Windows Copilot, we selected a more precise style setting to enhance accuracy.

Our approach was zero-shot prompting, leveraging the models' pre-existing knowledge base in a structured, case-based format. The evaluation consisted of 13 specific criteria to measure the effectiveness of each AI model.

AI Diagnosis Model Results

Accuracy

• GTP-4o mini: Typically provides accurate diagnoses for straightforward and well-defined cases (e.g., PID, post-traumatic epilepsy). However, in more complex cases (e.g., advanced lung cancer, neuroborreliosis), the model often oversimplifies or misses nuances.
• Copilot: Offers generally accurate diagnoses in common medical cases. However, in more complex cases, it provides less specificity and depth, making it less reliable than o1 mini.
• Gemini: Provides accurate diagnoses across a wide range of cases, often performing slightly better than Copilot. It handles common and some moderately complex conditions well but lacks the fine-tuned accuracy for rare or highly nuanced cases.
• o1 mini: Consistently offers the most accurate and detailed diagnoses, even in complex and ambiguous cases. It correctly identified rare and challenging diagnoses, such as advanced non-small cell lung cancer (NSCLC) and neuroborreliosis.

Hypothesis Generation and Evaluation

• GTP-4o mini: Tends to focus on a single diagnosis, rarely exploring alternative hypotheses. This approach works well in clear cases but fails in ambiguous or complex cases.
• Copilot: Generates reasonable hypotheses but focuses primarily on the most likely one, without thoroughly considering less obvious possibilities.
• Gemini: Performs similarly to Copilot but occasionally provides a slightly broader evaluation of alternatives. It is generally effective but does not excel in generating multiple competing hypotheses.
• o1 mini: Consistently offers the most thorough hypothesis generation, exploring a wide range of possibilities and evaluating each one thoroughly. This was evident in cases with complex presentations, such as advanced lung cancer and neuroborreliosis.

Handling Ambiguity and Uncertainty

• GTP-4o mini: Struggles with ambiguity, often committing to a single diagnosis without addressing uncertainties. It performs best when patient data is complete and clear.
• Copilot: Handles moderate levels of uncertainty but lacks depth in ambiguous cases. It tends to provide confident diagnoses without acknowledging complexities.
• Gemini: Manages uncertainty reasonably well in typical cases but does not explore alternative diagnoses in depth when symptoms are ambiguous.
• o1 mini: Best at handling uncertainty by considering multiple diagnoses and explaining why certain conditions are less likely. It excels in cases where patient data is incomplete or ambiguous, such as advanced metastatic lung cancer and potential neuroborreliosis.

Differential Diagnosis

• GTP-4o mini: Provides minimal exploration of differential diagnoses, focusing on the most obvious option.
• Copilot: Offers basic differential diagnoses but often lacks breadth, usually focusing on one or two likely possibilities.
• Gemini: Provides some exploration of differential diagnoses, especially in more complex cases, but generally ranks fewer alternatives than o1 mini.
• o1 mini: Consistently offers the broadest and most detailed differential diagnoses, ranking multiple possibilities and explaining their relevance. This was particularly strong in cases like idiopathic intracranial hypertension (IIH) and post-traumatic epilepsy.

Chain-of-Thought Reasoning

• GTP-4o mini: Provides logical but simplified reasoning, often lacking the step-by-step approach needed for complex cases.
• Copilot: Offers clear but basic reasoning, suitable for straightforward cases but lacking depth in more complex cases.
• Gemini: Provides reasonable chain-of-thought reasoning, although not as detailed or multi-step as o1 mini. It tends to perform well in typical cases.
• o1 mini: Offers the most detailed and logical reasoning, with clear step-by-step explanations that connect symptoms, test results, and diagnoses. This approach was particularly valuable in complex cases like metastatic lung cancer and IIH.

Clinical Relevance

• GTP-4o mini: Provides standard clinical recommendations aligned with guidelines but lacks depth in suggesting advanced treatments or tests.
• Copilot: Offers relevant and practical clinical recommendations but does not suggest advanced or personalised treatment options.
• Gemini: Provides clinically relevant recommendations but lacks the advanced treatment suggestions and detailed follow-up plans that o1 mini offers.
• o1 mini: Offers the most detailed and clinically relevant treatment recommendations, including advanced therapies (e.g., molecular testing, immunotherapy) and personalised care plans. It consistently aligns with clinical guidelines while providing depth in treatment options, especially in complex cases like NSCLC.

Interpretability

• GTP-4o mini: Provides clear but overly simplified explanations, often lacking the detail needed in complex cases.
• Copilot: Offers clear and understandable reasoning but can be brief in its explanations, especially in complex or ambiguous cases.
• Gemini: Provides reasonably clear and concise explanations but lacks the depth of o1 mini.
• o1 mini: Offers the clearest and most detailed explanations, breaking down complex cases in a way that is easy to follow and understand. This was particularly useful in high-risk cases like metastatic lung cancer and advanced neuroborreliosis.

Adaptability

• GTP-4o mini: Struggles with adaptation, providing generalised diagnoses and treatments without much customisation for individual patients.
• Copilot: Offers some adaptability but generally follows a standardised approach, lacking customisation for unique patient presentations.
• Gemini: Reasonably adaptable, offering some level of customisation based on patient history, but does not adapt as well as o1 mini.
• o1 mini: Most adaptable, tailoring diagnoses and treatments based on individual patient histories, risk factors, and specific test results. This was evident in cases involving complex comorbidities or chronic conditions.

Comorbidities

• The ability to manage complex cases involving multiple conditions.
• GTP-4o mini: Does not handle comorbidities well, typically focusing on a single diagnosis without addressing multiple conditions.
• Copilot: Manages comorbidities reasonably well in common cases but lacks depth in more complex presentations.
• Gemini: Provides some consideration of comorbidities, especially in moderately complex cases, but does not handle them as well as o1 mini.
• o1 mini: Consistently addresses comorbidities and associated complications, providing comprehensive management plans for multiple conditions. This was particularly evident in cases like advanced lung cancer with cachexia and anaemia.

Robustness

• GTP-4o mini: Performs well in clear-cut cases but struggles when data is incomplete or ambiguous.
• Copilot: Reasonably robust in handling typical cases but does not perform well in highly complex cases with incomplete data.
• Gemini: Offers moderate robustness, handling some complexity but struggling with highly nuanced or ambiguous cases.
• o1 mini: Most robust, consistently handling incomplete or noisy data and offering clear diagnoses in complex and ambiguous situations. This was evident in cases like NSCLC with metastasis and neuroborreliosis.

Comparative Performance

• GTP-4o mini: Provides reliable but basic performance, suitable for common cases but not on par with expert-level decision-making in complex cases.
• Copilot: Consistent and reliable in common medical cases but does not match the expert-level performance of o1 mini in more complex cases.
• Gemini: Offers consistent performance in routine cases and moderately complex situations but lacks the advanced reasoning of o1 mini.
• o1 mini: Consistently outperforms other models, providing expert-level performance in handling complex and ambiguous cases, with high consistency across all patient cases.

User Experience

• GTP-4o mini: Fast and easy to use but lacks depth, making it more suitable for simple cases.
• Copilot: Provides a good balance between usability and clarity, making it suitable for typical clinical workflows.
• Gemini: Offers a user-friendly experience with clear and concise explanations, integrating well into clinical workflows.
• o1 mini: While detailed, it remains user-friendly and offers actionable recommendations, though the depth of information may be overwhelming in simpler cases.

Model Efficiency

• GTP-4o mini: Most efficient in terms of speed but sacrifices depth and accuracy in complex cases.
• Copilot: Offers reasonable efficiency, balancing speed with practical recommendations.
• Gemini: Provides good efficiency with a balance of detail and speed.
• o1 mini: Least efficient due to the level of detail and depth, but this trade-off is acceptable in complex cases where accuracy and thoroughness are critical.

Model-by-Model Comparison: Which AI Performs Best in Medical Diagnosis?

1. o1 mini: Emerges as the top model for medical diagnosis, particularly adept at handling complex, ambiguous cases. It excels in generating detailed and accurate diagnoses, offering a comprehensive evaluation of possible conditions, and evaluating differential diagnoses.
2. Gemini: Known for its user-friendliness and efficiency, Gemini performs well in clear-cut clinical scenarios. However, it falls short in more complex cases, lacking the depth of o1 mini.
3. Copilot: This model provides reliable, practical diagnoses in routine medical settings but struggles with deeper diagnostic challenges and abductive reasoning in intricate cases.
4. GTP-4o mini: Fast and straightforward, it is ideal for uncomplicated cases where the diagnosis is clear. However, its simplicity is a limitation in more complex diagnostic situations.

Key Takeaways: Choosing the Right AI Model for Clinical Diagnosis

Choosing the right AI model depends heavily on the specific needs of the clinical environment and the complexity of the case. The o1 mini model is particularly valuable in settings where in-depth analysis and comprehensive patient management are essential. In contrast, models like Gemini and Copilot are more suited to cases with less complexity.