Can Large Language Models Learn to Ask Better Questions?
- Sofia Ng
- 4 days ago
- 2 min read
Ever had a conversation with an AI assistant that felt... well, a little off? You can have a conversation and it just feels off, the responses and follow up questions feel disconnected or do not add to the flow.
Or worse—it didn't ask for more details at all. In the real world, good conversations rely on asking the right follow-ups, whether it’s a doctor diagnosing an illness or a lawyer building a case. But AI? It’s still catching up.
That’s what a team of researchers—Shuyue Stella Li, Jimin Mun, Faeze Brahman, Jonathan S. Ilgen, Yulia Tsvetkov, and Maarten Sap wanted to tackle. Their recent study looks at how Large Language Models (LLMs) can learn to ask better questions, particularly in clinical reasoning.
Why Does Question Quality Matter?
In fields such as healthcare, legal analysis, and investigative journalism, simply providing answers isn’t enough. Clinicians, for instance, follow a structured process to ask follow-up questions that refine diagnoses. This study highlights a significant gap in LLMs: they often fail to proactively seek critical missing information, which makes them unreliable in expert-driven domains.
The authors developed ALFA (Alignment via Fine-grained Attributes), a framework that decomposes a “good” question into multiple attributes such as clarity, focus, answerability, medical accuracy, diagnostic relevance, and avoidance of bias. Their aim? To align AI models so they can ask better, more contextually aware questions.
How Did They Train AI to Ask Better Questions?
The team introduced the MediQ-AskDocs dataset, built from 17,000 real-world clinical interactions and enriched with 80,000 counterfactual question variations. Using this dataset, they trained models to:
Identify when additional information is required.
Ask follow-up questions that are precise and medically relevant.
Reduce diagnostic errors by refining their approach to questioning.
By breaking down question quality into distinct attributes and training models using preference-based optimization, ALFA achieved a 56.6% reduction in diagnostic errors compared to standard instruction-tuned models. In side-by-side comparisons, human evaluators preferred ALFA-generated questions 64.4% of the time.
AI in Decision Support Roles
This study sheds light on the broader challenge of making AI more interactive and useful in decision-making environments. Currently, AI assistants often struggle with:
Retaining context over multiple interactions.
Understanding when their knowledge is incomplete.
Adjusting their questioning style based on the expertise of the user.
The research suggests that explicitly training models to ask better questions could significantly improve AI’s role in expert-driven tasks. This applies not only to healthcare but also to areas like legal compliance, scientific research, and complex troubleshooting.
Limitations and Future Directions
While ALFA demonstrates clear improvements, it’s not without limitations:
The training data is sourced from online health forums, which may not fully reflect real-world clinical settings.
The method still requires significant human oversight to validate AI-generated questions.
Aligning LLMs in other expert domains will require tailored datasets and attribute-specific training.
This research highlights a shift in AI development, from models that answer questions to those that actively engage in effective questioning. With further refinement, AI could become a more reliable assistant supporting professionals rather than just offering pre-scripted responses.
For those interested in exploring the dataset or methodology, the researchers have made their resources available at:
What do you think—will AI ever match human intuition when it comes to asking the right questions?