Querying Regulated Clinical Data Without Generated Numbers
A language model in front of roughly 200,000 lines of clinical SQL, with metric definitions, arithmetic and validation rules held in deterministic code so that a wrong answer cannot arrive as a fluent sentence.
- Generative AI
- Model Context Protocol
- Regulated Environments
- Data Engineering
- 32
- Failure modes permanently closedRules held in the deterministic engine at the time of writing. Each one is a distinct wrong answer found in production by a clinician or domain expert and then converted into code, so that category of error cannot recur for any user. The count is a measure of errors closed, not of system complexity.
- Zero
- Generated arithmetic in a reported figureEvery number in a daily or weekly leadership report is computed in deterministic code and handed to the model, which writes only the interpretation around it. The model is never asked to work a value out.
Overview
Almost every organization has now built a GenAI demo that worked. Far fewer have one running in production against data that matters, and the gap between those two states is where most of the money goes.
In most text to SQL applications a bad query returns something obviously broken. Here it returns a number, a clinician acts on it, and nothing indicates anything went wrong. The design problem was therefore not how to make the system capable. It was how to make it unable to be confidently wrong.
The answer is a division of labor enforced in the architecture rather than in prompting. Metric definitions, arithmetic and validation rules are owned by deterministic code. Interpreting the question, navigating a schema too large to describe, planning retrieval and explaining the answer are owned by the model.
The Context
The client is not named. It is a healthcare technology company selling clinical operations and revenue cycle software to hospital systems in the United States. Behind its products sits a database built up over many years: more than a hundred stored procedures, many running to a couple of thousand lines each, plus a comparable number of interdependent tables and views.
Somewhere inside roughly two hundred thousand lines of SQL is the definition of every clinical and operational metric the business reports on. How a case is counted. What makes a cancellation. Which procedures roll up into which service line. Much of that logic was never documented anywhere else.
The organization had not lost its expertise. It had lost readable access to it. The knowledge still existed. It was only executable, not legible.
The Challenge
Questions became tickets. They queued behind engineering work, they were answered by the few people who knew the schema, and by the time an answer arrived the decision had often been made without it. Meanwhile those engineers were doing data extraction rather than the modernization work the business needed from them.
Text to SQL is one of the oldest demonstrations in the GenAI catalog and one of the least reliable in production. Three things make this environment harder than the demo.
- The schema is too large to describe. A hundred plus tables and views will not fit usefully into a prompt, and truncating it means the model invents the parts it cannot see
- The meaning is in the procedures, not the schema. A model can read column names and guess, and guessing is precisely the failure mode you cannot accept
- The cost of a plausible wrong answer is high, because the answer is a number that looks exactly like a correct one
The organizations that get past this stage tend to have made one decision early. They were explicit about what the model is not permitted to do.
What Was at Stake
A system that is right most of the time is worse than one that is wrong most of the time, because people stop checking. That is the point at which an organization discovers its AI system has been quietly wrong for a while, usually because volume rose past what anyone was actually reading.
In a regulated business the build is rarely the hard part. Approval is. A second line validation function will not accept a system whose correctness rests on prompt quality, because a prompt is not evidence and cannot be re-run against a control. Most GenAI programs stall at exactly that point, because the evidence was never designed in, and retrofitting it usually means rebuilding.
Our Role
Kainskep built the AI layer and the deterministic layer it depends on, and operates both.
- AI engineering: agent design and multi step retrieval planning, Model Context Protocol server implementation and schema grounding, prompt versioning and evaluation loops
- Deterministic engineering: the rule engine, metric calculation in code, parser led extraction, and the hybrid architecture that decides what the model is and is not permitted to own
- Data engineering: the transformation layer and metric definitions the AI reads from, plus the schema and lineage knowledge that makes grounding possible
- Security and compliance: read only enforcement, identity inherited access, and enterprise terms preventing training on customer data
- Operations: token cost management, monitoring and observability reporting to leadership, fallback behavior and multi environment deployment
What Broke First
None of this was right the first time. The controls exist because specific things went wrong before them, and the list is more useful than a description of the finished design.
Message validation produced confident errors
A clinical messaging validation workload initially leaned on model judgment and hallucinated findings. It was rebuilt as a hybrid: a deterministic rule engine in Python does the validation, with model based judgment layered on top rather than underneath. The rules decide. The model assists.
Free text extraction invented structure
A workload assessing whether a case was ready for billing was moved to a parser led approach, so extraction follows a defined structure rather than the model producing whatever shape seems reasonable in the moment.
Summaries were only as good as the instructions behind them
Early reporting output was inconsistent about which data to pull and how to present it. Prompts were versioned and refined against clinical review, and formatting was standardized so leadership sees the same shape of report every time and can spot an anomaly by eye.
Some metrics were simply wrong
When a figure looked off, the cause was usually a data variant nobody had seen before rather than a model failure. Those were fixed in the deterministic layer, which is exactly where they should have been fixed.
Costs needed an owner
Token consumption is managed deliberately rather than discovered on an invoice, and monitoring of the AI workloads goes to leadership as a standing report rather than being assembled whenever somebody asks for it.
Where the Line Sits
Language models are good at language, interpretation and navigating ambiguity. They are unreliable at arithmetic, and they have no way of knowing whether a business rule they inferred is the one the organization actually uses. So the division of labor is enforced in the architecture, not in a prompt.
Deterministic code owns the numbers
Metric definitions, all arithmetic and aggregation, and the validation rules. A metric means what the organization says it means, not what a model infers this time. Numbers are computed, never generated, and an established rule is applied identically every time.
- Metric definitions
- Arithmetic
- Validation rules
The model owns the language
Interpreting the question, navigating a schema too large to fit in a prompt, planning multi step retrieval, and explaining the answer. Ambiguity resolution is the actual value being added, and it is where models genuinely outperform a static query.
- Interpretation
- Schema navigation
- Explanation
Reports are computed, then narrated
Daily and weekly summaries go to hospital leadership as narrative reports. Every number in them is calculated in code and handed to the model, which writes the interpretation. It is never asked to work anything out, because a model asked to work something out will sometimes get it wrong in a sentence that reads perfectly.
Grounded in the real schema
The agent reaches the database through Model Context Protocol servers rather than through prompt stuffing or fine tuning, so it works from the actual structure instead of a remembered approximation. This is the single most important control against fabricated table and column names. Ambiguous questions are decomposed into a retrieval strategy the way an experienced database administrator would work through an unclear request.
Read only, under the caller's own permissions
No generated statement can alter a record, and the connection operates under the requesting user's own database permissions. There is no shared privileged service account through which everyone quietly acquires the rights of the most privileged person in the organization. Standing one of those up is faster to build and creates a permanent, invisible privilege escalation path.
- Read only
- Identity inherited
- No shared service account
Two detection layers, one correction loop
Evaluation test cases with known correct answers run as a regression suite. A separate model judges whether a generated answer is correct, catching the class of failure that is fluent, well formed and wrong. A dashboard shows performance as a trend, and a failed evaluation raises an alert rather than sitting in a report nobody opens. Underneath that, expert review catches the subtle and the novel.
- Test cases
- Model based judging
- Alerting
Model based judging is an effective net for fluent nonsense and a poor substitute for domain expertise on genuinely ambiguous questions. It sits underneath expert review rather than replacing it, which is what makes the pair hold together.
What We Delivered
Two products, and the machinery that keeps them correct.
A conversational product
Clinical and management staff ask questions in natural language and get answers from the production database. In production at one site, used daily by a small group.
Narrative reporting products
Daily and weekly summaries to hospital leadership with downloadable versions. These run more widely than the conversational product.
A deterministic rule engine
Written in Python. It holds the validation rules and the metric calculations, and it is where a caught error goes to become permanent.
Model Context Protocol servers
The grounding interface between the agent and the schema, with schema documentation and logs held in dedicated storage rather than pasted into context. The platform runs on managed cloud AI infrastructure under an enterprise agreement where customer data is not used for model training.
An evaluation suite and dashboard
Known questions with known correct answers, executed against the system rather than reasoned about in a design document, with failures alerting the team so a regression is treated like a broken build.
Operational reporting on the AI workloads
Token cost and observability reported to leadership as a standing report rather than assembled on request.
One useful side effect emerged that was not the original goal. Because the system has to comprehend the procedural logic in order to answer questions about it, it also surfaces dependencies and gaps inside that legacy code. A tool built to serve clinicians turned out to be useful to the engineers modernizing the system underneath them.
Outcomes
Accuracy here is not a fixed property of the model. Errors are usually caught by clinicians, including hospital users who know immediately when a figure does not match what they saw on the floor. What happens next is the mechanism: a caught error is not simply corrected in the output, it is converted into a rule in the deterministic engine, so the same error cannot occur again, for anyone.
- Every number in a reported figure is computed in code rather than generated
- Accumulated correctness survives model upgrades, because it lives in code rather than in a prompt
- A prompt change, a model version change or a schema change is a testable event rather than an act of faith
- Degradation is visible as a trend rather than concluded after complaints
Most AI case studies claim accuracy. Very few describe a mechanism by which accuracy increases over time. If your domain experts are catching errors and those corrections are not becoming permanent rules somewhere in your system, you are paying for the same mistake repeatedly.
What This Page Does Not Cover
Published in anonymized form. Client identity is withheld, deployment scope is described as at the time of writing, and clinical outcomes at individual hospitals are held by those institutions rather than reported here. Three limits are worth stating precisely rather than implying away.
- Read only is an integrity control, not a disclosure control. The agent cannot alter data. It can read whatever the requesting user is entitled to read, which in a clinical database includes patient information. That access is not new, since the same person could always query the database directly
- What an engineer puts into a model context during debugging is governed by permissions, training and access review rather than by an automated egress control. Adding that control is future work, not something already in place
- The conversational product is in production at one site with a small group of daily users, while the reporting products run more widely. That difference matters when assessing what has been proven
- Model selection, prompt contents, and the specific rules held in the engine
- Client identity, its customers, and any volume or usage figure
Related Capabilities
The disciplines this engagement drew on, and where each is described in full.
AI Engineering
Agent design, schema grounding, evaluation loops, and the production engineering that decides what a model is structurally prevented from owning.
AI engineering for production systemsAI Strategy & Advisory
Drawing the boundary before anything is built: what the model will never own, how it will be grounded, and what evidence a validation function will need.
AI strategy and advisoryData Engineering & Machine Learning
The transformation layer and metric definitions the AI reads from, and the lineage knowledge that makes grounding possible at all.
Data engineering and machine learning