Bridging Natural Language and Interactive What-If Interfaces via LLM-Generated Declarative Specification
An LLM-powered workflow turns plain-English 'what-if' questions into interactive dashboards — but only gets it right 80% of the time after automated repair.
The Thesis
What-if analysis — the practice of asking 'what happens to revenue if we raise prices by 10%?' — is central to business decision-making but painful to set up in today's tools. This paper proposes a two-stage system: a large language model (LLM) first translates a natural language question into a structured specification written in a purpose-built language called PSL (Praxa Specification Language), and then that specification is automatically compiled into an interactive visual interface with sliders, filters, and linked charts. The catch is reliability: even with automated error detection and repair using few-shot prompting (showing the model examples of correct outputs), only 80% of specifications are generated correctly, and the remaining 20% can silently produce dashboards that look plausible but answer the wrong question. The intermediate specification layer — rather than going directly from language to UI — is the paper's core architectural bet, because it creates a place where errors can be caught before they reach the user.
Catalyst
LLMs capable of generating structured, domain-specific code reliably enough to be useful in production have only become available in the past two years, making the 'NL to specification' step feasible at all. At the same time, the failure modes of pure chatbot-style data analysis (hallucinated answers, context drift in long conversations) have become well-documented problems that enterprise buyers are actively trying to solve. The authors' benchmark of 405 questions across 11 analysis types provides a more rigorous empirical foundation than most prior work in this space, giving the approach a credibility floor.
What's New
Prior approaches split into two camps: traditional BI tools like Tableau or Power BI require analysts to manually wire up parameters and visualizations — a process that can take hours or days — while LLM chatbot interfaces like those built on GPT-4 generate answers conversationally but degrade in accuracy as the conversation grows longer and context accumulates. This paper inserts an intermediate representation (the PSL specification) between the natural language input and the rendered interface, which is the key structural difference. That layer enables automated validation and repair — something neither pure chatbot nor manual BI workflows support — and the authors show it improves success rates from 52% to 80% on their benchmark.
The Counter
An 80% success rate after automated repair sounds promising until you consider that 1-in-5 dashboards is either broken or silently wrong — and enterprise users often cannot tell which. The paper's benchmark covers 405 questions across 11 analysis types, but these are curated by the researchers, not drawn from real analyst workflows where questions are messier, domain jargon is heavier, and the stakes of a misleading chart are higher. The PSL specification language is described as Praxa's proprietary format, which means the entire approach depends on adoption of a non-standard intermediate language that has no existing ecosystem. More fundamentally, the paper does not show a user study: we don't know whether the 80% of correctly generated interfaces are actually faster, less error-prone, or more trusted than a skilled analyst using Tableau directly. The comparison to 'fragile chatbot interfaces' is used as motivation, but no head-to-head usability comparison is presented. Finally, as frontier LLMs improve, the direct NL-to-UI approaches the paper critiques will also improve, potentially closing the reliability gap without requiring a new specification language.
Longs
- CRM — Salesforce embeds AI-driven analytics directly into enterprise workflows where what-if analysis is a core use case
- DDOG — Datadog's observability platform increasingly adds scenario-based analysis features
- MSFT — Power BI and Copilot integration is the most direct incumbent surface area for this technology
- TLND (Qlik, private) — data integration vendors building natural-language analytics layers
- BOTZ (robotics/automation ETF) — indirect exposure to AI-driven workflow automation broadly
Shorts
- Tableau (Salesforce) — its manual drag-and-drop parameter setup is directly the friction this paper targets; an automated alternative erodes its core differentiation for scenario analysis
- Microsoft Power BI — same dynamic; Copilot integration helps but the chatbot approach has the fragility problems the paper documents
- ThoughtSpot — search-based analytics whose NL-to-query approach bypasses specification validation and may inherit the same silent-error failure modes
Enablers (Picks & Shovels)
- Anthropic Claude and OpenAI GPT-4 — the state-of-the-art LLMs benchmarked in the paper for specification generation
- Few-shot prompting frameworks (LangChain, DSPy) — infrastructure for the targeted repair stage
- Vega-Lite and similar declarative visualization grammars — prior art that PSL builds on conceptually
- Apache Arrow / DuckDB — fast in-browser query engines that make client-side compiled interfaces feasible
Private Watchlist
- Hex Technologies — collaborative data notebook platform building NL-to-analysis features
- Sigma Computing — cloud BI tool actively investing in natural language query interfaces
- Praxa — the company behind the PSL specification language described in the paper
- Equals — spreadsheet-database hybrid with a stated focus on reducing manual dashboard setup
Resources
The Paper
What-if analysis (WIA) is an iterative, multi-step process where users explore and compare hypothetical scenarios by adjusting parameters, applying constraints, and scoping data through interactive interfaces. Current tools fall short of supporting effective interactive WIA: spreadsheet and BI tools require time-consuming and laborious setup, while LLM-based chatbot interfaces are semantically fragile, frequently misinterpret intent, and produce inconsistent results as conversations progress. To address these limitations, we present a two-stage workflow that translates natural language (NL) WIA questions into interactive visual interfaces via an intermediate representation, powered by the Praxa Specification Language (PSL): first, LLMs generate PSL specifications from NL questions capturing analytical intent and logic, enabling validation and repair of erroneous specifications; and second, the specifications are compiled into interactive visual interfaces with parameter controls and linked visualizations. We benchmark this workflow with 405 WIA questions spanning 11 WIA types, 5 datasets, and 3 state-of-the-art LLMs. The results show that across models, half of specifications (52.42%) are generated correctly without intervention. We perform an analysis of the failure cases and derive an error taxonomy spanning non-functional errors (specifications fail to compile) and functional errors (specifications compile but misrepresent intent). Based on the taxonomy, we apply targeted repairs on the failure cases using few-shot prompts and improve the success rate to 80.42%. Finally, we show how undetected functional errors propagate through compilation into plausible but misleading interfaces, demonstrating that the intermediate specification is critical for reliably bridging NL and interactive WIA interface in LLM-powered WIA systems.