Converse with Data Using NLP / LLM
The Holy Grail!
Inside the page
Industry analysts have been predicting that “the future of BI is conversational” for quite some time now. Giant leaps made recently by the various LLM models reinvigorated that quest. Yet production-grade conversational BI solutions are very hard to engineer. Business users are still looking for insights in BI dashboards, and data analysts are still hand-writing SQL queries against their databases to answer ad-hoc business questions.
Why Isn't Conversational BI Here Yet?
The vast majority of enterprise data is still in structured data stores and accessible mainly through SQL queries. For any conversational BI solution there has to be an engine that will translate a user’s natural language question into a valid SQL or a Pandas dataframe formula.
Engineers have tried to build “Natural Language to SQL” (NL2SQL) engines since the 70s using rules-based techniques, which would very quickly get too complex to be useful. But with the advancement of transformers that have enabled tools like GitHub Copilot, OpenAI Code Interpreter, or Langchain’s pandas_dataframe_agent and SQL Agent, it would seem this should be a trivial problem to solve. It is not.
Two Ways to Build an LLM-Based NL2SQL Engine
- Fine-tuning your own LLM — This approach requires taking an existing LLM and training it further using NL<>SQL pairs relating to the company’s structured data. A couple of challenges here are that a) coming up with the training dataset is hard and expensive, and b) the most powerful LLM model around (GPT-4) cannot be fine-tuned (as of this writing).
- Leveraging in-context learning — The latest LLM models (like GPT-4-32K) can write SQL quite well out of the box and have enough context window for quite a bit of few-shot training, and for an agent to try to recover from errors by performing follow-ups using chain-of-thought techniques. The idea here is to build an LLM agent on top of GPT-4 that can implement NL2SQL with few-shot learning.
The Challenges of an In-Context Learning Solution
So what are the challenges of deploying solution #2? Here are seven we have encountered:
- Table and column descriptions — Even the best data teams often do not have clear documentation about tables, columns, and metadata. With the rise of ELT, where data is simply dumped in the warehouse from various sources and transformed on query, the situation becomes even worse. The table and column names might be the only info available to the engine at configuration time.
- Missing context and metadata — There are often business definitions which live in a data analyst’s head and are not in the underlying data. In a real-world home rental marketplace we encountered, what constitutes an “active listing” is a combination of WHERE clauses that differ based on the value of another column specifying the building type. In rare cases these live as Views on the table, but more often they’re just stored in a query in the BI tool or dashboard.
- Incomplete info in the question, lack of “common sense” — Take “what was the average rent in Los Angeles in May 2023?” A reasonable human receiving this question would simply assume it’s about Los Angeles, CA, or would confirm with the asker in a follow-up. An LLM, however, usually translates this into a query filtering only on city and date, pulling in data for both Los Angeles, CA and Los Angeles, TX without a column to tell the two apart.
- Speed — For the engine to be “conversational,” response times must be fast (sub-30s). This is often very hard to achieve, especially if the agent tries to recover from errors or evaluate generated responses with subsequent LLM calls.
- Complex queries — While GPT-4 writes simple SQL queries very well, it can often stumble on complex queries that require aggregations and joins. This is exacerbated when a column name contains an action that can be done in SQL (for example, “Average” or “SUM”), and in join operations on data warehouses where foreign keys aren’t clearly enforced like they are in production databases.
- Privacy and data leaking — Many organizations do not want their database data or schema being sent to companies like OpenAI, since it can leak into their training corpus.
- Validation — There is no known way to identify cases where the system returns a syntactically valid but incorrect SQL query — for example, if the user asks for an “average” value and the system runs an AVG instead of picking a column literally called
average_price.
The Road Ahead
So is enterprise conversational BI impossible in 2024? Will there be a few more years of academic papers and company AI hackathon projects before a solution can be deployed in production? We don’t think so.
Our Take
While the challenges are definitely real, we believe that with the right tool, an enterprise data team can deploy solutions to enable business users to self-serve ad-hoc data questions from the company data warehouse. In the coming weeks we will be releasing a number of open source and hosted tools to address this.
Stop processing documents.
Start automating them
Get your first 500 documents processed free. No credit card, no IT team required.
Free trial · 500 documents · No templates needed · Live in under 4 hours