Chat with Documents with Improved
Response Accuracy
Inside the page
Chatting with documents using natural language is one of the most sought-after use cases of LLMs by enterprises. A cute short-name for this problem is RAG — Retrieval Augmented Generation. However, RAG lacks transparency in revealing what it retrieves, making it uncertain which questions the system will handle well. This results in valuable information getting lost amid a mass of irrelevant text, which isn’t ideal for a production-grade application.
The Problem with Traditional Invoice Processing
After almost a year of building with LLMs, I’ve learned many techniques to improve RAG performance and summarized some of my lessons here. In this section, I’ll go over a few tested techniques to improve RAG performance:
- Adding additional info in the header or footer of the chunk
- Adding metadata in each chunk
- Adding summarized info in each chunk
- Using Langchain’s “Parent Document Retrieval” with two sets of chunk sizes
These inefficiencies don’t just impact productivity they increase the risk of late payments, duplicate entries, and strained vendor relationships.
The Pre-Retrieval Step
Despite tremendous recent interest in applying NLP to a wider range of real-world applications, most NLP papers, tasks, and pipelines assume raw, clean text. However, many of the documents we encounter in the wild aren’t so clean — many are visually structured documents (VSDs) such as PDFs. Conventional preprocessing tools for VSDs mainly focus on word segmentation and coarse layout analysis. PDFs are versatile and preserve the visual integrity of a document, but they often pose a significant challenge when it comes to extracting and manipulating their contents.
Clean, Accurate Extraction
We’ve all heard “garbage in, garbage out.” I think it applies to RAG too, but many people skip past this step and focus on optimizing everything after it — even though it’s the most crucial one. You can’t simply extract text from your documents, drop it into a vector database, and expect reliable, accurate answers. Extraction of text and tables from documents has to be semantically accurate and coherent.
Here’s an example from my own experience: I had 10 resumes from different candidates. Each candidate’s name appeared at the beginning of their resume, but the rest of the pages (each resume was about 2 pages long) never mentioned the name again. In cases like this, chunks can lose that information once a resume is split up by chunk size. One easy fix is to add that info — like the candidate’s name — to each chunk as a header or footer.
Unlike rule-based systems, InfoMiner learns and adapts to new formats over time—no rigid templates or manual configuration required.
Chunk Optimization
The second technique is chunk optimization. Depending on your downstream task, you need to determine the optimal chunk length and how much overlap each chunk should have. If a chunk is too small, it may not include all the information the LLM needs to answer the query; if it’s too big, it may contain too much irrelevant information, which reduces vector search accuracy, confuses the LLM, and can sometimes be too large to fit into the context window.
From my own experience, you don’t have to stick to one chunk-optimization method for every step in your pipeline. For example, if your pipeline involves both high-level tasks like summarization and low-level tasks like generating code from a function definition, you could use a bigger chunk size for summarization and smaller chunks for coding reference.
This enables your team to resolve issues before approval, minimizing errors and delays.
Seamless Workflow Integration
InfoMiner integrates effortlessly with your existing systems—ERP, accounting software, or workflow tools—so your team can work smarter using familiar platforms, while AI handles the heavy lifting.
Matching Search Type to Query
When a query requires the LLM to search across many documents and return a list of them as the answer, it’s better to use the similarity_search_with_score search type.
When a query requires the LLM to perform a multi-step search to arrive at an answer, you can add the instruction “think step by step” to the prompt. This helps the model break the query down into multiple sub-queries.
Improving Generation Quality
After you retrieve the relevant chunks from your database, there are still techniques you can use to improve generation quality. You can use one or several of the following, depending on the nature of your task and the format of your text chunks.
If your task depends heavily on one specific chunk, a commonly used technique is reranking or scoring. As mentioned earlier, a high score in vector similarity search doesn’t always mean the highest relevance. It’s worth running a second round of reranking or scoring to pick out the chunks that are actually useful for generating the answer. You can ask the LLM to rank the documents by relevance, or use other methods like keyword frequency or metadata matching to refine the selection before passing the documents to the LLM for a final answer.
Balancing Quality & Latency
There are a few other tips I’ve found useful for balancing generation quality and latency. In production, your users may not want to wait for a multi-step RAG process to finish — especially when it involves a chain of LLM calls. The following choices can help improve the latency of your RAG pipeline.
The first is to use a smaller, faster model for some steps. You don’t necessarily need the most powerful model (often the slowest) for every step in the RAG process. For example, for simple query rewriting, generating hypothetical documents, or summarizing a text chunk, a faster model — like a 7B or 13B local model — is often good enough. Some of these smaller models may even be capable of generating a high-quality final output for the user.
Stop processing documents.
Start automating them
Get your first 500 documents processed free. No credit card, no IT team required.
Free trial · 500 documents · No templates needed · Live in under 4 hours