Architecture Matters More Than Scale: A Comparative Study of Retrieval and Memory Augmentation for Financial QA Under SME Compute Constraints
For small businesses doing financial Q&A, how you structure an AI's memory matters more than model size — at least on two standard benchmarks.
The Thesis
This paper tests four ways to wire up a small, locally-run language model for financial question-answering, targeting small and medium-sized enterprises (SMEs) that cannot afford cloud GPU bills or large API budgets. The central finding is an 'architectural inversion': structured memory — storing key facts in an explicit, queryable format — wins on precise numerical tasks, while retrieval-augmented generation (RAG), which pulls relevant documents at query time, wins on conversational, context-heavy tasks. Neither approach dominates across all task types, which is itself a useful empirical result. The proposed fix is a hybrid system that routes queries to the right architecture depending on task type. The catch is that everything runs on a single 8-billion-parameter model and two benchmarks, so generalization is uncertain.
Catalyst
Instruction-tuned 8B models capable of useful financial reasoning only became widely available in 2023–2024, making local deployment on SME-grade hardware newly plausible. At the same time, the FinQA and ConvFinQA benchmarks — the two evaluation sets used here — have become standard enough that results can be meaningfully compared across papers. The combination of affordable small models and established benchmarks created the conditions for this kind of controlled architectural comparison.
What's New
Prior financial LLM research largely focused on large, cloud-hosted models or on proprietary fine-tuning, making results irrelevant for organizations without significant infrastructure. Earlier comparisons of RAG versus memory architectures — such as work on MemGPT and various RAG pipelines — were not conducted under explicit compute constraints, and few studies used both a deterministic numerical benchmark (FinQA) and a conversational benchmark (ConvFinQA) in the same experiment. This paper's contribution is isolating architectural choice as the variable while holding model size and infrastructure constant, producing a more actionable comparison for resource-constrained deployers.
The Counter
This paper runs one model on two benchmarks and calls it a comparative study. The 'architectural inversion' finding — interesting as it is — rests on a sample size of exactly two task types, which is not enough to support a general deployment framework. The benchmarks used, FinQA and ConvFinQA, have been around long enough that any reasonably careful implementation will look good on them; they may not reflect the messy, poorly-formatted financial documents that SMEs actually work with. More importantly, the paper's headline contribution, the hybrid routing framework, is never actually tested. We are asked to trust that dynamically switching between architectures will outperform either alone, but there is no experiment showing this. The SME framing is also somewhat cosmetic: the constraints imposed (no cloud GPU, 8B parameter limit) are reasonable, but the paper does not evaluate total cost of ownership, latency, or the human labor needed to maintain structured memory schemas — the real friction points for small businesses. A practitioner reading this paper would still need to run their own evaluation before deploying anything.
Longs
None listed.
Shorts
- Cloud-based financial AI API vendors — if SMEs can achieve comparable accuracy locally, the case for paying per-token API fees weakens.
- One-size-fits-all RAG platform vendors — the paper's finding that RAG underperforms memory on numerical tasks challenges marketing claims that RAG is sufficient for financial workflows.
Enablers (Picks & Shovels)
- Meta's Llama model family (the 8B instruction-tuned models that make local deployment feasible)
- LangChain and LlamaIndex (open-source orchestration frameworks for building RAG and memory pipelines)
- FinQA and ConvFinQA benchmark datasets (the evaluation infrastructure this paper depends on)
- FAISS and Qdrant vector stores (the retrieval backends typically used in RAG pipelines like this)
Private Watchlist
- Unstructured.io (document parsing infrastructure for RAG pipelines)
- Vectara (enterprise RAG platform with compliance focus)
- Qdrant (open-core vector database used in local RAG deployments)
The Paper
The rapid adoption of artificial intelligence (AI) and large language models (LLMs) is transforming financial analytics by enabling natural language interfaces for reporting, decision support, and automated reasoning. However, limited empirical understanding exists regarding how different LLM-based reasoning architectures perform across realistic financial workflows, particularly under the cost, accuracy, and compliance constraints faced by small and medium-sized enterprises (SMEs). SMEs typically operate within severe infrastructure constraints, lacking cloud GPU budgets, dedicated AI teams, and API-scale inference capacity, making architectural efficiency a first-class concern. To ensure practical relevance, we introduce an explicit SME-constrained evaluation setting in which all experiments are conducted using a locally hosted 8B-parameter instruction-tuned model without cloud-scale infrastructure. This design isolates the impact of architectural choices within a realistic deployment environment. We systematically compare four reasoning architectures: baseline LLM, retrieval-augmented generation (RAG), structured long-term memory, and memory-augmented conversational reasoning across both FinQA and ConvFinQA benchmarks. Results reveal a consistent architectural inversion: structured memory improves precision in deterministic, operand-explicit tasks, while retrieval-based approaches outperform memory-centric methods in conversational, reference-implicit settings. Based on these findings, we propose a hybrid deployment framework that dynamically selects reasoning strategies to balance numerical accuracy, auditability, and infrastructure efficiency, providing a practical pathway for financial AI adoption in resource-constrained environments.