AI Financial Risk Analyst

The problem

A plain LLM asked about a company's risks and mitigations will hallucinate plausible-sounding answers, and a plain RAG system can retrieve the right text but cannot reliably connect a specific risk to its specific mitigation across a long document. The goal was a system that retrieves the right context, reasons over structured relationships between risks and mitigations, and says so honestly when a connection genuinely is not in the source material.

My approach

• Architecture: a planner agent decides per-question whether to route to RAG only, graph only, or both, rather than always running the full hybrid pipeline. Not every question needs a graph traversal, and running one anyway would add latency for no benefit.
• Technical decision: RAG started as a single-pass retrieve-and-answer pipeline, then moved to a map-reduce pattern. The open-source model used, flan-t5-base, has a 512-token limit and cannot reason well over several long retrieved chunks at once. Map-reduce summarizes each chunk individually first, then combines the summaries into a final answer, which works around the model's own context limitation instead of fighting it.
• Trade-off: entity extraction from unstructured filings is messy by nature. The same company came back from the LLM as 'GOOGL', 'Alphabet Inc.', and 'google (goog)' across different chunks. Left as-is, the knowledge graph would silently fragment into duplicate company nodes. A normalization map resolves these to one canonical node before insertion.
• Similar problem, risks and mitigations extracted as free text produce near-duplicate phrasings, 'cyber attack risk' and 'cybersecurity risk' read as different strings to a graph database. Sentence embeddings plus agglomerative clustering group these before they become separate nodes, so the graph reflects actual distinct risks rather than wording variance.
• Guardrails: prompts explicitly forbid inventing a mitigation that is not in the source, and define what to do when no direct mitigation is found, infer carefully from context and say so, rather than returning a dead-end 'not found' or fabricating one to sound complete.
• Observability: every agent decision, RAG call, and graph query is traced through LangSmith, so a wrong answer can be traced back to which agent and which retrieval step produced it, not just re-run and hoped to fix itself.

Results

• Answer relevance improved by 49% and groundedness by 5.4% over baseline RAG, while context relevance stayed constant since both systems share the same retrieval base, the gain came specifically from structured graph reasoning, not from retrieving different chunks.
• Reduced end-to-end response latency from roughly 80 seconds to 30 seconds, a 62.5% reduction, through reduced redundant LLM calls and more efficient pipeline orchestration.
• The largest real improvement was structural rather than metric-based: risks are explicitly categorized and linked to their mitigations, instead of producing a flatter, less explainable answer the way baseline RAG does.

What I would do differently

The system currently runs on a local LLaMA 3 model through Ollama, which is not directly deployable to the cloud as-is. That was a reasonable choice for local development and keeping the project free to run, but it means the current version cannot be handed a public URL without first swapping in a hosted model. Next time, I would build the LLM call behind an interface from day one, so switching between a local and a hosted model is a config change, not a rewrite.