RAG for regulated industries: how to build a compliant AI knowledge assistant

Retrieval-augmented generation (RAG) is an architecture in which a large language model answers a question using passages retrieved from a trusted knowledge base, rather than relying only on what it learned during training. The technique was introduced by Lewis et al. in 2020 and has become the standard way to apply LLMs to private, fast-changing information.

A RAG system has two halves. Retrieval finds the most relevant passages for a question; generation writes an answer constrained to those passages and cites them. Improve either half and the answers improve.

• Traceability. Every answer cites the passage it came from, so a compliance officer can check it.
• Currency. Updating knowledge means re-indexing documents, not retraining a model.
• Access control. Retrieval can enforce the same permissions as the source systems, so users only see answers drawn from documents they may read.
• Data minimization. Confidential data stays in a searchable index rather than being baked into model weights — and removing a document from the index removes it from future answers.

1. 01Source registry. Each collection has an owner, a refresh schedule and an access policy.
2. 02Permission-aware ingestion. Documents are parsed with their structure — headings, tables, clause numbers — and every chunk carries the access rights of its source.
3. 03Hybrid retrieval and re-ranking. Keyword and vector search run together, and a re-ranker orders the best passages first.
4. 04Query-time permission filter. Results are filtered against the user's rights before the model sees them.
5. 05Grounded generation. The model answers only from retrieved passages, cites them, and declines when they don't contain the answer.
6. 06Guardrails. Personal-data redaction where needed, defences against prompt injection hidden in documents, and topic boundaries.
7. 07Audit logging. Questions, retrieved sources, answers and user feedback are logged for review.
8. 08Evaluation and monitoring. A fixed test set is re-run on every change, and production quality is tracked continuously.

When documents contain personal data, GDPR applies to the RAG system like any other processing: you need a lawful basis, data minimization and appropriate security. A model provider processing data on your behalf needs a processor agreement under Article 28, and transfers outside the European Economic Area require a valid transfer mechanism under Chapter V.

In practice, regulated firms keep the index and the application inside their own cloud environment in an EU region, use enterprise model endpoints configured so prompts and documents are not used for training, or self-host open-weight models when data must never leave their infrastructure.

For banks, payment and e-money institutions, investment firms and other financial entities, DORA has applied since 17 January 2025. A cloud platform, a model API and an AI development partner are all ICT third-party service providers. Each contract must be recorded in the register of information, the provider must be assessed before contracting, and the contract must include the clauses set out in Articles 28–30, covering data location, access and return, audit, incident assistance and exit.

Concentration risk matters too. In November 2025 the European Supervisory Authorities designated the first 19 critical ICT third-party providers, including AWS, Google Cloud and Microsoft. An exit strategy — for example, the ability to swap the model behind the assistant — is part of a resilient design.

Usually not. An internal assistant that answers staff questions from policies and procedures is not listed in Annex III. It becomes high-risk if it is used for an Annex III purpose — for example, evaluating the creditworthiness of individuals (point 5(b)) or making decisions about recruitment, promotion or termination of employees (point 4) — and those obligations apply from 2 December 2027.

Transparency obligations under Article 50 apply from 2 August 2026: people interacting with an AI system must be informed that they are doing so, unless it is obvious. For customer-facing assistants, that means a clear disclosure in the interface.

Build a test set of real questions with expert-approved answers before you build the assistant, and score every version against it:

Organizations in Cyprus and Greece often hold records in both Greek and English. Multilingual embedding models let a question in one language retrieve passages written in the other, and the test set should include questions in both languages.

For self-hosted deployments, open Greek-language models are now available: Meltemi, released by the Institute for Language and Speech Processing (ILSP) of the Athena Research Center in 2024 as the first open Greek LLM, and Llama-Krikri, announced by ILSP in February 2025.

• Stale index. Documents change but the index doesn't. Fix: scheduled re-indexing and a freshness date on every source.
• Broken tables and clauses. Naive chunking splits a table or a clause from its heading. Fix: structure-aware parsing.
• Permission leaks. The assistant answers from documents the user may not see. Fix: permissions on every chunk and filtering before generation.
• Over-confident answers. The model fills gaps from general knowledge. Fix: grounding instructions, citations and refusal tests.
• Prompt injection. A document contains instructions aimed at the model. Fix: treat retrieved text as data, and test with adversarial documents.
• No content owner. Nobody is accountable for source quality. Fix: name an owner for every collection.

Keap Technologies builds RAG and LLM applications for banks, fintechs and law firms in Cyprus, across the EU and in the US. Tell us about your use case.