Private AI for Law Firms:Keep Privilege Intact

Generative AI has become indispensable for document review, contract analysis and legal research. But for a law firm, the convenience of pasting a privileged document into a public chatbot carries a specific, under-appreciated risk: it can be construed as disclosure to a third party — and a court may treat that disclosure as a waiver of attorney-client privilege.

This guide explains the privilege problem in concrete terms, then walks through the architecture that lets a firm use every leading model — Claude, GPT, Gemini, and self-hosted open models — without ever letting client data leave infrastructure the firm controls.

The RiskWhy public AI quietly waives privilege

Attorney-client privilege protects confidential communications made for the purpose of obtaining legal advice. The doctrine is fragile: privilege can be waived by voluntary disclosure to anyone outside the privileged relationship. When a lawyer pastes a confidential memo into a consumer AI tool, the document is transmitted to, processed by, and frequently retained on a vendor's servers.

• The document leaves the firm's custody and control.
• The vendor — a third party — now holds the privileged content.
• Default consumer terms may permit the provider to retain inputs or use them to improve models.
• There is typically no audit record of exactly what the model saw.

DefinitionsWhat "private AI" actually means

"Private AI" is an overloaded term. Vendors use it to mean very different things, and the differences matter enormously for privilege. Three models are common:

A defensible firm posture combines both ends of this table: open-weight models (Llama, Mistral, Qwen) self-hosted for the most sensitive work, and closed APIs reached only over isolated egress with contractual zero-retention terms for everything else.

ArchitectureA reference architecture for privilege-grade AI

The goal is simple to state and harder to engineer: the model can answer questions over the firm's documents, but the documents never leave a boundary the firm controls, and every interaction is logged. A practical pipeline looks like this:

1. Ingestion — pleadings, contracts and case files are indexed into a private vector store (e.g. Qdrant + PostgreSQL) running inside the firm's environment, encrypted at rest with keys the firm holds.
2. Authorisation — every query passes a zero-trust gateway that resolves the user, their role, and the specific matter they are cleared to see before any retrieval runs.
3. Retrieval — vector search returns only document chunks that pass row-level permission checks scoped to client and matter, so one client's files can never surface in another's query.
4. Sanitisation — names, IDs and contact details are masked at the chunk level before any context is assembled for a model.
5. Inference — open models run on GPUs in the firm's region; closed APIs are reached over isolated egress with zero-retention terms.
6. Audit — every query, document touch and token is written to a tamper-evident log with a content hash, producing the defensible record regulators and opposing counsel expect.

EconomicsIs private AI actually affordable for a 30-lawyer firm?

The common objection is cost. In practice, a self-hosted deployment for a small-to-mid firm is far cheaper than most partners assume. As a public reference point, a documented engagement to self-host a 70B-parameter open model for a law firm (Llama 3 70B on vLLM with a vector store) ran at roughly $1,200/month in hosting on top of a one-time setup. The setup, not the compute, is where the value and the risk concentrate.

That is the gap our compliance offering is built for: a managed setup plus ongoing advisory, where the firm pays its cloud provider directly and we never take custody of regulated data.

ChecklistA privilege-preservation checklist

• Inference for privileged matters never leaves infrastructure the firm controls.
• Encryption keys are held by the firm, not the vendor.
• Matter-level access controls are enforced in the retrieval layer, not just in the UI.
• Every AI interaction is written to an immutable, tamper-evident audit log.
• A signed DPA (and a BAA where health data is involved) is in place with any processor.
• Closed-API usage is governed by contractual zero-retention and no-training terms.
• Staff have a clear, written policy on which tools are approved for privileged work.