Choosing a local model for a BFSI deployment
Licensing, not benchmark score, is usually what kills a model in a bank's risk committee. Here's how to pick one that survives legal review.
Curated by Kailas Lovlekar · July 2026. AI models, hardware and pricing are evolving at a rapid pace — information in this article is for broad reference only.
Every bank evaluating on-prem agentic AI eventually asks the same question: which model do we actually run? The instinct is to start with a leaderboard — MMLU score, coding benchmarks, context window. That's the wrong starting point for a regulated institution. The first filter that determines whether a model is usable at all is its license, followed by whether its weight format is safe to load, and only then does raw capability matter.
This guide covers how PhantomOps' engineering team screens open-weight models before they're approved for a BFSI deployment, grouped the way a bank's legal and infrastructure teams actually think about them.
1. Start with the license, not the leaderboard
Open-weight models fall into two practical buckets for a commercial deployment:
Permissive licenses (Apache 2.0 / MIT) — the easiest legal sign-off
These carry no usage caps, no royalty terms, and no requirement to report back to the model publisher. For a bank's legal team, this is close to a rubber stamp.
| Model family | License | Typical sizes | Where it fits |
|---|---|---|---|
| Qwen 2.5 / 3 | Apache 2.0 — except the 3B and 72B variants, which ship under Qwen's own research license. Check the specific size before assuming Apache 2.0. | 7B–32B (Apache), plus 3B/72B (custom) | Structured data extraction, financial math, SQL/code generation |
| Mistral Small / NeMo | Apache 2.0 | 8B–24B | Enterprise RAG, document summarisation, multilingual compliance text |
| Microsoft Phi-4 / Phi-4-mini | MIT | 3.8B–14B | High-throughput micro-tasks, classification, desktop co-pilots |
Custom open-weight licenses — require a legal review cycle
These models are frequently state-of-the-art, but their license text includes bespoke terms — acceptable-use policies, commercial user caps, or attribution requirements — that a bank's legal team has to read and sign off on individually. Budget for this as a real line item in your deployment timeline, not a formality.
| Model family | License | Typical sizes | Where it fits |
|---|---|---|---|
| Meta Llama 3.x / 4 | Meta Llama Community License | 8B, 70B, 405B | General enterprise assistants, RAG over filings/prospectuses, agentic workflows |
| Google Gemma 2 / 3 | Gemma Terms of Use | 2B, 9B, 27B | Single-GPU workstation deployments, localised team co-pilots |
Neither list is static — publishers change license terms between releases (Mistral Small moved to Apache 2.0 in a recent release cycle after starting more restrictive). Re-verify the license on the exact model card and exact parameter size you intend to deploy at every procurement cycle, not just once at initial evaluation.
2. Then match capability to the actual workload
Once a shortlist clears legal, the choice comes down to what the agent is actually doing:
- Structured extraction & financial math (trade confirmation parsing, credit risk processing) — Qwen 2.5/3 consistently leads open models on math and coding benchmarks, with strong structured JSON output.
- Retrieval-augmented generation over long documents (compliance files, prospectuses) — Mistral Small/NeMo and Llama 3.x offer efficient long-context handling and clean safetensors weights.
- Low-latency, high-volume micro-tasks (classification, KYC field checks, desktop tools) — Phi-4-mini runs comfortably on modest hardware without sacrificing much reasoning quality for its size.
- Deep quantitative reasoning (portfolio modelling, multi-step risk simulation) — reasoning-focused MoE architectures such as DeepSeek-R1/V3 outperform general chat models, at the cost of a much larger hardware footprint (see Part 3).
3. Don't conflate model size with capability window
It's worth separating two numbers that get conflated in vendor conversations: parameter count (the size of the model, driving hardware cost) and context window (how much text it can read in one request, measured in tokens, not parameters). A smaller model with a long context window can still process an entire loan file; a larger model with a short context window can't. Size your hardware around parameters (Part 3), but size your document workflows around context window — check both independently rather than assuming one implies the other.
Where PhantomOps lands
For most BFSI deployments we start customers on Qwen 2.5/3 or Mistral Small in safetensors format, served through vLLM — permissively licensed, no remote-code execution risk (more on that in Part 2), and strong out-of-the-box performance on financial document workloads. We move to larger Llama or DeepSeek deployments only once a specific workload justifies the extra hardware cost and the license review is already cleared.
Sources: Qwen2.5 release notes and per-model license files (Hugging Face); Mistral model cards; Microsoft Phi-4 model card (MIT license).