TL;DR

AI bills are rising even as the price per token is going down: Running AI models, known as inference, is on track to become the biggest chunk of a bank’s AI budget. Banks pay this cost every time a model reads a document, answers a client, or checks a transaction. Better chips, leaner models, and smarter software keep pushing the unit price of that work down. Yet total bills are still climbing because usage is growing much faster than prices are falling.

Architecture is now a margin decision: No single provider or set up works for every banking task. Banks will inevitably run hybrid infrastructure, mixing frontier model APIs, cloud services, and their own hardware. Because the exact same KYC review can cost $0.63 or $0.03 depending on the setup behind it, how a bank accesses inference is no longer just an IT choice. It is a financial one.

The real leverage sits in the optimisation layer: Investors have poured billions into companies selling raw AI capacity. We think the more durable opportunity, and the biggest lever for banks, lies in the independent software that makes every chip and agent do more for the same cost.

AI is eating venture capital, and fintech is paying for it

AI venture funding climbed from $137bn in 2022 to $273bn in 2025, and has already hit $496bn in the first three quarters of 2026. Fintech has gone the opposite way. Funding dropped from roughly $100bn in 2022 to around $55bn a year since, while deal counts fell from over 8,000 to about 2,500 so far this year. In Q3 2026 alone, AI raised $100bn against fintech’s $11bn, a 9-to-1 ratio that stood at just 1.3-to-1 in early 2022.

That capital is also heavily concentrated. In Q3 2026, 85% of AI funding went into rounds of $100m or more, with OpenAI, Anthropic, and xAI taking roughly half of all AI venture dollars this year. At first glance, those numbers look frothy, and parts of the market probably are. Look at the revenue growth, though, and you can see what is driving the appetite: the fastest-growing AI companies now hit $100min annual revenue in about 18 months (if not less), compared to roughly seven years for the top cloud companies of the last cycle.

Whatever you think of the valuations, the underlying adoption is real.This is a generational technology shift, and for a bank, sitting on the sidelines is not an option. The practical question is what it will cost to run AI at scale once these tools move from pilot projects into daily production.

Running AI is becoming the bigger cost

That day-to-day cost sits in inference, which is the work of running a model every time a workflow triggers it. Global inference demand is projected by McKinsey to jump from 21 GW of data center capacity in 2025 to 93 GW by 2030, overtaking both training and traditional computing to become the largest data center workload by 2028. For a bank, once an AI workflow goes live, inference becomes a recurring operating expense that scales with volume.

Cheaper tokens, bigger bills

Inference is priced in tokens (each roughly three-quarters of a word), and token prices have collapsed. Every new chip generation cranks out more tokens per hour, smaller models are matching larger ones, serving software is squeezing more out of each GPU, and providers are slashing prices to win market share. OpenAI’s flagship model dropped from $30 per million input tokens in March 2023 to $2 in September 2026.

Yet cheaper tokens haven’t led to smaller bills, because usage is outpacing the price drops:

•      Weekly tokens routed through OpenRouter jumped from 0.5 trillion in January 2025 to 126 trillion in September 2026.

•      Enterprise generative AI spend grew from $1.7bn in 2023 to $37bn in 2025, with Gartner projecting around $64bn this year on a different definition.

•      In a recent industry survey, 93% of enterprises admitted they spent more on AI than planned.

Every efficiency gain gets eaten up by higher volume. AI agents only speed this up, since a single user request can quietly trigger dozens of model calls in the background. AI costs won’t manage themselves; they are quickly becoming a major P&L line item that needs a clear architecture and a dedicated owner.

How a bank runs AI sets its margin

Take a standard KYC review. Say a bank handles 100,000 requests a year, and each one triggers ten model calls of around 25,000 tokens. That adds up to 25 billion tokens a year. Assuming 80% of reviews complete successfully, each completed review costs:

•      $0.63 via a public API

•      $0.28 on rented GPU capacity

•      $0.03 on the bank’s own hardware

The model and the workflow are identical, yet the cost varies by 20x.

Granted, these are modeled figures that leave out staffing, model risk, and hardware depreciation, so the real-world gap is narrower. Still, the underlying point holds. Banks need to track the cost per successful workflow, not the price per token. Once you multiply that KYC review across hundreds of automated workflows, from onboarding and trade surveillance to credit memos,your infrastructure setup stops being a back-office IT decision and starts dictating your operating margin.

The hardware bottleneck: scarce chips make software the lever

If owning hardware is so much cheaper, the obvious move would be to just buy more chips. The catch is that you can’t easily get them. The eight largest “neoclouds” (specialist GPU cloud providers) have about 3 GW of capacity live today against targets of nearly 36 GW by 2030. Meanwhile, the rental price of Nvidia’s H100 rose about 40%in just five months.

Those software gains are already showing up. On the exact same chip, better low-level code in an open-source serving engine recently delivered about 1.5x more output per GPU in Semi Analysis benchmarks. Better yet, the biggest gains show up in workloads that can afford to wait a few seconds. That happens to describe most back-office banking tasks, from document review to reconciliation and compliance checks.

More models, more chips, more complexity

Getting that efficiency is getting harder, though, because options are multiplying at both ends of the stack. On the model side, banks can now pick between Western closed models, Western open models, and Chinese open models,with the leader board for any given task reshuffling every few months. On the hardware side, Nvidia now competes with AMD, Google’s TPUs, and Amazon’s Trainium, each running its own software stack.

Where a model comes from is separate from where it runs; a Chinese open-weight model can run on-premises in the UK inside a bank’s own security perimeter. The catch is that every new model or chip requires re-tuning. If a bank doesn’t have a software layer to keep its workloads portable, it ends up paying for vendor lock-in twice: first in daily run-rate costs, and again in engineering time whenever it wants to switch.

The future bank will run on hybrid infrastructure

This leaves banks with several ways to buy inference rather than just one vendor. We count seven distinct ways to buy inference, ranging from calling a frontier lab’s public API to running models on racks the bank owns outright. Each option splits the operational burden differently between the vendor and the bank, trading speed-to-market against control:

•      Frontier APIs get you the best models fastest, but raise tough data residency questions.

•      Hyperscalers come pre-approved by procurement, but concentrate vendor risk.

•      Owned hardware gives full control, but demands heavy upfront capital and hard-to-find engineering talent.

•      Renting raw GPUs from neoclouds is technically an option, though not one banks use yet.

In our view, regulated work loads are steadily moving toward routes the bank directly controls: its own private cloud tenancies and its own hardware. And that is precisely where efficiency and portability matter most. Once a bank runs its own capacity, the bank, not the vendor, eats the cost of an idle chip or a painful migration.

Everyone wants to own inference

Suppliers have done the same math from the other side of the table. A megawatt of capacity rented out as bare metal earns about $13m a year; sold as finished AI output, that same megawatt pulls in $75m or more. Consequently, every vendor is marching toward the customer: specialist GPU clouds are building managed AI services, AI serving platforms are locking up physical capacity, and hyperscalers are doing all the above.

Venture investors have piled into the same trade. Inference startups raised $0.3bn in 2022, $5.2bn in 2025, and another $4.2bn through August 2026.The biggest checks this year went to serving platforms: Fireworks AI and Baseten raised $1.5bn each, Together AI raised $800m, and Modal brought in$750m.

Still, that fat margin isn’t guaranteed. Because an inference provider pays for raw chip hours but charges customers per token generated, its profit hinges entirely on how much output it can squeeze from each GPU. Inference demand may be booming, but raw efficiency decides who actually makes money, both for the providers selling the compute and the banks buying it.

Who is building the optimisation layer

That dynamic is why we spend most of our time looking one layer down. We track 66 independent companies across five technical layers, running from the metal up to the bank’s workflows:

•      Kernel & runtime software that makes models run faster on the exact same chip.

•      Compression tools that shrink models to cut memory and compute costs.

•      Serving engines that handle requests from thousands of users at once.

•      Fleet management software that keeps expensive clusters of chips busy.

•      Cross-hardware platforms that let banks run any model on any chip, privately.

Not all of them are chasing the same gains, though. Most software optimisation pays off best when a workload can wait a few seconds, but a growing group of these companies is solving the other side of the problem: workloads where latency is what matters, such as client-facing assistants, fraud checks at the point of payment or real-time trading support, where the answer has to arrive instantly rather than cheaply.

Most of these businesses are still at the Seed or Series A stage because late-stage capital has chased the big serving platforms. At the same time,major chip makers and cloud vendors have already acquired 15 companies like them since 2024, including high-profile buyouts like Run:ai and Neural Magic. For a bank, this consolidation cuts both ways: independent optimisation tools are what keep a hybrid strategy portable, yet every acquisition pulls a neutral tool back inside a single vendor’s walled garden.

Where do we go from here?

For financial institutions, the question is no longer whether to deployAI, but how to build a financially efficient hybrid architecture that doesn’t hand margin over to whoever sells the compute. In our conversations with banks, four roadblocks come up again and again:

•      Cost Visibility: Few banks can tell you their true cost per successful workflow; most just see a blunt monthly cloud invoice.

•      Portability: Every new model or chip still requires manual re-tuning, making switching slow and expensive.

•      Capacity: The deployment routes that give banks the most security and control are the exact places where GPUs are hardest to get.

•      Ownership: Most banks still haven’t decided which layers of the AI stack they need to own versus rent.

Optimisation software goes a long way toward fixing the first three problems, but it hits a natural ceiling. Every technical gain covered in this piece makes the same token cheaper to produce. None of it asks whether the bank should be paying for that token in the first place. Today, a bank reviewing a routine KYC file often rents a massive general-purpose model just to do one narrow, repetitive job.

The big serving platforms have noticed. Fireworks, Baseten, Together AI, and CoreWeave are all rolling out tools to help enterprises fine-tune and train custom models right alongside running them.

In F1 terms, you can build the best engine and design the cleanest aero, and still lose because you’re racing with someone else’s driver. That’s why the top teams develop their own talent. If the cheapest token is the one a smaller, specialized model never has to generate, the next phase of this race won’t be decided by who runs the model, but by who owns it.

If you are a financial institution figuring out how to run AI in production, a founder building the software that makes everyone else’s compute go further, oran investor looking at the same space, we’d love to connect, cperez-pla@illuminatefinancial.com.

Sources

CB Insights, State of AI Q3 2026 and State of Fintech Q3 2026; Crunchbase, financial services deal counts and AI round sizes (2026).

Bessemer Venture Partners, The State of AI 2025.

McKinsey & Company, The future of AIworkloads (Feb 2026).

OpenAIAPI pricing; OpenRouter rankings via N10 Assets;Menlo Ventures, 2025: The State of Generative AI in the Enterprise; Gartner AI models and platforms forecast,via IT Brief; McKinsey FinOps survey (May 2026).

Illuminate KYC framework; Lenovo TCO model cited by Evercore ISI(2026); Maybank Securities (Sep 2026); UBS (2026).

Franklin Templeton, Perspectives on Neoclouds (Oct 2026); SemiAnalysis H100 rental index, viaCrypto Briefing; SemiAnalysis InferenceX.

FIA, Formula 1 Technical and Sporting Regulations (power unithomologation; aerodynamic testing restrictions).

PitchBook, inference startup venture funding (to 28 Aug 2026);company announcements.

Fireworks AI, Fireworks RFT; Baseten, Baseten Training is GA; Business Wire, Baseten Acquires Parsed; Sacra, Together AI and Refuel.ai; Techzine, CoreWeave training andinference.