Your Lakehouse Bill Doubled and Nobody Changed the Workload

Four things drive a lakehouse bill up with no workload change: idle compute, small files, stale partitioning, capacity overruns. What to check for each one.

Your lakehouse bill is up by half, or it has doubled. Same dashboards. Same nightly loads. Same number of people logging in. Finance wants to know what changed, and nobody can tell them.

In platform and migration work the answer is almost always one of four things, and none of them is a workload change. Compute that never idles long enough to switch off. Small files that nothing is compacting. A partition scheme chosen when the table was a tenth of its current size. Capacity overruns that reach your business as a slow report rather than a bigger invoice. Three of the four are configuration. One is a rebuild. Knowing which is which is most of the problem.

Compute that never switches off

Databricks sets the default auto-stop on pro and classic SQL warehouses to 45 minutes, and on serverless warehouses to 10. That suits an analyst on a continuous shift. It is expensive for the real pattern, which is one query at 9:05, another at 9:50, another at 10:40: the warehouse never sits idle long enough to stop, so a few minutes of work bills as a full morning.

The opposite pattern has a cost too. Snowflake bills per second with a 60-second minimum each time a warehouse starts, so orchestration that resumes and suspends a warehouse dozens of times an hour pays that minimum dozens of times.

Both platforms are doing exactly what they document, and the defaults are sensible for the usage they assume. What is missing is anyone checking them against how the warehouse is actually used.

First move: for each warehouse, compare a week of active query minutes against billed minutes. The gap tells you what to set the stop time to.

Small files that nothing is compacting

Streaming ingestion and frequent MERGE statements write many small files, and query engines pay a cost for every file they open, so the same query against the same data gets slower and more expensive each month.

Databricks has largely automated this. Per Microsoft's documentation, predictive optimization runs OPTIMIZE, VACUUM and ANALYZE on Unity Catalog managed tables, enabled by default for accounts created on or after November 11, 2024, with older accounts on a rollout due to finish in August 2026. Its scope is published plainly: it does not run ZORDER, it skips Z-ordered files, and tables outside Unity Catalog management are not included.

That published scope is the useful part. Most estates are only partly governed by Unity Catalog, so coverage is partial by definition, and the tables left out are usually the oldest and largest.

First move: take your twenty biggest tables, check the Predictive Optimization field on each, and put the ones outside it on a scheduled OPTIMIZE with a named owner.

A partition scheme chosen when the table was small

This is the expensive one. Databricks guidance is blunt: under 1 TB, do not partition. From 1 TB to 100 TB, use liquid clustering instead, because partitioning "likely negatively affects performance more often than it helps." Where partitions exist, each should hold at least 1 GB.

Most estates were partitioned by day while the table still held tens of gigabytes, because that is the pattern everyone learned. Several terabytes later the same scheme produces thousands of partitions of a few megabytes each, and every query pays to open all of them.

The line worth putting in front of a CIO is in the same documentation: an ineffective partitioning strategy "might negatively affect query performance and require a full rewrite of data to fix." That is a scheduled migration with a test plan, not a setting, which is exactly why it goes unfixed for years while the bill compounds.

First move: list partition count against table size for your top tables. Anything under 1 TB that is partitioned at all is a candidate to drop partitioning and let clustering do the work, and that one usually pays for itself fastest.

Overruns that look like slowness instead of money

Fabric is built differently, and that changes where the cost shows up rather than whether it exists. Capacity is bought up front, so an overrun does not arrive as a larger invoice.

Microsoft documents what happens instead: interactive work smooths over five to 64 minutes, background work over 24 hours, and once a capacity has consumed more than 10 minutes of future capacity, Fabric delays interactive jobs by 20 seconds at submission. Past 60 minutes it rejects them.

That is sound design. It also means the signal arrives somewhere finance is not looking. Your business sees a Power BI report taking 20 seconds to respond and files a performance ticket. The cause is a Spark job someone scheduled without a capacity conversation.

First move: watch carryforward in the capacity metrics app rather than waiting for complaints, and keep heavy background jobs off the capacity your business-facing reports run on.

Attribution is a design decision, not a report

Those four moves will recover the spend once. What stops it coming back is attribution, and that has to be designed in rather than bolted on when finance asks: query tags on every orchestrated workload so spend traces to a job and an owner, capacity and workspace topology that maps to a budget holder, and compaction, clustering and stop times owned by someone by name.

That matters more than the money. Without it, your next platform decision gets made on anecdote, and the standard response to an unexplained bill is to move platforms, which reproduces all four causes on new infrastructure. Agent workloads raise the stakes, because an agent may chain a dozen queries where a person runs one, and it does not stop at 6pm.

Coding agents will write the notebook. They will not choose your partition strategy or decide who owns the compaction job.