Benchmaxxing

I've read a few posts this past week – every single one promises something incredible, each with a big cool number: real-time queries 16x faster, storage 40x faster, point lookups at O(1).

All three come from well-known engineering blogs people trust, all three are (probably) reproducible, and all three, as seems now a standard in the industry, measure something other than what you think you're reading.

At its June keynote, Databricks launched Reyden — a new real-time compute engine — claiming up to 16x better performance than existing real-time serving stacks.

On stage they went further: a live demo where a competitor (ClickHouse) crashed outright while Reyden held steady. No env been given—no hardware spec, no cluster size, no cache state, no config.

Melvyn Peignon, at ClickHouse, tried to piece together that information and reproduce the crash using the only detail Databricks had actually named: TPC-H, scale factor 1, query 6. Under these settings alone, ClickHouse did not crash, meaning the exact failure shown onstage can't be reproduced from what Databricks themselves disclosed.

Nobody's calling the demo itself a lie — the crash really happened, in front of a full room; nobody is disputing that. The problem is that "some box" is the entire benchmark, and Databricks is the only one who knows what that box was.

A benchmark nobody can rerun isn't actually a benchmark, it's just someone's word you're supposed to take on faith. There is actually an official, dictionary term for the practice — obscurantism, also known as benchmarketing.

Snowflake showed up with the same complaint from a different angle. Rudi Leibbrandt took Databricks' own published Reyden numbers and configured Snowflake to match the comparison properly. At 100GB, Snowflake beat the published figure: 12.5 seconds against 13.2. At 1TB the gap widened: 28.2 seconds against 71.

This isn't a new move either. Kostas Pardalis at typedef.ai recently traced this pattern back 40 years: the same benchmarketing tricks — hidden hardware configurations, unrealistic scenarios, hidden costs, tuning specifically for the test — were already being run by Oracle and Sybase back in the early ‘80s, when the term benchmarketing itself was born.

Ever since, plenty of EULAs (End User Licensing Agreements) have flatly banned publishing independent benchmarks without the vendor's consent. It's called the DeWitt Clause, named after the professor whose 1983 benchmark made Oracle look bad, after which Larry Ellison tried to have him fired. SQL Server's license still says: "You may not disclose the results of any benchmark test ... without Microsoft's prior written approval". Oracle's license contains a similar restriction.

So the problem isn't simply that vendors publish the benchmarks most convenient to them. In some cases the licensing terms actively discourage independent benchmarks from being published at all. That can make vendor-produced benchmarks the most accessible — and sometimes the only legally publishable.

WarpStream runs a Kafka compatible streaming platform directly on object storage, so every write is really a file write to a bucket, and they published a whole post about shaving latency off exactly that write path. They applied several tricks to do it:

• pre-warmed writer pool — a batch of ready-to-go writers sitting around instead of building a new one from scratch for every write.
• flush before use — so the first-write penalty lands on that throwaway call instead of on real data.
• async closes off the critical path — closing the file happens in the background and doesn't hold the write up waiting for it to finish.

Again: it's all real, and a good read. The 40x is create plus write plus close, measured in isolation: 80-100ms down to 2-3ms. Wow — what a beat!

BUT the number for a production cluster is mentioned in the same post: two to three times faster than regional GCS.

Not forty. Two to three times.

Note that rapid storage bills more than 10x per GiB, which is why the same post also describes compacting the data back down into standard buckets.

Nobody hid anything in that post, it's extremely transparent. The metrics for the isolated path are in one paragraph, the metrics for the cluster number is in another, and in a classic marketing move, the title conveys the best looking one. (Honestly, I probably would've done the same lol)

Spotify's engineering blog on Random Access Parquet: an external key-to-row index that claims to turn a point lookup into O(1) with no chain of dependent loads. Normally, to find one row in a data lake, you first have to read one file just to learn what to read next, several times in a row. RAP hands you the exact address in a single request. Genuinely good work.

That being said, O(1) describes the lookup. It doesn't describe the read. From the same post, on unmodified files: RAP "reads the entire page containing the target row — potentially a 4MB page to extract 100 bytes".

Half the design section is about arranging your data at write time so that O(1) actually buys you something.

So the O(1) is honest, the bytes aren't free, and the headline, of course, carried the half that sells better.

The two that do disclose technically don't lie — the number is real and reproducible in exactly the scenario picked for the headline (the isolated call at WarpStream, the lookup alone at Spotify). Databricks skipped that step: no scenario was named, so nobody outside the building could tell you which one produced the crash. Either way the mechanism is the same — pick the scenario, or skip naming it, because both give you the prettiest number nobody can check against their own setup.

You can be completely honest and still hand thousands of engineers a number that fits NONE of them. Honest and fucking useless are not mutually exclusive.

The genre has settled into a shape, the same mechanic every time: pick the operation your change affects the most, measure it in isolation, put that in the title. Then bury the system-level number deep in the text, where it's technically disclosed and practically invisible — nobody scrolls a four-thousand-word blog post down to paragraph nineteen because it's boooring.

At least a vendor compares its own thing against its own past.

Take ClickBench—a benchmark where databases run the same queries and get compared head to head. In June, a QuestDB engineer picked apart a skew in it: between runs, a self-hosted database gets a clean start — the test harness clears the page cache and restarts the server, so the next measurement sees the system as if it had just booted. A managed service like Snowflake or BigQuery physically can't get that—the same harness has no access to their server to restart it, so all three "cold" runs actually hit the same live server that never got restarted. Which means the "cold" ranking is really about who had access to restart a server — architecture has nothing to do with it.

A comparison where the experiment's own conditions are unfair to one side isn't a benchmark.

People size clusters on these same numbers and justify migrations with them. Someone reads 5x, multiplies it by their current bill, builds a slide, gets a quarter of engineering time approved — and ends up with plus twenty percent, because they, like most people, run four cores per node, not thirty-two like the top row. And it's not just that the engineer believed the number. The manager approving that quarter isn't a neutral judge either — they've already half-decided the migration is needed, they just need confirmation for their own slide to leadership. 5x gives them that confirmation faster than reading the fine print on the config. Explaining that arithmetic at Monday's retro definitely won't be the guy who wrote the Databricks blog post.

Nobody's going to write that retro at all. The blog post was accurate. The engineer just applied the number to their own system, which is the only thing anyone ever does with a benchmark.

Not the caveat in paragraph nineteen. The actual sentence: we tried this and it came out to eight percent, and eight percent doesn't pay for a migration. Almost nobody publishes that sentence — and that's exactly why it would be worth writing. A post like that would save someone a quarter of engineering time instead of costing them one.

The reason you almost never see it isn't dishonesty. It's that nobody gets promoted for publishing a null result, and vendor blogs are hiring funnels and sales collateral first, engineering journals a distant second.

So here's the rule I use now, and I'd suggest you use it too: when you see a multiplier in a title, go find the configuration. It's always in the text, and never in the title.

What's the last benchmark you actually re-ran yourself?