Data Partitioning: Partition. Regret. Repeat

Ever tried moving a giant couch into a tiny apartment?

You eyeball the hallway. "Yeah, it'll fit"

You try turning it upright. "Nope"

You remove the door. "Still no"

Two hours later, you're still trying to wedge it in, sweating and cursing.

That's what bad partitioning feels like.

Partitioning is half science, half black magic, but like everything in engineering, a few good practices can keep you from shooting yourself in the foot.

First rule of partitioning: design for your actual workload, not the one you wish you had. Blindly partitioning without knowing your access patterns is how you end up solving the wrong problem — and adding new ones for free.

Before you touch a schema sit down and ask yourself the following questions:

• Read vs Write Patterns — Are you mostly reading, mostly writing, or doing both at full blast? — Do writes need strict ordering (like event logs, telemetry), or can they be spread across shards?
• Access Patterns — Do most queries target single records, small ranges, or large scans?
• Latency Expectations — Real-time? Near real-time? "We don't care, it runs overnight"? — Smaller partitions help with fast lookups; bigger partitions reduce metadata overhead for batch workloads and help with throughput.
• Geography & Compliance Needs — Does your data need to stay in a particular region for legal or performance reasons (think GDPR, CCPA, multi-region replication)?. — Partitioning can help or hurt — depending on how you implement it.

Think 2+ years ahead. Changing a partitioning strategy on a live system is painful — it's expensive, slow, and often involves downtime. It's like getting a bad tattoo: seems like a good idea at first, but later on you're filled with regret.

Partitioning is cheap to plan, but expensive to change later.

Once you know the reality of your workload, match your partitioning design to it. Not the other way around.

Split data based on ranges of values — timestamps, IDs, whatever progresses linearly.

It's good for:

• Time-series data (logs, metrics, IoT, etc.)
• Archival use cases where older data goes cold

Potential challenges:

• Hot partitions if all writes go to "now"
• Unbalanced reads if queries focus only on recent or specific ranges
• Needs regular rebalancing if ranges grow unevenly.

Take a field (like user_id), hash it, mod by N — bam, even distribution.

It's good for:

• High-cardinality keys with uniform access (e.g., user profiles, sessions)
• Write-heavy workloads where balance matters more than locality

Potential challenges:

• No natural ordering, so range queries are not really fit
• Harder to prune partitions during scans
• Rehashing = pain during resharding