shards

The large number of shards per process mean that average load across shards will be relatively balanced.

Especially when one of those processes has to take down nodes to avoid some shards having different stored procedures than other shards.

Since the number of shards per index isn't configurable after index creation, shard size growth and disparity is unavoidable.

Or, the difference between buying much more database hardware and having smaller shards to avoid index contention.

So, on a minimum three-node setup, three replicas + three shards per table are doable?

This makes me want to find some shards to play some Ultima Online this weekend.

You can easily get commodity servers with 256GB of RAM per node x 10 shards.

In addition you can shard the table into 3 shards for additional performance gains.

Writes initially only happen on primary shards, and then propagate to replicas.

It let's you host your own private servers (shards in UO terms).

You start off with code that makes no sense at all, and gradually as you brush away soil from the pot shards, a story emerges.

I'm just not sure why the shards are so imbalanced in your example.

The main trick is decomposition of the application, data model, and operations such that every process may have a thousand discrete and disjoint shards of "stuff" it shares with no other process.

"Writes initially only happen on primary shards, and then propagate to replicas." - also true but you still want this to be as distributed as possible.

"I was under the impression that shards are kept uniform in size as ES will try and equally spread data to all shards in an index, so that there aren't imbalances." This is true by default and it's fine for simple document retrieval but it doesn't scale well if you want to do complex aggregations on arbitrary filters of large datasets.