etcd

I agree with teraflop that just because etcd gives strong guarantees, that doesn't mean your application logic built on top of etcd primitives shares them.

We tried running etcd alongside Postgres in an early prototype, but ran into issues with etcd cluster stability when destroying and recreating nodes.

You'd run the proxy listening on localhost, and then have it connected to the stable etcd cluster elsewhere.

Latest version (0.7) also introduced grid wide etcd so your application can also use that for custom service discovery.

Autoscale the Postgres machines as much as you want after that while leaving etcd on stable machines.

In the end, instead of the likes of etcd, table storage with a caching client was sufficient...

Both Synapse and Nerve support custom backends using whatever system you want (zookeeper, etcd, DNS, etc).

The problem with etcd members on every Postgres node is that clusters fixed nodes or members.

There's already a few topics on the mailing list about etcd losing its data and Kubernetes doesn't know how to recover.

However it would be easier to install etcd on every Postgres node and just make a golang library that sets the master of Postgres to the etcd master (etcd also has a leader).

Amazon is keeping DynamoDB Available, you don't have to manage an etcd cluster), and great scaling potential (vs tuning and scaling individual HAProxy nodes).

The code relies on functionality in etcd to prevent a race condition.

And if clients and the seed and the master are all "wrong", then that's ok because at least they all "consistently" disagree with etcd.

It has no way of knowing which one is considered to be the "real" one by etcd; and even if it tried to check, that check would itself be subject to race conditions.

Etcd & flannel are required, the rest is just wiring a set of binaries.

Etcd itself can provide this for operations on its own datastore, but that doesn't mean it can be used as a perfect failure detector for another system (which is impossible in the general case).