namenode

Services which build on Hadoop What's indicating that the NameNode is a scaling bottleneck?

Either of these scenarios causes the NameNode to double-count block replicas, incorrectly determine that a block is over-replicated, and remove those replicas permanently from those DataNodes.

I think the client assumes the data is written after $dfs.namenode.replication.min blocks have been written, which I think is 1 by default.

There are production Hadoop clusters with PBs of data, and the NameNode has not been an issue for scalability.

In practice -- again, see my experience writing jobs processing pbs of data on 1000s of nodes -- hadoop experiences namenode issues.

You're right, file blocks do not flow through the NameNode, only metadata updates do.

In reality, there's very little demand for namenode scalability at the current moment, but once this changes, the community will implement it.

It is also worth noting that Namenode isn't a spof in Hadoop 2.

Very interesting, the results are already impressive (hot stand by namenode).

But given that a cluster is easy to setup (I made a really simple NameNode + Two Data nodes in 45 minutes, going cold), it might not be a bad idea.

> The NameNode is a single point of failure.

Hadoop 2.x lets you run a standby namenode with automatic failover.

Symptoms If data loss has occurred, the NameNode reports “missing blocks” on the NameNode Web UI.

Namenode HA[1] has been in for a while and QJM[1] went in more recently.