Metrics system design

Hadoop provides a framework to collect internal events and metrics and report them to the external system. The external system could be simply writing to a file or a tool like Ganglia. The new Hadoop Metrics2 framework has been revamped to integrate better with Ganglia.

The best things about the framework are the pluggable output plugins and the ability to reconfigure it without the need to restart the daemons.

The metrics have three main parts:

• Producer: The producer is the source of metrics generation and produces metrics for the upstream

• Consumers: They are basically the sinks of the framework, as they consume the metrics generated by the producers

• Pollers: They poll the sources and deliver data to the sink or consumers