To make sure that the monitoring and metrics collection system is working at the optimal performance, it must be designed and tuned for it.
In the case of Nagios, make sure to have a right mix of active and passive checks for services.
The performance of the total number of checks deployed as active checks and the number of nodes on which they will be executed, depends upon the resources that the Nagios server has in terms of memory and CPU cores.
Also, the network plays an important role, as it important to understand how bandwidth monitoring will take place.
Other best practice is to always have a hierarchy of the Nagios configuration layouts. Make use of host groups, service, and templates and having groups for everything makes adding nodes very easy.
Define smart check rather than doing checks every minute. For example, doing a disk check every minute might not make sense, as it does not grow that often.
Optimize plugins so as to reduce the load on the system...
