Modern app and infrastructure monitoring and alerting should be based on labeled time series, and black box system metrics according to Bjorn Rabenstein, a Production Engineer at SoundCloud. In this talk Rabenstein covers some high level concepts of Prometheus including why you want “white-box” ad service-based monitoring in the modern world. If you can graph it you can alert it – including potential problems based on parameters like disk space. With Prometheus high-level alerting can be combined with the granularity to inspect individual components.
You can watch this video also at the source.