While you could add a job within Prometheus to include the configuration to scrape Prometheus metrics from a specific endpoint, the installation we did earlier includes a configuration that will update what it is looking at dynamically, based on annotations on pods. One of the benefits of Prometheus is that it has support for automatically detecting changes in your cluster, based on annotations, and it can look up the endpoints for the pods that back a service.
Since we deployed Prometheus using Helm, you can find the relevant configuration embedded within the values.yaml file. Look for the Prometheus job kubernetes-service-endpoints, and you will find both the configuration and some documentation of how it can be used. If you don't have the files locally, you can view this configuration at https://github.com/kubernetes/charts/blob/master/stable/prometheus/values.yaml#L747-L776.
This configuration looks for services within the cluster that have...
