Download as PDF, PPTX
![Sample Configurations
{
"endpoint" : "http://localhost:8000/nginx_status",
"metrics_config" : [
{ "name" : "activeConnections",
"metric_type" : "gauge",
"units" : "number of active connections",
"data_type" : "int",
"polling_frequency" : 10,
"regex" : "Active connections: ([0-9]+)"
},
{ "name" : "reading",
"metric_type" : "gauge",
"units" : "number of reading connections",
"data_type" : "int",
"polling_frequency" : 10,
"regex" : "Reading: ([0-9]+) .*"
}
]
}
Get all prometheus metrics:
{
"endpoint" : "http://localhost:9100/metrics",
}
Get selected prometheus metrics:
{
"endpoint" : "http://localhost:8000/metrics",
"metrics_config" : [
{ "scheduler_binding_latency",
"scheduler_e2e_scheduling_latency",
"scheduling_algorithm_latency"
}
]
}](https://image.slidesharecdn.com/nativecontainermonitoring-150820002027-lva1-app6891/75/Native-container-monitoring-17-2048.jpg)
![Sample Configurations
{
"endpoint" : "http://localhost:8000/nginx_status",
"metrics_config" : [
{ "name" : "activeConnections",
"metric_type" : "gauge",
"units" : "number of active connections",
"data_type" : "int",
"polling_frequency" : 10,
"regex" : "Active connections: ([0-9]+)"
},
{ "name" : "reading",
"metric_type" : "gauge",
"units" : "number of reading connections",
"data_type" : "int",
"polling_frequency" : 10,
"regex" : "Reading: ([0-9]+) .*"
}
]
}
Get all prometheus metrics:
{
"endpoint" : "http://localhost:9100/metrics",
}
Get selected prometheus metrics:
{
"endpoint" : "http://localhost:8000/metrics",
"metrics_config" : [
{ "scheduler_binding_latency",
"scheduler_e2e_scheduling_latency",
"scheduling_algorithm_latency"
}
]
}](https://crownmelresort.com/image.slidesharecdn.com/nativecontainermonitoring-150820002027-lva1-app6891/75/Native-container-monitoring-17-2048.jpg)
More Related Content
Similar to Native container monitoring
![Support, Monitoring, Continuous Improvement & Scaling Agentic Automation [3/3]](https://cdn.slidesharecdn.com/ss_thumbnails/agenticcommunityseries-day3-cfd-251120170304-ddef8112-thumbnail.jpg?width=640&height=640&fit=bounds)
Native container monitoring
- 1. Container native Monitoring RohitJnagal Anushree Narasimha
- 3. Overview ● Monitoring forcontainers ● Monitoring in a distributed system ● cAdvisor Application Metrics ● In cAdvisor ● Plumbing through a cluster Future Work ● Heapster/Kubedash ● Templates Outline
- 4. Monitoring in threeparts Collection ● collecting data and metrics ● making it discoverable ● plumbing it to higher-level systems Processing ● Ingesting, aggregation ● Analytics Managing ● Actions based on signals ● Alerts. Pagers!
- 5. Monitoring in threeparts Collection ● collecting data and metrics ● making it discoverable ● plumbing it to higher-level systems Processing ● Ingesting, aggregation ● Analytics Managing ● Actions based on signals ● Alerts. Pagers!
- 6. Collection in servers/VMs ●Node agent ● Knows the binary to monitor ● Monitoring logic plugged into agent Moving to containers ● Same node agent - understands multiple applications. ● Applications can push data to agent. ● Sidecars ● Applications publish data. Pulled off by off- host agents.
- 7. Near-instant data ● schedulersin cluster management tools ● load balancers ● Alerting systems Slower feedback loops ● Autoscaling / Autoupdaters ● CPI2 Historical Data ● Offline analysis. consumers
- 8. Monitoring in distributed environment •Hosts are invisible • Containers can hop around • Monitoring needs to track and correlate multiple containers
- 9.
- 10. cAdvisor Analyzes resource usageand performance characteristics of running containers Native Docker support, work with LXC and any other container format Knows containers deeply and monitors their performance github.com/google/cadvisor
- 11.
- 12. API Node and containerspec http://host:8080/api/v2.0/machine http://host:8080/api/v2.0/spec/redis?type=docker Hierarchical Container stats: http://host:8080/api/v2.0/stats/nginx?type=docker http://host:8080/api/v2.0/stats?recursive=true Others: /summary, /events, /storage, /attributes, /ps
- 13.
- 14. Application Metrics Every containeris packaged with its monitoring data. Monitoring moves with the container. Use container composability to stack up metrics from all layers. Use metadata in image or runtime (Docker labels!) to configure monitoring
- 15. Configuring containers FROM redis ADDredis_config.json /var/cadvisor/redis_config.json LABEL io.cadvisor.metric.redis="/var/cadvisor/redis_config.json" In cAdvisor read Labels “io.cadvisor.metric.*” read /rootfs/proc/<pid>/root/<config path>
- 16. Monitoring Configuration Holds metadataabout metrics ● Endpoint (Location to collect metrics from) ● Name of metric ● Type (Counter, Gauge, ...) ● Data Type (int, float) ● Units (kbps, seconds, count) ● Polling Frequency ● Regexps (Regular expressions to be used to collect a metric)
- 17. Sample Configurations { "endpoint" :"http://localhost:8000/nginx_status", "metrics_config" : [ { "name" : "activeConnections", "metric_type" : "gauge", "units" : "number of active connections", "data_type" : "int", "polling_frequency" : 10, "regex" : "Active connections: ([0-9]+)" }, { "name" : "reading", "metric_type" : "gauge", "units" : "number of reading connections", "data_type" : "int", "polling_frequency" : 10, "regex" : "Reading: ([0-9]+) .*" } ] } Get all prometheus metrics: { "endpoint" : "http://localhost:9100/metrics", } Get selected prometheus metrics: { "endpoint" : "http://localhost:8000/metrics", "metrics_config" : [ { "scheduler_binding_latency", "scheduler_e2e_scheduling_latency", "scheduling_algorithm_latency" } ] }
- 18. App Metrics inaction
- 19. Endpoint for custommetrics: http://localhost:8080/api/v2.0/appmetrics/containerName Application metrics being collected can be discovered from the spec: http://localhost:8080/api/v2.0/spec/containerName Regular stats api also reports application metrics: http://localhost:8080/api/v2.0/stats/containerName API
- 20. Kubernetes Open-source Container orchestrationfrom Google Declarative pattern for managing containers Physical hosts abstracted out as resources Inspired and informed by Borg Kubernetes UI powered by cAdvisor github.com/kubernetes/kubernetes
- 21. App Metrics forKubernetes All kubernetes components run in containers All system services on node runs in containers All kubernetes nodes run cAdvisor (built into kubelet) All kubernetes components expose prometheus metrics App-metrics for kubernetes components!
- 22.
- 23.
- 24. Templates Add templates forapplications that have stable stats API LABEL io.cadvisor.metric.type=redis Infer monitoring information ● Lookup ports through docker inspection ● hit known endpoints (e.g.: /metrics for prometheus) ● overrides through config
- 25. Tags Adding tags tospecific metrics Convey metric intent to processors eg. Autoscalers
- 26. Ongoing work Endpoints Storage drivers Standardconfig syntax Automagic Plumbing through heapster/kubedash/kubernetes/...
- 27. Thank you! Rohit Jnagaljnagal@google Anushree Narasimha anushree.bnp@gmail cAdvisor github.com/google/cadvisor kubernetes kubernetes.io irc #google-containers