Why you should use Elastic for infrastructure metrics

1
Why you should use
Elastic for Infrastructure
Metrics
Dimitri Mazmanov
Principal Product Manager
Observability
Carlos Pérez-Aradros
Tech Lead
Observability

2
This presentation and the accompanying oral presentation contain forward-looking statements, including statements
concerning plans for future offerings; the expected strength, performance or benefits of our offerings; and our future
operations and expected performance. These forward-looking statements are subject to the safe harbor provisions
under the Private Securities Litigation Reform Act of 1995. Our expectations and beliefs in light of currently
available information regarding these matters may not materialize. Actual outcomes and results may differ materially
from those contemplated by these forward-looking statements due to uncertainties, risks, and changes in
circumstances, including, but not limited to those related to: the impact of the COVID-19 pandemic on our business
and our customers and partners; our ability to continue to deliver and improve our offerings and successfully
develop new offerings, including security-related product offerings and SaaS offerings; customer acceptance and
purchase of our existing offerings and new offerings, including the expansion and adoption of our SaaS offerings;
our ability to realize value from investments in the business, including R&D investments; our ability to maintain and
expand our user and customer base; our international expansion strategy; our ability to successfully execute our
go-to-market strategy and expand in our existing markets and into new markets, and our ability to forecast customer
retention and expansion; and general market, political, economic and business conditions.
Additional risks and uncertainties that could cause actual outcomes and results to differ materially are included in
our filings with the Securities and Exchange Commission (the “SEC”), including our Annual Report on Form 10-K for
the most recent fiscal year, our quarterly report on Form 10-Q for the most recent fiscal quarter, and any
subsequent reports filed with the SEC. SEC filings are available on the Investor Relations section of Elastic’s
website at ir.elastic.co and the SEC’s website at www.sec.gov.
Any features or functions of services or products referenced in this presentation, or in any presentations, press
releases or public statements, which are not currently available or not currently available as a general availability
release, may not be delivered on time or at all. The development, release, and timing of any features or functionality
described for our products remains at our sole discretion. Customers who purchase our products and services
should make the purchase decisions based upon services and product features and functions that are currently
available.
All statements are made only as of the date of the presentation, and Elastic assumes no obligation to, and does not
currently intend to, update any forward-looking statements or statements relating to features or functions of services
or products, except as required by law.
Forward-Looking Statements

3
Evolving Architectures ~↑ Monitoring Complexity
Hardware & software trends
are evolving in tandem
Higher resource utilization
increases monitoring complexity
• Orchestration/Hypervisor
• Dynamic/ephemeral jobs
• You can no longer "point" to
where that job lives
Shift to cloud-native yields
maintainable code, with costs
• Traditional licensing models don't scale
as well as your applications
• Hurdles with autoscaling
Monitoring Complexity

4
Applications
VMs/Containers
Other DBs,
Services &
Middleware
Orchestration InfrastructureAPM
Metrics
Logs
Uptime
Uptime
APM Metrics
APM Logs
APM
APM
Metrics
Logs
Uptime
Metrics
Logs
Uptime
APM

5
• Support the full stack
• Easily ingest from new sources
• Monitor dynamic ecosystems
• Ability to interact with your data
– Aggregations and visualizations
– Different views based on who is looking
• Rich and flexible alerting
• Long term, reliable storage
• Bonus points for full Observability
Needs from a monitoring solution
Core features and functionally

6
Ingesting Metrics to
Elastic

8
Instructions
right in Kibana
Growing list of integrations
● Download and install
Metricbeat
● Edit the configuration for
destination
● Enable and configure the
module
● Start the beats
● Explore!

9
● Deploy Elastic Agent
● Choose the integration type
● Register and configure the data
source
● Specify the data you want to
collect
● Explore!
Elastic Fleet
Centralized ingest and configuration

10
Use your existing shippers
Core features and functionality
Your App
Prometheus
Exporter
Your App
Prometheus
Exporter
Metricbeat +
Elasticsearch
Prometheus
Server
Metricbeat +
Elasticsearch Azure Monitor

11
Autodiscover
Automatically monitor new containers
● Perfect for dynamic ecosystems
● Automatically picks up new
instances
● Works with K8s, Docker, AWS, etc.
● Hints based auto-discovery for K8s
● Full context backed by Elastic
Common Schema

Storing Metrics in Elasticsearch
● Metrics stored as numeric fields
○ Depending on expected values:
float, double, integer...
● Dimensions/labels normally stored
as keyword
● Several metrics per document
○ more efficient
○ one doc per combination of
dimensions (time series)
{
"@timestamp": "2018-09-27T10:08:38",
"system": {
"cpu": {
"nice": 8,
"user": 2,
},
“load”: 1.2,
},
"host": "frontend01.bigorg.dev",
"zone": “europe-west”,
...
}
Data model

Storing Metrics in Elasticsearch
{
"@timestamp": "2018-09-27T10:08:38",
"system": {
"cpu": {
"nice": 8,
"user": 2,
},
“load”: 1.2,
},
"host": "frontend01.bigorg.dev",
"zone": “europe-west”,
...
}
Correlation

15
Elastic Common Schema
Established, predictable fields

● Several types for numbers
double, integer, float
depending on size needs…
● Distributed Histograms (7.6
● IPs
query by IP/subnet
● Geo
Map your metrics
● Dates
Rich typing and
filtering
Much more than single type
numbers and string labels

Powerful aggregations
• Common metric aggs (sum, avg, count, min, max…)
• With more choices on top!
– Mutate data / calculate metrics at query time with scripting
– Grouping is not limited to labels: Geo proximity, filters, ranges

Index lifecycle management
Reduce storage costs as data ages
1
2
3
1 2 3
Hot Nodes Cold Nodes
Warm
Nodes
1

Rollups
Reduce storage costs as data ages

Distributed by design
• Horizontally scalable
• Cross cluster search
• Cross cluster replication
Easy to scale

21
Powerful data store
Beyond Time Series
● Inverted index + columnar store
● Optimized numeric field types (BKD
● Powerful aggregations framework
● Fast response even for
high-cardinality queries
● ILM & Data Rollups
● With all of the benefits of the
Elastic Stack

22
Making metrics
actionable with Elastic

23
Dashboards &
Visualizations
Out-of-the-box visibility
● Ship with most integrations
● Mix and match for your needs
● Leverage Kibana drilldowns for
custom navigation paths
● Of course, dedicated Metrics
and Logs apps

24
Metrics App
Birds-eye view or drill down

25
Integrated Alerting
Automatically detect and alert
● Many types of alerts
● Prefiltering based on context
● Multiple facets per alert
○ CPU and Memory
○ Network TX and RX
● Automatically split alerts on
chosen field (per
container/pod/host)
● Deviations in logging rates

26
Machine Learning
Automatically detect and alert
● Automate anomaly detection at
scale and across disparate data
sources
● Find patterns in your logs
● Automatically call out anomalies
and outliers

27
Full Observability
Unified data, UI and alerting

Why you should use Elastic for infrastructure metrics

More Related Content

What's hot

Similar to Why you should use Elastic for infrastructure metrics

More from Elasticsearch

Recently uploaded

Why you should use Elastic for infrastructure metrics