Open-source programmed in Go to be fast and scalable
This page contains my notes on learning and using InfluxDB for managing and analyzing time-series data.
- Open-source (MIT)
- No external dependencies (written in Go)
- SQL-like query language
- Input data “Line Format” (not exactly JSON)
- Stores data in compressed format
- Horizontally scaleable (across several servers)
- Kapacitor component recognizes anomalies
- Integrates with TensorFlow
The components around InfluxDB is calle the “TICK” stack for Telegraf, InfluxDB, Chronograf, and Kapacitor.
Telegraf is a plugin-driven server agent for collecting and reporting metrics. It pulls into one place metrics from StatsD, Redis, Elasticsearch, PostgreSQL, and more.
InfluxDB is a time-series database built from the ground up to handle high write and query loads.
Chronograf is a graphing and visualization application for performing ad hoc exploration of data.
Kapacitor is a data processing framework providing alerting, anomaly detection, and action frameworks.
PROTIP: Instead of Kapacitor, many use Grafana to display dashboards.
Another way of looking at this:
Charts (files that describe a set of Kubernetes resources) read by Helm.
platform is open sourced at
Where Influx makes money is on their Cloud platform: 14 days free, then https://cloud.influxdata.com/plan-picker on AWS EC2 us-west-2 region
- $0.35 per hour
- $2.08 per hour
Install Influx CLI
Influx DB does not use JSON, but a “line protocol” of its own design.
Numbers are assumed to be floating point by default.
Integer values must have an “i” (such as value=12i).
Since the Influx database is built for time series data,
Date values do not have a label. DateTime stamps are 19 digits consisting of the number of seconds since the 1970 Epoch, plus 11 more numbers for a precise nanosecond, as in “1445299200000000000”.
QUESTION: Sample code and libraries for clients communicating with the database.
curl https://s3-us-west-2.amazonaws.com/influx-sample-data/NOAA.txt > NOAA_data.txt influx -import -path=NOAA_data.txt -precision=s influx USE NOAA_water_database precision rfc3339 SHOW SERIES SHOW FIELD KEYS # data types
The InfluxDB CLI works with a SQL-like language.
influx CREATE DATABASE mydb SHOW DATBASES USE mydb SELECT percentile(busy,90) FROM cpu WHERE time > now() - 1h SELECT MEAN(busy) FROM ... GROUP BY location SELECT * FROM h2o_quality GROUP by time(10m) ORDER BY time DESC LIMIT 4 WHERE time > now() - 1h
Kapacitor recognizes anomalies
// detect system down var period = 2m var every = 30s // select the stream var sys_data = stream |from () .database('telegraf') .measurement('system') .where(lambda): "host" =~ /tot.*/ OR "host" =~ ~/prod.*/ .groupBy('host','cluster_id') |window() .period(period) .every(every)
InfluxDB created their own Storage Engine to define sharding distributing read/writes on several disks, making sure. Indexing. Examples include InnoDB, MyISAM, Falcon, XtraDB (MySQL, etc.)
Influx claims “350,000 writes per second on commodity hardware”.
Benchmarking InfluxDB Storage Engines: v0.10, v0.9,and v0.8 by Todd Persen, VP of Engineering January 28, 2016
Time-Structured Merge (TSM) Tree
Built-in WAL with queryable in-memory cache
TODO: Compare against ELK stack and NoSQL databases.
The Influx server is written the Go programming lanuage.
Integrates with TensorFlow
See video “How to Manage TensorFlow with InfluxData” (The TensorFlow Jupyter notebook for weather prediction is shown from 17:24)
- #influxdb #influxdata on Twitter ???
- @influxdb on Twitter
- @influxdbNews on Twitter
- @hostedinflux on Twitter
- LinkedIn group of InfluxData Developers
- LinkedIn company profile
- YouTube channel has no videos. Related: Grafana
- LinkedIn group
- Meetup.com - none
- Google Groups for developers
- 100% recommendation from 2 employees on Glassdoor
Paul Dix, CTO of InfluxData (NYC resident)
Monitoring InfluxCloud with InfluxDB [28:06] at GrafanaCon 1 Dec 2016 (1)
Metric-driven development with Ansible, InfluxDB, and Grafana at Red Hat Summit
Introduction to InfluxDB [53:08] at Hakka Labs
InfluxDB Storage Engine Internals [43:42] at Hakka Labs
- Watch Everything, Watch Anything: Anomaly Detection at Salt Lake City DevOps Days
Open Source Summit North America 2017 Monday, September 11, 2017 - Thursday, September 14, 2017