Skip to content

Latest commit

 

History

History
108 lines (79 loc) · 4.62 KB

victoriametrics.md

File metadata and controls

108 lines (79 loc) · 4.62 KB
Raw

TSBS Supplemental Guide: VictoriaMetrics

VictoriaMetrics is fast, cost-effective and scalable time-series database. It supports InfluxDB, Graphite and OpenTSDB protocols for ingesting data and also can be used as long-term remote storage for Prometheus. This supplemental guide explains how the data generated for TSBS is stored, additional flags available when using the data importer (tsbs_load_victoriametrics), and additional flags available for the query runner (tsbs_run_queries_victoriametrics).

To install all required tools pls do following:

# Install desired binaries. At a minimum this includes tsbs_generate_data,
# tsbs_generate_queries, one tsbs_load_* binary, and one tsbs_run_queries_*
# binary:
$ cd $GOPATH/src/github.com/timescale/tsbs/cmd
$ cd tsbs_generate_data && go install
$ cd ../tsbs_generate_queries && go install
$ cd ../tsbs_load_victoriametrics && go install
$ cd ../tsbs_run_queries_victoriametrics && go install

This should be read after the main README.

Data format

Data generated by tsbs_generate_data for VictoriaMetrics is serialized in the same format as for InfluxDB. This is "pseudo-CSV" format, each reading is composed of a single line where the name of the table is the first item, a comma, followed by several comma-separated items of tags in the format of <label>=<value>, a space, several comma-separated items of fields in the format of <label>=<value>, a space, and finally the timestamp for the reading.

An example for the cpu-only use case:

cpu,hostname=host_0,region=eu-central-1,datacenter=eu-central-1b,rack=21,os=Ubuntu15.10,arch=x86,team=SF,service=6,service_version=0,service_environment=test usage_user=58.1317132304976170,usage_system=2.6224297271376256,usage_idle=24.9969495069947882,usage_nice=61.5854484633778867,usage_iowait=22.9481393231639395,usage_irq=63.6499207106198313,usage_softirq=6.4098777048301052,usage_steal=44.8799140503027445,usage_guest=80.5028770761136201,usage_guest_nice=38.2431182911542820 1451606400000000000

Remember to set -timestamp-start and -timestamp-end flags in range of VictoriaMetrics retention period. All data out of retention period will be automatically deleted after insertion.

One of the ways to generate data for insertion is to use scripts/generate_data.sh:

FORMATS=victoriametrics SCALE=100 TS_START=2019-09-01T00:00:00Z TS_END=2019-09-03T00:00:00Z  ./scripts/generate_data.sh

tsbs_load_victoriametrics

See recommendations for insertion in InfluxDB protocol.

One of the ways to load data in VictoriaMetrics is to use scripts/load_victoriametrics.sh:

./scripts/load_victoriametrics.sh

Assumed that VictoriaMetrics is already installed and ready for insertion on default port 8428. If not - please set DATABASE_PORT variable accordingly.

Additional Flags

-urls (type: string, default: http://localhost:8428/write)

Comma-separated list of URLs to connect to for inserting data. It can be just a single-version URL or list of VMInsert URLs. Workers will be distributed in a round robin fashion across the URLs.

Generating queries

VictoriaMetrics query language MetricsQL has some limitations comparing to SQL. Because of this VictoriaMetrics query generator lacks for implementation of query types for devops use-case:

  • groupby-orderby-limit - results are always ordered by time and can't be limited;
  • lastpoint - can't be queried if datapoint is older than 5 minutes;
  • high-cpu-1, high-cpu-all - can't be queried without grouping by step.

The iot use-case wasn't implemented yet.

Of of the ways to generate queries for VictoriaMetrics is to use scripts/generate_queries.sh:

 FORMATS=victoriametrics SCALE=100 TS_START=2019-09-01T00:00:00Z TS_END=2019-09-03T00:00:00Z \
 QUERY_TYPES="cpu-max-all-8 double-groupby-1" ./scripts/generate_queries.sh

Consider to generate queries with same params as generated data.

tsbs_run_queries_victoriametrics

To run generated queries follow examples in documentation:

cat /tmp/bulk_queries/victoriametrics-cpu-max-all-8-queries.gz | gunzip | tsbs_run_queries_victoriametrics

Additional flags

-urls (type: string, default: http://localhost:8428)

Comma-separated list of URLs to connect to for querying. It can be just a single-version URL or list of VMSelect URLs. Workers will be distributed in a round robin fashion across the URLs.