Skip to content

Latest commit

 

History

History
442 lines (334 loc) · 17.3 KB

MANUAL.md

File metadata and controls

442 lines (334 loc) · 17.3 KB
Raw

Manual

Quick start

The code below is similar to the one found on the README.md file but with comments removed and rows numbered for better reference.

final String serverURL = "http://127.0.0.1:8086", username = "root", password = "root";
final InfluxDB influxDB = InfluxDBFactory.connect(serverURL, username, password);  // (1)

String databaseName = "NOAA_water_database";
influxDB.query(new Query("CREATE DATABASE " + databaseName));
influxDB.setDatabase(databaseName);                                       // (2)

String retentionPolicyName = "one_day_only";
influxDB.query(new Query("CREATE RETENTION POLICY " + retentionPolicyName
        + " ON " + databaseName + " DURATION 1d REPLICATION 1 DEFAULT"));
influxDB.setRetentionPolicy(retentionPolicyName);                         // (3)

influxDB.enableBatch(
    BatchOptions.DEFAULTS
        .threadFactory(runnable -> {
            Thread thread = new Thread(runnable);
            thread.setDaemon(true);
            return thread;
        })
);                                                                        // (4)

Runtime.getRuntime().addShutdownHook(new Thread(influxDB::close));        // (5)

influxDB.write(Point.measurement("h2o_feet")                              // (6)
    .time(System.currentTimeMillis(), TimeUnit.MILLISECONDS)
    .tag("location", "santa_monica")
    .addField("level description", "below 3 feet")
    .addField("water_level", 2.064d)
    .build());

influxDB.write(Point.measurement("h2o_feet")                              // (6)
    .tag("location", "coyote_creek")
    .addField("level description", "between 6 and 9 feet")
    .addField("water_level", 8.12d)
    .build());

Thread.sleep(5_000L);

QueryResult queryResult = influxDB.query(new Query("SELECT * FROM h2o_feet")); // (7)

System.out.println(queryResult);
// It will print something like:
// QueryResult [results=[Result [series=[Series [name=h2o_feet, tags=null,
//      columns=[time, level description, location, water_level],
//      values=[
//      [2020-03-22T20:50:12.929Z, below 3 feet, santa_monica, 2.064],
//      [2020-03-22T20:50:12.929Z, between 6 and 9 feet, coyote_creek, 8.12]
//      ]]], error=null]], error=null]

Connecting to InfluxDB

(1) The InfluxDB client is thread-safe and our recommendation is to have a single instance per application and reuse it, when possible. Every InfluxDB instance keeps multiple data structures, including those used to manage different pools like HTTP clients for reads and writes.

It's possible to have just one client even when reading or writing to multiple InfluxDB databases and this will be shown later here.

Setting a default database (optional)

(2) If you are not querying different databases with a single InfluxDB client, it's possible to set a default database name and all queries (reads and writes) from this InfluxDB client will be executed against the default database.

If we only comment out the line (2) then all reads and writes queries would fail. To avoid this, we need to pass the database name as parameter to BatchPoints (writes) and to Query (reads). For example:

// ...
String databaseName = "NOAA_water_database";
// influxDB.setDatabase() won't be called...
String retentionPolicyName = "one_day_only";
// ...

BatchPoints batchPoints = BatchPoints.database(databaseName).retentionPolicy(retentionPolicyName).build();

batchPoints.point(Point.measurement("h2o_feet")
    .time(System.currentTimeMillis(), TimeUnit.MILLISECONDS)
    .tag("location", "santa_monica")
    .addField("level description", "below 3 feet")
    .addField("water_level", 2.064d)
    .build());

// ...
influxDB.write(batchPoints);
// ...
QueryResult queryResult = influxDB.query(new Query("SELECT * FROM h2o_feet", databaseName));
// ...
influxDB.close();

It's possible to use both approaches at the same time: set a default database using influxDB.setDatabase and read/write passing a databaseName as parameter. On this case, the databaseName passed as parameter will be used.

Setting a default retention policy (optional)

(3) TODO: like setting a default database, explain here how it works with RP.

Enabling batch writes

(4) TODO: explanation about BatchOption parameters:

  // default values here are consistent with Telegraf
  public static final int DEFAULT_BATCH_ACTIONS_LIMIT = 1000;
  public static final int DEFAULT_BATCH_INTERVAL_DURATION = 1000;
  public static final int DEFAULT_JITTER_INTERVAL_DURATION = 0;
  public static final int DEFAULT_BUFFER_LIMIT = 10000;
  public static final TimeUnit DEFAULT_PRECISION = TimeUnit.NANOSECONDS;
  public static final boolean DEFAULT_DROP_ACTIONS_ON_QUEUE_EXHAUSTION = false;

Configuring behaviour of batch writes when the action queue exhausts

With batching enabled, the client provides two options on how to deal with action queue (where the points are accumulated as a batch) exhaustion.

  1. When dropActionsOnQueueExhaustion is false (default value), InfluxDB#write will be blocked till the space is created in the action queue.
  2. When dropActionsOnQueueExhaustion is true, new writes using InfluxDB#write will dropped and droppedActionHandler will be called. Example usage: 
    influxDB.enableBatch(BatchOptions.DEFAULTS.dropActionsOnQueueExhaustion(true)
                                          .droppedActionHandler((point) -> log.error("Point dropped due to action queue exhaustion.")));

Configuring the jitter interval for batch writes

When using large number of influxdb-java clients against a single server it may happen that all the clients will submit their buffered points at the same time and possibly overloading the server. This is usually happening when all the clients are started at once - for instance as members of cloud hosted large cluster networks.
If all the clients have the same flushDuration set this situation will repeat periodically.

To solve this situation the influxdb-java offers an option to offset the flushDuration by a random interval so that the clients will flush their buffers in different intervals:

influxDB.enableBatch(BatchOptions.DEFAULTS.jitterDuration(500));

Error handling with batch writes

With batching enabled the client provides two strategies how to deal with errors thrown by the InfluxDB server.

  1. 'One shot' write - on failed write request to InfluxDB server an error is reported to the client using the means mentioned above.
  2. 'Retry on error' write (used by default) - on failed write the request by the client is repeated after batchInterval elapses (if there is a chance the write will succeed - the error was caused by overloading the server, a network error etc.) When new data points are written before the previous (failed) points are successfully written, those are queued inside the client and wait until older data points are successfully written. Size of this queue is limited and configured by BatchOptions.bufferLimit property. When the limit is reached, the oldest points in the queue are dropped. 'Retry on error' strategy is used when individual write batch size defined by BatchOptions.actions is lower than BatchOptions.bufferLimit.

Ensure application exit when batching is enabled

BatchOptions.DEFAULTS creates a non-daemon thread pool which prevents the JVM from initiating shutdown in the case of exceptions or successful completion of the main thread. This will prevent shutdown hooks (many frameworks and plain JVM applications use these to close/ cleanup resources) from running, preventing graceful termination of the application.

Thus, configuring batch options with a daemon thread pool will solve this issue and will for example ensure that the registered (5) shutdown hook is run to close the InfluxDB client properly (flushing and closing of resources will happen).

Close InfluxDB Client on JVM Termination

(5) In order to ensure that in-flight points are flushed and resources are released properly, it is essential to call influxDB.close() the client when it is no longer required.

Registering a shutdown hook is a good way to ensure that this is done on application termination regardless of exceptions that are thrown in the main thread of the code. Note that if you are using a framework, do check the documentation for its way of configuring shutdown lifecycle hooks or if it might already be calling close automatically.

Writing to InfluxDB

(6) ...

----8<----BEGIN DRAFT----8<----

Any errors that happen during the batch flush won't leak into the caller of the write method. By default, any kind of errors will be just logged with "SEVERE" level. If you need to be notified and do some custom logic when such asynchronous errors happen, you can add an error handler with a BiConsumer<Iterable<Point>, Throwable> using the overloaded enableBatch method:

influxDB.enableBatch(BatchOptions.DEFAULTS.exceptionHandler(
        (failedPoints, throwable) -> { /* custom error handling here */ })
);

----8<----END DRAFT----8<----

Writing synchronously to InfluxDB (not recommended)

If you want to write the data points synchronously to InfluxDB and handle the errors (as they may happen) with every write:

----8<----BEGIN DRAFT----8<----

InfluxDB influxDB = InfluxDBFactory.connect("http://172.17.0.2:8086", "root", "root");
String dbName = "aTimeSeries";
influxDB.query(new Query("CREATE DATABASE " + dbName));
String rpName = "aRetentionPolicy";
influxDB.query(new Query("CREATE RETENTION POLICY " + rpName + " ON " + dbName + " DURATION 30h REPLICATION 2 DEFAULT"));

BatchPoints batchPoints = BatchPoints
                .database(dbName)
                .tag("async", "true")
                .retentionPolicy(rpName)
                .consistency(ConsistencyLevel.ALL)
                .build();
Point point1 = Point.measurement("cpu")
                    .time(System.currentTimeMillis(), TimeUnit.MILLISECONDS)
                    .addField("idle", 90L)
                    .addField("user", 9L)
                    .addField("system", 1L)
                    .build();
Point point2 = Point.measurement("disk")
                    .time(System.currentTimeMillis(), TimeUnit.MILLISECONDS)
                    .addField("used", 80L)
                    .addField("free", 1L)
                    .build();
batchPoints.point(point1);
batchPoints.point(point2);
influxDB.write(batchPoints);
Query query = new Query("SELECT idle FROM cpu", dbName);
influxDB.query(query);
influxDB.query(new Query("DROP RETENTION POLICY " + rpName + " ON " + dbName));
influxDB.query(new Query("DROP DATABASE " + dbName));

----8<----END DRAFT----8<----

Reading from InfluxDB

(7) ...

Query using Callbacks

influxdb-java now supports returning results of a query via callbacks. Only one of the following consumers are going to be called once :

this.influxDB.query(new Query("SELECT idle FROM cpu", dbName), queryResult -> {
    // Do something with the result...
}, throwable -> {
    // Do something with the error...
});

Query using parameter binding (a.k.a. "prepared statements")

If your Query is based on user input, it is good practice to use parameter binding to avoid injection attacks. You can create queries with parameter binding with the help of the QueryBuilder:

Query query = QueryBuilder.newQuery("SELECT * FROM cpu WHERE idle > $idle AND system > $system")
        .forDatabase(dbName)
        .bind("idle", 90)
        .bind("system", 5)
        .create();
QueryResult results = influxDB.query(query);

The values of the bind() calls are bound to the placeholders in the query ($idle, $system).

Advanced Usage

Gzip's support

influxdb-java client doesn't enable gzip compress for http request body by default. If you want to enable gzip to reduce transfer data's size , you can call:

influxDB.enableGzip()

UDP's support

influxdb-java client support udp protocol now. you can call following methods directly to write through UDP.

public void write(final int udpPort, final String records);
public void write(final int udpPort, final List<String> records);
public void write(final int udpPort, final Point point);

Note: make sure write content's total size should not > UDP protocol's limit(64K), or you should use http instead of udp.

Chunking support

influxdb-java client now supports influxdb chunking. The following example uses a chunkSize of 20 and invokes the specified Consumer (e.g. System.out.println) for each received QueryResult

Query query = new Query("SELECT idle FROM cpu", dbName);
influxDB.query(query, 20, queryResult -> System.out.println(queryResult));

QueryResult mapper to POJO

An alternative way to handle the QueryResult object is now available. Supposing that you have a measurement CPU:

> INSERT cpu,host=serverA,region=us_west idle=0.64,happydevop=false,uptimesecs=123456789i
>
> select * from cpu
name: cpu
time                           happydevop host    idle region  uptimesecs
----                           ---------- ----    ---- ------  ----------
2017-06-20T15:32:46.202829088Z false      serverA 0.64 us_west 123456789

And the following tag keys:

> show tag keys from cpu
name: cpu
tagKey
------
host
region
  1. Create a POJO to represent your measurement. For example:
public class Cpu {
    private Instant time;
    private String hostname;
    private String region;
    private Double idle;
    private Boolean happydevop;
    private Long uptimeSecs;
    // getters (and setters if you need)
}
  1. Add @Measurement,@TimeColumn and @Column annotations (column names default to field names unless otherwise specified):
@Measurement(name = "cpu")
public class Cpu {
    @TimeColumn
    @Column
    private Instant time;
    @Column(name = "host", tag = true)
    private String hostname;
    @Column(tag = true)
    private String region;
    @Column
    private Double idle;
    @Column
    private Boolean happydevop;
    @Column(name = "uptimesecs")
    private Long uptimeSecs;
    // getters (and setters if you need)
}

Alternatively, you can use:

@Measurement(name = "cpu", allFields = true)
public class Cpu {
    @TimeColumn
    private Instant time;
    @Column(name = "host", tag = true)
    private String hostname;
    @Column(tag = true)
    private String region;
    private Double idle;
    private Boolean happydevop;
    @Column(name = "uptimesecs")
    private Long uptimeSecs;
    // getters (and setters if you need)
}
  1. Call InfluxDBResultMapper.toPOJO(...) to map the QueryResult to your POJO:
InfluxDB influxDB = InfluxDBFactory.connect("http://localhost:8086", "root", "root");
String dbName = "myTimeseries";
QueryResult queryResult = influxDB.query(new Query("SELECT * FROM cpu", dbName));

InfluxDBResultMapper resultMapper = new InfluxDBResultMapper(); // thread-safe - can be reused
List<Cpu> cpuList = resultMapper.toPOJO(queryResult, Cpu.class);

Writing using POJO

The same way we use annotations to transform data to POJO, we can write data as POJO. Having the same POJO class Cpu

String dbName = "myTimeseries";
String rpName = "aRetentionPolicy";
// Cpu has annotations @Measurement,@TimeColumn and @Column
Cpu cpu = new Cpu();
// ... setting data

Point point = Point.measurementByPOJO(cpu.getClass()).addFieldsFromPOJO(cpu).build();

influxDB.write(dbName, rpName, point);

QueryResult mapper limitations

  • If your InfluxDB query contains multiple SELECT clauses, you will have to call InfluxResultMapper#toPOJO() multiple times to map every measurement returned by QueryResult to the respective POJO;
  • If your InfluxDB query contains multiple SELECT clauses for the same measurement, InfluxResultMapper will process all results because there is no way to distinguish which one should be mapped to your POJO. It may result in an invalid collection being returned;
  • A Class field annotated with @Column(..., tag = true) (i.e. a InfluxDB Tag) must be declared as String.

QueryBuilder

An alternative way to create InfluxDB queries is available. By using the QueryBuilder you can create queries using java instead of providing the influxdb queries as strings.

Generic POJO super classes

POJO classes can have generic super classes, for cases where multiple measurements have a similar structure, and differ by type(s), as in:

public class SuperMeasurement<T> {
    @Column
    @TimeColumn
    private Instant time;
    @Column
    T value;
    // Other common columns and tags
}

public class SubMeasurement extends SuperMeasurement<String> {
    // Any specific columns and tags
}

InfluxDBMapper

In case you want to save and load data using models you can use the InfluxDBMapper.

Other Usages

For additional usage examples have a look at InfluxDBTest.java

Publishing

This is a link to the sonatype oss guide to publishing. I'll update this section once the jira ticket is closed and I'm able to upload artifacts to the sonatype repositories.

Frequently Asked Questions

This is a FAQ list for influxdb-java.