InfluxDB inserts data very slowly.

[KONEONE]

In InfluxDB version 1.8, I am inserting two types of data: one is detailed data, and the other is aggregated data computed from the detailed data. The aggregated data includes new fields such as count, sum, avg, max, and min, all of which are of type float. When using the same code, inserting detailed data is very fast, achieving approximately 200k/s. However, when inserting aggregated data, I frequently encounter statusCode=504 errors, and the insertion speed struggles to reach even 1k/s. I would like to know where I can log the write operations or how I should debug this issue.

[davidby-influx]

There are two things you should look at: what error message is returned with the 504 errors, if any, and the server logs for more details on the write failures. Turning your [log level](Structured logging) up to debug will provide more information.

[KONEONE]

There are two things you should look at: what error message is returned with the 504 errors, if any, and the server logs for more details on the write failures. Turning your [log level](Structured logging) up to debug will provide more information.

I am currently using Spark's Kafka backpressure mechanism, which has effectively alleviated the issue. However, I still haven't identified the root cause of the problem. I am attempting to investigate further by leveraging HTTP-related log configurations in InfluxDB. Thank you for your response. If I manage to identify the root cause later, I will share it. Thank you once again!

[KONEONE]

There are two things you should look at: what error message is returned with the 504 errors, if any, and the server logs for more details on the write failures. Turning your [log level](Structured logging) up to debug will provide more information.

I now know what the root cause is. When I use Spark to consume multiple Kafka topics, Spark treats the data from multiple topics as the same type of data for processing. However, each of my topics corresponds to a different database in InfluxDB. As a result, I was frequently creating batchPoint objects for insert operations. After identifying this issue, I modified my approach to consume the Spark topics sequentially. Following this change, my program started working perfectly