ThreadPoolExecutor alway alive after close batching API and database client

[Dev-HexoTech]

Specifications

• Client Version: 1.40.0
• InfluxDB Version: 2.7.1
• Platform: Ubuntu
• Python Version: 3.10.9

Code sample to reproduce problem

import threading
import time
from influxdb_client import (
    InfluxDBClient
)
from influxdb_client.client.write_api import (
    WriteOptions,
    WriteType,
)


def check_thread_alive():
    """Check if all thread are correctly closed."""
    msg: str = f"{len(threading.enumerate())} threads running"
    for thread in threading.enumerate():
        msg += f"\n > {thread.name}"
    print(msg)


data_points = [
    {
        "measurement": "temperature",
        "tags": {"location": "room1"},
        "time": "2024-02-02T12:00:00Z",
        "fields": {"value": 25.5}
    },
    {
        "measurement": "humidity",
        "tags": {"location": "room1"},
        "time": "2024-02-02T12:00:00Z",
        "fields": {"value": 60}
    },
]

print(">>>>>> START")
check_thread_alive()

print(">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> create InfluxDBClient")
client = InfluxDBClient(
    url="localhost:8086",
    org="test",
    token="LLxHucOTfD1zSHVSadNRJRzZh_nNZGNf1KrHCieOj847ucB2RcLCBJZogP2zNtMxvAZMlAWsSOHgSJChaO7b3A==")
check_thread_alive()

print(">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> create write_api")
batch_write_api = client.write_api(
    write_options=WriteOptions(
        write_type=WriteType.batching))
check_thread_alive()

print(">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> close write_api")
batch_write_api.close()
batch_write_api.flush()
check_thread_alive()

print(">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> close InfluxDBClient")
client.close()
check_thread_alive()

print(">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wait 5 s")
time.sleep(5)
check_thread_alive()
print(">>>>>> STOP")

Expected behavior

When we close the batching API, associated threads are closed

Actual behavior

When we close the batching API, the associated thread ThreadPoolExecutor is alway alive

Additional info

My output

>>>>>> START
1 threads running
 > MainThread
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> create InfluxDBClient
1 threads running
 > MainThread
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> create write_api
2 threads running
 > MainThread
 > ThreadPoolExecutor-1_0
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> close write_api
2 threads running
 > MainThread
 > ThreadPoolExecutor-1_0
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> close InfluxDBClient
2 threads running
 > MainThread
 > ThreadPoolExecutor-1_0
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wait 5 s
2 threads running
 > MainThread
 > ThreadPoolExecutor-1_0
>>>>>> STOP

[bednar]

Hello @Dev-HexoTech,

Thank you for using our client and for bringing this issue to our attention. Your feedback is valuable to us.

We are prioritizing a review of the matter you've reported and will take a closer look at it as soon as possible. Please stay tuned for updates as we work to understand and address the issue.

If you have any more information or further insights that could help in our investigation, please feel free to share.

Regards

[bednar]

Hello,

Following up on the issue you reported regarding the incorrect disposal of ThreadPoolScheduler, we have implemented a fix in our development branch. The details of the fix can be found in this pull request: PR #641.

To ensure that this solution effectively resolves the issue you encountered, we kindly ask if you could verify the fix on your end. You can do so by installing the development version of the client that includes this update. Here's the command to install it:

pip install git+https://github.com/influxdata/influxdb-client-python.git@fix-scheduler

This will allow you to test the changes directly. Your feedback on whether this update addresses the problem as expected would be highly valuable to us.

Thank you very much for your assistance and for helping us improve the quality of our client.

Best regards.

[Dev-HexoTech]

Hello,

Thanks for this fix, but I have a little side effect. The ThreadPoolExecutor is not close after the call of the close method. So wee need to have an arbitrary sleep to wait the close of the thread.

We can see that in my code example. After the stop, the ThreadPoolExecutor is still hir but after the sleep of 5s, the thread is close

Thanks