Using send_blocks_stream_query with the timeout option causes memory leak on iroha

[shazz]

Hi,

I don't know if this is an iroha or a iroha-python issue but in brief, I was using send_blocks_stream_query to monitor some transactions using a separate thread and to get a change to stop the thread I was using the timeout parameter set to 2sec.

    def get_blocks(self, callback):

        if self.thread is None and self.task is None:
            from threading import Thread
            self.task = GetBlocksTask(self)

            t = Thread(target = self.task.run, args=(callback, ))
            t.start()
            self.thread = t
        else:
            raise ValueError("Thread already started, stop it first")

    def stop_tasks(self):
        self.task.terminate()
        self.thread.join()

class GetBlocksTask:

    def __init__(self, helper):
        self._running = True
        self.helper = helper

    def terminate(self):
        self._running = False

    def run(self, callback):
        query = self.helper.iroha.blocks_query()
        IrohaCrypto.sign_query(query, self.helper.private_key)

        while self._running:
            try:
                for block in self.helper.net.send_blocks_stream_query(query, timeout=2):
                    callback(block)
            except Exception as e:
                # print("weird way to stop the thread...")
                pass

        print("Stopping loop")

From the python client point of view it works well but after a day, I saw that the iroha server memory increased from 20Mb to 2GB and more than 26K pids where created (one per 2s):

CONTAINER ID   NAME             CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
f54613a786e1   prometheus       0.00%     146.9MiB / 7.776GiB   1.84%     35.3MB / 2.46MB   0B / 4.1kB        8
00ea27268cbc   cadvisor         2.91%     108.2MiB / 7.776GiB   1.36%     3.8MB / 229MB     217kB / 0B        15
7c8a719e3648   explorer         0.00%     50MiB / 7.776GiB      0.63%     5.85kB / 314B     1.39MB / 0B       12
0aa1ad98c8fb   iroha0           0.02%     2.32GiB / 7.776GiB    29.84%    97.2MB / 107MB    20.5kB / 0B       26346
6ba0c59e1475   iroha1           0.02%     13.88MiB / 7.776GiB   0.17%     103MB / 112MB     20.5kB / 0B       15
36df6fdf5dd7   iroha2           0.02%     10.41MiB / 7.776GiB   0.13%     66MB / 64.3MB     20.5kB / 0B       13
dc8036cc6350   some-postgres0   0.00%     80.86MiB / 7.776GiB   1.02%     39.9MB / 17MB     1.68MB / 144MB    16
222d185b7a03   some-postgres1   0.00%     80.66MiB / 7.776GiB   1.01%     6MB / 4.82MB      815kB / 107MB     16
63fc52e43dbe   some-postgres2   0.00%     79.24MiB / 7.776GiB   1.00%     5.35MB / 4.56MB   6.38MB / 98.5MB   16
0f3d9f6fdef6   traefik          0.42%     34.68MiB / 7.776GiB   0.44%     197MB / 205MB     7.68MB / 188kB    11

So I stopped using the timeout parameter and I found another way to kill the thread but in any case, I get this is a serious bug.

Thanks!

[stepanLav]

Hi @shazz, thanks for the report.
I ran it on my latest version of Iroha for reproducing. Could you clarify which version of Iroha you are using?

[shazz]

Hi @stepanLav!
I'm using the latest docker image: hyperledger/iroha:latest (so 1.2.1 I guess)