Memory Leak?

[willtebbutt]

MWE (less connection to a database with a large table):

using LibPQ

# This works fine if you run it _before_ doing LibPQ stuff.
# Allocates about 28GB of RAM (fine on my machine if not much else
# is going on. Tailor this to consume lots of memory, but not more
# than you might reasonably have available in a Julia session with
# nothing else going on).
# Note that this function doesn't return anything, so Julia is free
# to use the memory for other stuff once it's run.
function allocate_large_array_less_than_RAM()
    x = fill(1.0, 3_500_000_000)
    return nothing
end

# Nothing escapes this function, so memory should be available for
# Julia to use once execution is complete.
# The idea is that the amount that this allocates + however much
# allocate_large_array_less_than_RAM allocates should be more than
# the amount of RAM available.
# This ensures that, if the problem is indeed LibPQ failing to free up
# some memory that has been handed over to C, Julia will complain about
# not having enough memory when we run this, followed by
# allocate_large_array_less_than_RAM.
function get_data()
    conn = <set-up-connection>
    result = execute(
        conn,
        "SELECT column from large_table;";
        binary_format=true,
    )
    data = LibPQ.columntable(result)
    return nothing
end

println("Allocating a large array -- this runs fine before we run get_data")
allocate_large_array_less_than_RAM()
println("Getting lots of data")
get_data()
println("Trying to allocate a large array again. Yields an OutOfMemory error on my machine")
allocate_large_array_less_than_RAM()

# Running get_data 10 times seems to increase memory usage.
# On my machine, this eventually kills my process, and it appears to be
# because memory usage continually increases with each iteration
# (I'm just staring at htop).
for _ in 1:10
    get_data()
end

Based on the above, this looks like a memory leak.

For reference, I have 32GB of RAM and 0 swap.

edit: julia 1.7.3, LibPQ 1.13.0

[ImreSamu]

question:

• calling Base.close() Method ( "Clean up the memory used by the PGresult object. The Result will no longer be usable." ) - helps?

As I see - there are 10 new "conn = " .. without "close()"

[willtebbutt]

Interestingly, this doesn't seem to make a difference (tested locally, the issue persists).

I think the fact that we have finalisers explains this.
The PGResult has a finaliser attached to it here, so that shouldn't be a problem.
Additionally, the connections appear to have a finaliser attached to them here, and note that the implementation of Base.close for connections appears to do similar things to what the finaliser does (I don't know enough about how @async is handled to know if there are differences though).

[willtebbutt]

The other thing to say is that my system appears to be allocating quite a lot more memory than I would have anticipated. In particular I'm downloading roughly 160 million Float64s, which should occupy (very roughly) 1.5GB of RAM. In practice I think roughly 5.5GB is being allocated -- I don't know whether this is at all relevant, but though I would mention it in case it is.

[tpgillam]

I can't reproduce this on Julia 1.7.2, LibPQ 1.13.0 . Running on MacOS 12.4 under Rosetta.

This is the code I've run - I'm using a data-generating query instead of reading data from a table:

using LibPQ

function connection(f, dsn::AbstractString)
    conn = LibPQ.Connection(dsn)
    return try
        f(conn)
    finally
        close(conn)
    end
end

_localhost_dsn() = "<snip>"  # My postgres server connection details
local_connection(f) = connection(f, _localhost_dsn())

function get_data(n::Integer, not_null::Bool)
    return local_connection() do conn
        r = execute(
            conn,
            "SELECT s.data::float8 FROM generate_series(0,$n,0.1) AS s(data)";
            binary_format=true,
            not_null,
        )
        LibPQ.columntable(r)
        return nothing
    end
end

for i in 1:10
    println(i)
    get_data(4_000_000, false)
    GC.gc()
end

What type & how many rows were you retrieving in your get_data? If I increase the number of rows in the example above by a factor of 10, it still seems to be OK.

FYI, in my case I tried both not_null=true and not_null=false, and didn't notice any material difference.

The only thing that seemed to make a temporary difference was commenting out the call to GC.gc(). But if I made that call after the loop instead, I'd get back to somewhere sane.