OOM when working with 1 GB PostgreSQL heap (table) file

[pnsafonov]

What version are you using (fq -v)?

$ fq -v
0.0.9 (linux amd64)

How was fq installed?

fq is build from source.

My branch:
https://github.com/pnsafonov/fq/tree/postgres

Can you reproduce the problem using the latest release or master branch?

I can reproduce problem on my branch, where PostgreSQL parsers are implemented.

1 GB file:
https://github.com/pnsafonov/fq_testdata_postgres14
https://github.com/pnsafonov/fq_testdata_postgres14/raw/master/16397

What did you do?

I am trying to parse a heap (relation, table) file of maximum size (1 GB for PostgeSQL).
fq consumes 90-100 times memory than file size. For 80 mb file fq requires 7,5 GB RAM.

time fq -d pgheap -o flavour=postgres14 ".Pages[0].PageHeaderData.pd_linp[0, 1, 2, -1] | tovalue" 16397
Killed

real    0m50.794s
user    1m11.962s
sys     0m8.994s

Kenel messages:

sudo dmesg | tail -2
[193541.830725] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-1.scope,task=fq,pid=454783,uid=1000
[193541.830748] Out of memory: Killed process 454783 (fq) total-vm:31508780kB, anon-rss:26629332kB, file-rss:272kB, shmem-rss:0kB, UID:1000 pgtables:58860kB oom_score_adj:0

1 GB File:

ls -alh 16397
-rw-r----- 1 pavel pavel 1.0G Aug 31 08:38 16397

Memory profiler results:

go tool pprof mem.prof
File: postgres.test
Type: alloc_space
Time: Aug 30, 2022 at 6:16pm (MSK)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top 40
Showing nodes accounting for 31041.34MB, 97.35% of 31886.71MB total
Dropped 270 nodes (cum <= 159.43MB)
Showing top 40 nodes out of 87
      flat  flat%   sum%        cum   cum%
16563.23MB 51.94% 51.94% 17983.80MB 56.40%  github.com/wader/fq/pkg/decode.(*D).TryFieldScalarFn.func1
 5007.15MB 15.70% 67.65%  5007.15MB 15.70%  github.com/wader/fq/pkg/decode.(*D).fieldDecoder
 2168.92MB  6.80% 74.45%  3948.79MB 12.38%  github.com/wader/fq/pkg/decode.(*D).FillGaps
 2109.38MB  6.62% 81.06%  2109.38MB  6.62%  github.com/wader/fq/pkg/decode.(*D).AddChild (inline)
 1327.83MB  4.16% 85.23%  1327.83MB  4.16%  github.com/wader/fq/pkg/ranges.Gaps
  872.54MB  2.74% 87.96% 26209.93MB 82.20%  github.com/wader/fq/pkg/decode.(*D).FieldStruct
  587.02MB  1.84% 89.81%   587.02MB  1.84%  internal/reflectlite.Swapper
  513.59MB  1.61% 91.42%   513.59MB  1.61%  fmt.Sprintf
  425.01MB  1.33% 92.75%  1012.03MB  3.17%  github.com/wader/fq/pkg/decode.(*Value).postProcess.func1
  324.51MB  1.02% 93.77%   324.51MB  1.02%  github.com/wader/fq/pkg/bitio.NewSectionReader (inline)
     288MB   0.9% 94.67%   372.51MB  1.17%  github.com/wader/fq/pkg/decode.(*D).TryFieldScalarU16.func1
  236.02MB  0.74% 95.41%   696.61MB  2.18%  github.com/wader/fq/pkg/scalar.RawSym
  174.50MB  0.55% 95.96%  9928.56MB 31.14%  github.com/wader/fq/format/postgres/flavours/postgres14/common14.decodeInfomask
  104.50MB  0.33% 96.29%  2701.76MB  8.47%  github.com/wader/fq/format/postgres/flavours/postgres14/common14.decodeTChoice.func1
      61MB  0.19% 96.48%  1440.14MB  4.52%  github.com/wader/fq/format/postgres/flavours/postgres14/common14.decodeInfomask2
   56.53MB  0.18% 96.65%  1510.17MB  4.74%  github.com/wader/fq/format/postgres/flavours/postgres14/common14.DecodeItemIds.func1
      48MB  0.15% 96.80%   311.03MB  0.98%  github.com/wader/gojq.(*compiler).compileCallInternal
      48MB  0.15% 96.95%   460.59MB  1.44%  github.com/wader/fq/pkg/scalar.glob..func10.1
   46.51MB  0.15% 97.10%   410.05MB  1.29%  github.com/wader/gojq.(*compiler).compileFuncDef
      24MB 0.075% 97.18%   369.99MB  1.16%  github.com/wader/gojq.(*compiler).compileTerm
   15.05MB 0.047% 97.22% 31404.72MB 98.49%  github.com/wader/gojq.(*env).Next
       7MB 0.022% 97.25%   169.67MB  0.53%  github.com/wader/gojq.(*compiler).compileCall
       7MB 0.022% 97.27%   186.53MB  0.58%  github.com/wader/gojq.(*compiler).compileIf
       5MB 0.016% 97.28%   358.54MB  1.12%  github.com/wader/gojq.(*compiler).compile
    4.51MB 0.014% 97.30%   222.25MB   0.7%  github.com/wader/fq/pkg/interp.(*Interp).Eval.func2
    4.50MB 0.014% 97.31%   375.04MB  1.18%  github.com/wader/gojq.(*compiler).compileQuery
    3.50MB 0.011% 97.32%   205.43MB  0.64%  github.com/wader/gojq.(*compiler).compileFunc
       3MB 0.0094% 97.33% 26212.93MB 82.21%  github.com/wader/fq/format/postgres/flavours/postgres14/common14.decodeHeapPages
       3MB 0.0094% 97.34% 26212.93MB 82.21%  github.com/wader/fq/pkg/decode.(*D).FieldArray
    1.50MB 0.0047% 97.35% 26198.29MB 82.16%  github.com/wader/fq/format/postgres/flavours/postgres14/common14.decodeHeapPage
    0.50MB 0.0016% 97.35%   545.91MB  1.71%  github.com/wader/gojq.Compile
    0.50MB 0.0016% 97.35%   164.87MB  0.52%  github.com/wader/gojq.(*compiler).compileCallPc
         0     0% 97.35% 26212.93MB 82.21%  github.com/wader/fq/format/postgres.decodePgheap
         0     0% 97.35% 26080.73MB 81.79%  github.com/wader/fq/format/postgres/flavours/pgproee14.DecodeHeap (inline)
         0     0% 97.35%  2322.01MB  7.28%  github.com/wader/fq/format/postgres/flavours/pgproee14/ee14.DecodePageHeaderData
         0     0% 97.35% 26212.93MB 82.21%  github.com/wader/fq/format/postgres/flavours/postgres14/common14.DecodeHeap
         0     0% 97.35%  2235.26MB  7.01%  github.com/wader/fq/format/postgres/flavours/postgres14/common14.DecodeItemIds
         0     0% 97.35%  5454.55MB 17.11%  github.com/wader/fq/format/postgres/flavours/postgres14/common14.decodeTChoice
         0     0% 97.35%   887.09MB  2.78%  github.com/wader/fq/format/postgres/flavours/postgres14/common14.decodeTChoice.func1.1
         0     0% 97.35%  1419.14MB  4.45%  github.com/wader/fq/format/postgres/flavours/postgres14/common14.decodeTChoice.func2

I check with disassembler. TryFieldScalarFn.func1 is unnamed lambda in TryFieldScalarFn.

[wader]

Hello, at the moment fq has not been optimized to use less memory, focus has been to make it work at all :) The main reason it uses lots of memory at the moment is that it does very little "lazy" decoding, instead it will decode the full file and each field added will have to keep track of lots of metadata, it's name, parent, children, bit range, decode value, optional symbolic value, optional description string etc.

I'm not familiar with the pghep format but for example the mp4 decoder in fq has a option to skip decoding of individual samples if that not interesting (you can still decode individual samples manually) which speeds up decoding a lot.

But there are some options to improve the situation that i've thought about, most of them sadly quite complicated and not a easy task:

• Try to compact down decode.Value and scalar.S somehow. They are the ones using the most memory i think.
  • Add some kind of slice to keep track of optional things like symbolic value, display format etc
• Introduce interfaces for decode.Value and/or scalar.S
  • Can have type specific implementations that store less data
  • Can have implementation that only store actual value etc
  • Possible could have arrays that knows type and length
• Do lazy decoding somehow
  • Might behave strange when there are errors or broken files as it's not noticed until you do a query etc
  • Not sure how it would interact with probing and possible some other things
  • Probably would make some queries slower as you have to decode and possibly do IO as "query time"
• Do dull decode but only store ranges and then late decode again
  • Would detect errors
  • Probably would make some queries slower as you have to decode and possibly do IO as "query time"

Let me know if you have other ideas