We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
I'm observing huge performance hits when collecting values into a an object by using reduce:
reduce
gojq:
$ time gojq -n '[range(20000)] | reduce .[] as $item ({}; .[($item | tostring)] = {})' > /dev/null real 0m49.581s user 0m0.015s sys 0m0.031s
jq:
$ time jq -n '[range(20000)] | reduce .[] as $item ({}; .[($item | tostring)] = {})' > /dev/null real 0m0.156s user 0m0.109s sys 0m0.062s
It seems that a new map is allocated (and filled) on every iteration:
package main import ( "os" "runtime/pprof" "github.com/itchyny/gojq" ) func main() { fd, err := os.Create("profile.pprof") if err != nil { panic(err) } defer fd.Close() pprof.StartCPUProfile(fd) defer pprof.StopCPUProfile() query, err := gojq.Parse(`[range(20000)] | reduce .[] as $item ({}; .[($item | tostring)] = {})`) if err != nil { panic(err) } code, err := gojq.Compile(query) if err != nil { panic(err) } iter := code.Run(nil) for { v, ok := iter.Next() if !ok { break } if err, ok := v.(error); ok { panic(err) } } }
(pprof) list gojq.updateObject Total: 78.57s ROUTINE ======================== github.com/itchyny/gojq.updateObject in C:\Users\chrha\go\pkg\mod\github.com\itchyny\[email protected]\func.go 1.75s 34.67s (flat, cum) 44.13% of Total . . 1604:func updateObject(v map[string]any, k string, path []any, n any, a allocator) (any, error) { . . 1605: x, ok := v[k] . . 1606: if !ok && n == struct{}{} { . . 1607: return v, nil . . 1608: } . . 1609: u, err := update(x, path, n, a) . . 1610: if err != nil { . . 1611: return nil, err . . 1612: } . . 1613: if a.allocated(v) { . . 1614: v[k] = u . . 1615: return v, nil . . 1616: } . 830ms 1617: w := a.makeObject(len(v) + 1) 770ms 6.66s 1618: for k, v := range v { 980ms 27.17s 1619: w[k] = v . . 1620: } . 10ms 1621: w[k] = u . . 1622: return w, nil . . 1623:} . . 1624: . . 1625:func updateArrayIndex(v []any, i int, path []any, n any, a allocator) (any, error) { . . 1626: var x any
The text was updated successfully, but these errors were encountered:
No branches or pull requests
I'm observing huge performance hits when collecting values into a an object by using
reduce
:gojq:
jq:
It seems that a new map is allocated (and filled) on every iteration:
The text was updated successfully, but these errors were encountered: