Customizing or hiding Benchmark.ips's output format

[HertzDevil]

Benchmark.ips always prints its result to STDOUT in a nicely formatted table:

Benchmark.ips do |b|
  b.report("foo") { Array.new(1000, 0) }
  b.report("bar") { Array.new(5000, 0) }
end

foo   2.33M (429.67ns) (± 1.14%)  3.96kB/op        fastest
bar 906.57k (  1.10µs) (± 0.81%)  19.5kB/op   2.57× slower

But often I want to use those numbers somewhere else, e.g. plotting a graph, and to avoid parsing the above table again, I take the returned Benchmark::IPS::Job (the docs don't really mention this) and post-process the items into a JSON:

ips = Benchmark.ips { ... }
j = JSON.build do |json|
  json.object do
    json.field "items" do
      json.array do
        ips.items.each do |item|
          json.object do
            json.field "label", item.label
            json.field "mean", item.mean
            json.field "stddev", item.stddev
            json.field "bytes_per_op", item.bytes_per_op
          end
        end
      end
    end
  end
end

IMO this is still not straightforward enough, because I don't want to pollute STDOUT at all. It would be nice if Benchmark.ips itself supports those alternative formats out of the box. So I suggest implementing any of the following:

• Add a boolean parameter to suppress the interactive and final printing of results in Benchmark::IPS::Job#execute and #report, respectively.
• Allow passing an IO to Benchmark.ips and perform all the printing on that IO. Then one can pass File.open(File::NULL, "w") or a custom no-op IO to suppress the output.
• Create some kind of reporter interface and delegate printing to that. Something like:

  module Benchmark::IPS
    module Reporter
      abstract def report(items : Array(IPS::Entry), interactive : Bool) : Nil
      abstract def interactive? : Bool
    end
  
    class TableReporter
      include Reporter
  
      getter? interactive : Bool
  
      def initialize(@io : IO = STDOUT, @interactive : Bool = true)
      end
  
      def report(items : Array(IPS::Entry), interactive : Bool) : Nil
        # ...
        @io.print "\e[#{items.size}A" if interactive
      end
    end
  
    class Job
      property reporter : Reporter
  
      def initialize(calculation = 5, warmup = 2, interactive = STDOUT.tty?)
        # ...
        @reporter = TableReporter.new(STDOUT, !!interactive)
      end
  
      def report : Nil
        @reporter.report(ran_items, interactive: false)
      end
  
      private def run_calculation
        @items.each do |item|
          # ...
          if @reporter.interactive?
            run_comparison
            @reporter.report(ran_items, interactive: true)
          end
        end
      end
    end
  end

  Then I could write:

  class MyJsonReporter
    include Benchmark::IPS::Reporter
  
    def initialize(@io : IO = STDOUT)
    end
  
    def interactive? : Bool
      false
    end
  
    def report(items : Array(IPS::Entry), interactive : Bool)
      JSON.build(@io) { ... }
    end
  end
  
  Benchmark.ips do |b|
    b.reporter = MyJsonReporter.new
    # ...
  end

  Additionally, TableReporter and MyJsonReporter are both testable in isolation, without needing to mock a full benchmark run.

Similar points probably hold for Benchmark.bm as well.

[yxhuvud]

I slightly question the exposure of raw Array(IPS::Entry) rather than something that would allow a replacement of the inner structures.

That said it would be really cool if it were easier to export it to hdr histogram format so that it can be put into plotter .

[HertzDevil]

Benchmark::IPS::Entry is "public" insofar as it isn't marked as :nodoc:. Indeed a lot of things under Benchmark should have been protected or hidden.

ran_items is always a fresh Array, but the entries themselves are mutable and have reference semantics, so perhaps they have to be duplicated.

[straight-shoota]

I like the reporter pattern. It enables different formats.