Output: status, progress, measurements, reports, errors, and logs

[Manfred]

PerfCheck currently communicates at the user in two ways: a Logger instance and exceptions. This implementation makes sense in the context of a terminal application with a command-line interface but doesn't work well when communicating through a CI or desktop application.

There are a number of different types of information that PerfCheck needs to output:

1. Status of the run (started, finished, failed, broken, etc)
2. Progress of the run (git: Checking out `master', bundler: Bundling, http: Request /companies 243.3ms, etc)
3. Measurements and results in structured format (request_path: '/companies', latency: 24.3, query_count: 12, body: 'Hi!', diff: "-Hello\n+Hi!")
4. Formatted reports (Your branch is 20% less awesome than master)
5. Errors (source of the error, message, and structured details)
6. Logs (Rails server logs)

We don't want to keep doing this over the logger interface because it restricts the freedom we have in the UI when presenting this information.

Short term future

Because PerfCheck currently runs in the same process space as PerfCheck CI we can easily define some Interface in Ruby to pass this information back.

For example:

perf_check.publish(
  :measurement,
  request_path: '/sessions/new',
  latency: 24.3
)
perf_check.publish(
  :progress,
  "Checking out `#{branch}'
  branch: branch
)
perf_check.publish(
  :error,
  "Failed to check out `#{branch}'",
  branch: branch,
  output: stderr.read
)
perf_check.publish(
  :error,
  "Failed to check out `#{branch}'",
  branch: branch,
  output: stderr.read
)
perf_check.publish(
  :log, output,
  job_uuid: @uuid,
  name: 'Rails server log'
)

On the other end we can subscribe to these publish events and do something with it.

perf_check.on_published(:measurement, ->(**details) {
  perf_check.logger.info(
    PerfCheck::Measurement.table.row(
      *details.slice(:request_path, :latency)
    )
  )
})
perf_check.on_published(:progress, ->(message, **details) {
  perf_check.logger.info(message)
})

perf_check.on_published(:log, ->(output, **attributes) {
  Job.find_by!(uuid: job_uuid).logs.create!(attributes)
})

Long term future

At some point we might switch to a distributed architecture where PerfCheck runs in multiple instances and reports back to PerfCheck CI through HTTP or ZeroMQ or something.

perf_check.on_published(:measurement, ->(**details) {
  perf_check.zero_mq.send_string(JSON.dump(details))
})

[sudara]

What are the benefits of going an evented route vs. returning data from the CLI in a format like json? I think long ago we were goin to roll with having a --json option for the CLI... I'm guessing evented callbacks would allow us to do progressive UI updating and/or be more resilient to errors?

[Manfred]

Yes, those are the considerations.