Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add created column to main cpanstats table #24

Open
preaction opened this issue Jun 15, 2018 · 1 comment
Open

Add created column to main cpanstats table #24

preaction opened this issue Jun 15, 2018 · 1 comment

Comments

@preaction
Copy link
Member

The main cpanstats table (CPAN::Testers::Schema::Result::Stats) has two existing date/time-ish fields:

  • postdate, which is a YYYYMM stored as a mediumint
  • fulldate, which is a YYYYMMDDHHNN stored as a varchar

Neither of these is ideal, since date/time comparisons in MySQL want ISO8601 date/time strings (which the datetime column can store). Using non-standard date/time formats makes it much harder to do ad-hoc reporting based on time periods (like pulling sections of the report summaries in Grafana dashboards, or for generating historical statistics for stats.cpantesters.org).

We should add a new column, created, to the cpanstats table. This column should be a datetime column, and should be initially populated based on the fulldate column (using 00 for the seconds field). This new column should be automatically populated by CPAN::Testers::Schema::ResultSet::Stats->insert_test_report, the standard API for inserting data into the cpanstats table.

It is possible to get a more accurate date/time for the report by using the test_reports table for new reports, or the metabase.metabase table for older reports, but the metabase table will make it much more difficult (and if knowing the seconds is that important, we can always fix it later).

Once we have this new column, we can start updating any other code that uses postdate or fulldate to instead use created (like CPAN::Testers::Schema::ResultSet::Release->since and CPAN::Testers::API::Controller::Summary->summary). While visiting other repositories, consider moving any useful code here to the schema.

@preaction
Copy link
Member Author

For example, here's a SQL query I needed to write in Grafana:

SELECT
  CAST( UNIX_TIMESTAMP( STR_TO_DATE( fulldate, '%Y%m%d%H%i' ) ) / 900 AS signed ) * 900 as time,
  COUNT(tester) AS value,
  tester AS metric
FROM cpanstats
WHERE fulldate >= DATE_FORMAT( $__timeFrom(), '%Y%m%d%H%i' )
    AND fulldate <= DATE_FORMAT( $__timeTo(), '%Y%m%d%H%i' )
GROUP BY 1, tester
ORDER BY 1

I had to upgrade to Grafana 5.1, because the 4.3 we were on doesn't support $__timeFrom() and $__timeTo(), without which this would not be possible. If we had a simple created column, this query would be able to take full advantage of Grafana's SQL macros, and would look like:

SELECT
  $__timeGroup( created, '15m' ) as time,
  COUNT(tester) AS value,
  tester AS metric
FROM cpanstats
WHERE $__timeFilter( created )
GROUP BY 1, tester
ORDER BY 1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant