Skip to content

Commit

Permalink
Add: workflow to pack all results every day into a single archive
Browse files Browse the repository at this point in the history
This compresses the JSON by 1:100, meaning we pay hunderd times
less for long-term storage.
  • Loading branch information
TrueBrain committed Jan 5, 2024
1 parent 94f33c1 commit 52e1627
Showing 1 changed file with 61 additions and 0 deletions.
61 changes: 61 additions & 0 deletions .github/workflows/pack-results.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
name: Pack Survey Results

on:
schedule:
- cron: '0 5 * * *'
workflow_dispatch:
inputs:
date:
description: 'Date of the survey to pack'
required: true
default: '2024-01-01'

jobs:
publish:
runs-on: ubuntu-latest

name: Pack Survey Results

steps:
- name: Checkout
uses: actions/checkout@v3

- name: Install rclone
shell: bash
run: |
curl -sL https://rclone.org/install.sh | sudo bash
rclone config create --non-interactive --no-obscure openttd s3 \
provider Cloudflare \
access_key_id ${{ secrets.R2_SURVEY_ACCESS_KEY_ID }} \
secret_access_key ${{ secrets.R2_SURVEY_SECRET_ACCESS_KEY }} \
endpoint ${{ secrets.R2_SURVEY_ENDPOINT }} \
acl private
no_check_bucket true
- name: Download, pack, and upload survey results
shell: bash
run: |
date="${{ inputs.date }}"
if [ -z "$date" ]; then
date=$(date -d "yesterday" +%Y-%m-%d)
fi
year=$(date -d "${date}" +%Y)
month=$(date -d "${date}" +%m)
echo "Packing all results for ${date}"
echo "::group::Download survey results"
rclone copy -v openttd:survey-prod/${date} ${date}
echo "::endgroup::"
echo "::group::Pack survey results"
tar --xz -cfv ${date}.tar.xz ${date}
echo "::endgroup::"
echo "::group::Upload survey result pack"
rclone copy -v ${date}.tar.xz openttd:survey-packed-prod/${year}/${month}/${date}.tar.xz
echo "::endgroup::"
echo "::group::Remove survey results"
rclone delete -v openttd:survey-prod/${date}
echo "::endgroup::"

0 comments on commit 52e1627

Please sign in to comment.