Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Asia (Tokyo) Region Support & Parallel Benchmark Optimization #37

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Add Asia (Tokyo) Region Support & Parallel Benchmark Optimization #37

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
312882d
Add Asia (Tokyo) region support
gunta Oct 1, 2024
da68e39
Parallelize benchmark workflow with matrix strategy
gunta Oct 1, 2024
c2bbbaf
Add max parallel jobs flag to benchmark workflow
gunta Oct 1, 2024
888d478
Fix max_parallel type
gunta Oct 1, 2024
c343e70
Add Retry to benchmarks
gunta Oct 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 32 additions & 18 deletions .github/workflows/bench.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,26 +5,40 @@ on:
# Runs at 08:00 UTC every day
- cron: "0 8 * * *"
workflow_dispatch:
inputs:
max_parallel:
description: 'Maximum number of parallel jobs (1-20, use 1 for sequential execution)'
required: true
default: '10'
type: string

jobs:
run-benchmarks:
pre-warm:
runs-on: ubuntu-latest
steps:
- name: Pre-warm audio instance
run: curl -f -X POST "https://ai-benchmarks.fly.dev/bench?mode=audio&max_tokens=20&num_requests=1"

run-benchmarks:
needs: pre-warm
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
region: [sea, iad, cdg, nrt]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's fine to parallelize region since those are separate runners, but parallelizing medium will put additional load on the runners which we know can affect the measurements. I'd suggest either keeping each medium as a serial task or having a separate max-parallel config that we can use to experimentally roll this out.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I turned up in Singapore (sin) vs Narita for broader Asia coverage.

medium: [text, tools, image, audio]
max-parallel: ${{ fromJson(github.event.inputs.max_parallel || '10') }}

steps:
- name: Send Benchmark Requests
run: |
# Pre-warm any spun-down instances (only audio right now)
curl -f -X POST "https://ai-benchmarks.fly.dev/bench?mode=audio&max_tokens=20&num_requests=1"
# Run the benchmarks
declare -A max_tokens=(
["tools"]=100
)
default_max_tokens=20
regions=("sea" "iad" "cdg")
media=("text" "tools" "image" "audio")
for region in "${regions[@]}"; do
for medium in "${media[@]}"; do
max_tokens=${max_tokens[$medium]:-$default_max_tokens}
curl -f -X POST "https://ai-benchmarks.fly.dev/bench?mode=$medium&max_tokens=$max_tokens&spread=30&store" -H "fly-prefer-region: $region"
done
done
- name: Send Benchmark Request
env:
MAX_TOKENS: ${{ (matrix.medium == 'tools' && 100) || 20 }}
uses: nick-fields/retry@v3
with:
timeout_minutes: 15
max_attempts: 3
retry_on: error
command: |
echo "Running benchmark for ${{ matrix.medium }} in region ${{ matrix.region }}"
curl -f -X POST "https://ai-benchmarks.fly.dev/bench?mode=${{ matrix.medium }}&max_tokens=${{ env.MAX_TOKENS }}&spread=30&store" -H "fly-prefer-region: ${{ matrix.region }}"
echo "Benchmark completed for ${{ matrix.medium }} in region ${{ matrix.region }}"
2 changes: 1 addition & 1 deletion utils/GenerateLatestData.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
// Define the regions to search for files
const regions = ["cdg", "sea", "iad"];
const regions = ["cdg", "sea", "iad", "nrt"];

// Function to fetch data from a given URL
async function fetchData(url) {
Expand Down
9 changes: 9 additions & 0 deletions website/src/components/DataGrid.astro
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,15 @@ import "ag-grid-community/styles/ag-theme-quartz.css";
/>
<label for="cdgRegionSelector">Europe (Paris)</label>
</div>
<div>
<input
type="radio"
id="nrtRegionSelector"
name="selectedRegion"
value="nrt"
/>
<label for="cdgRegionSelector">Asia (Tokyo)</label>
</div>
</div>
</div>
<div class="ml-8 flex flex-col">
Expand Down