Skip to content

Latest commit

 

History

History
132 lines (101 loc) · 7.82 KB

README.md

File metadata and controls

132 lines (101 loc) · 7.82 KB
Raw

SmashScan

This project is intended to perform video analysis on pre-recorded footage of the game Super Smash Bros Melee. Once the project has progressed enough, I plan to integrate it with the SmashVods video database.

TODO List

If you are interested in helping out, join this repository's Slack and send a message.

  • Research ways to detect the percent counter (Either template matching, OCR, or neural networks)
  • Research ways to detect stock icons (Either template matching, feature matching, or neural networks)
  • Research ways to detect the timer (Either template matching, OCR, or neural networks)
  • Add additional game capabilities (SSB64, SSB4, PM, Rivals)

Milestones

  1. Train a neural network (DarkNet) to search for the six legal SSBM stages, regardless of a stream overlay.
  2. Improve video download speed and create a stage-timeline object to represent SSBM matches.
  3. Added non-stage training data to significantly improve stage-detection results.
  4. Added percent template matching to segment the video for quicker stage-detection.

Instalation Guide

  1. Create a Python 3.6 environment (required by tensorflow) locally or in a conda environment. Install the following python packages; The version numbers are not necessarily required, I just provided them as a reference for when I wrote this guide: numpy (1.11.0), matplotlib (1.5.1), cython (0.28.5), opencv-python (3.4.2.17), youtube-dl (2018.8.22 - 2018 version much faster than 2015-2017).
  2. Install tensorflow if you don't have an NVIDIA GPU. Otherwise, install tensorflow-gpu. I followed Mark Jay's tutorial for Ubuntu 18.04.
  3. Install the DarkFlow repository globally. I followed Mark Jay's tutorials to understand the basics of the DarkFlow.
  4. Clone this repo and create an annotations, output, and videos folder inside. The .gitignore doesn't like committing "empty" folders.

Testing Guide

Obtain the YouTube video-id for the video you wish to test on. The video-id is the set of characters after the "v=" in a YouTube video URL. Afterwards, use download.py to save the video to the videos directory.

# https://www.youtube.com/watch?v=dQw4w9WgXcQ
python download.py dQw4w9WgXcQ example-video-name.mp4

Use test.py to draw bounding boxes on the video. This uses the pre-trained weights (.pb) in the cfg folder.

python test.py example-video-name.mp4

Training Guide

Obtain the YouTube video-id for the video you wish to train on. The video-id is the set of characters after the "v=" in a YouTube video URL. Afterwards, use download.py to save the video to the videos directory.

# https://www.youtube.com/watch?v=dQw4w9WgXcQ
python download.py dQw4w9WgXcQ example-video-name.mp4

Use label.py to save frames to the images directory and labels to the annotations directory. The key-bindings used for label.py are listed below.

python label.py example-video-name.mp4

, - move backward 1 frame
. - move forward 1 frame
direction - move 10 frames
alt+direction - move 100 frames
ctrl+direction - move 1000 frames

left-click -> release - draw bounding box
r - begin recording once you are satisfied with the box drawn
enter - end recording and specify stage number in terminal
` - exit the labeler and save annotations & frames
q - exit the labeler without saving annotations & frames

Opening the images folder freezes my computer because of how many images it tries to display at once. If your running Linux, a useful command to run to see the most recently updated file in a directory is as follows.

ls -t images | head -1

# Eye of GNOME graphics viewer. Just an image viewer program.
eog 123456.png

Once you are satisfied with the new labels, move the images and annotations folders into the train folder of your darkflow installation directory. Then run the following two commands in separate terminals.

# Training command.
flow --model cfg/tiny-yolo-voc-6c.cfg --load bin/tiny-yolo-voc.weights --train --annotation train/annotations --dataset train/images --epoch 100 --gpu 1.0 --summary summary

# Optional command to track the progress visually.
tensorboard --logdir summarytrain

After some time, you can cancel training by hitting ctrl+c. The DarkFlow trainer saves checkpoints every couple minutes, so you aren't losing much progress. If I plan on training for a long period of time (over 12 hours), I'll typically set the checkpoints to be saved over a longer time span. AKA, I will add the flag --save 40000. which will save checkpoints every 2500 iterations (40000/16, where batch size = 16).

This is because DarkFlow has an issue with gradient descent over long periods of time and training will get stuck at NaN. DarkFlow's default.py recommends rmsprop, but I tried adam and sgd with no success. It seems to be a common issue with the trainer from the issues I've read on the repository :/

Next, convert the most recent checkpoint into something usable by SmashScan.

flow --model cfg/tiny-yolo-voc-6c.cfg --load -1 --savepb

The .pb (Protocol Buffer) format is a method for serializing structured data. The DarkFlow library saves the .meta and .pb files to the built_graph directory. Copy both of these files into the cfg directory of the SmashScan directory.

Label History

Below are a list of tournaments I trained on for the initial release. The size of these annotation-image pairs is >10GB, so I don't plan on uploading them anytime soon. This list is mainly for referencing what tournaments have been used.

Tournament Annotation Range Frames
GOML 2018 1 - 5019 5019
EVO 2018 5020 - 8274 3255
Low Tier City 6 8275 - 9246 972
Smash Factor 7 9247 - 12485 3239
Saints Gaming Live 2018 12486 - 14019 1534
Smash N Splash 4 14020 - 15720 1701
Dreamhack Austin 2018 15721 - 17299 1579
Combo Breaker 2018 17300 - 19091 1792
Momocon 2018 19092 - 20562 1471
Pound Underground 2018 20563 - 22747 2185
Smash Summit 6 22747 - 24852 2106
Aegis 2018 24853 - 26005 1153
Flatiron 3 26006 - 27383 1378
Noods Oakland 27384 - 29554 2171
The Mango 29555 - 30976 1422
No Fun Allowed 2 30977 - 33133 2157
The Gang 33134 - 34861 1730
Noods Noods Noods 34862 - 36760 1899
Valhalla 36761 - 38503 1743
Player Cams 38503 - 41547 3044
Holiday Bash 2017 41547 - 42628 1082
Smash Valley VII 42629 - 43944 1316
Heir 5 43945 - 45062 1117
Momocon 2018 (R) 45063 - 46170 1108
No Fun Allowed 2 (R) 46171 - 48285 2115
Poi Poundaz 48286 - 49782 1497