HSTforU: Anomaly Detection in Aerial and Ground-based Videos with Hierarchical Spatio-Temporal Transformer for U-net

This is the code for HSTforU: Anomaly Detection in Aerial and Ground-based Videos with Hierarchical Spatio-Temporal Transformer for U-net .

Project | Video | Paper (coming soon...)

Related Works

MoGuP: See MoGuP:Motion-guided Prediction for Video Anomaly Detection.

ASTNet: See Attention-based Residual Autoencoder for Video Anomaly Detection.

Updates

• [8/19/2023] Code of HSTforU is released (with password). We will remove the password after this paper is accepted.

Setup

The code can be run under any environment with Python 3.7 and above. (It may run with lower versions, but we have not tested it).

Install the required packages:

pip install -r requirements.txt

Clone this repo:

git clone https://github.com/vt-le/HSTforU.git
cd HSTforU/

We evaluate HSTforU on:

Dataset	Link
UCSD Ped2
CUHK Avenue
ShanghaiTech
Drone-Anomaly

A dataset is a directory with the following structure:

$ tree data
ped2/avenue
├── training
│   └── frames
│       ├── ${video_1}$
│       │   ├── 000.jpg
│       │   ├── 001.jpg
│       │   └── ...
│       ├── ${video_2}$
│       │   ├── 00.jpg
│       │   └── ...
│       └── ...
├── testing
│   └── frames
│       ├── ${video_1}$
│       │   ├── 000.jpg
│       │   ├── 001.jpg
│       │   └── ...
│       ├── ${video_2}$
│       │   ├── 000.jpg
│       │   └── ...
│       └── ...
└── ped2/avenue.mat

shanghaitech
├── training
│   └── frames
│       ├── ${video_1}$
│       │   ├── 000.jpg
│       │   ├── 001.jpg
│       │   └── ...
│       ├── ${video_2}$
│       │   ├── 00.jpg
│       │   └── ...
│       └── ...
├── testing
│   └── frames
│       ├── ${video_1}$
│       │   ├── 000.jpg
│       │   ├── 001.jpg
│       │   └── ...
│       ├── ${video_2}$
│       │   ├── 000.jpg
│       │   └── ...
│       └── ...
└── test_frame_mask
    ├── 01_0014.npy
    ├── 01_0015.npy
    └── ...

drone
├──bike
│  ├──training
│  │  └── frames
│  │      ├── ${video_1}$
│  │      │   ├── 0.jpg
│  │      │   ├── 1.jpg
│  │      │   └── ...
│  │      ├── ${video_2}$
│  │      │   ├── 00.jpg
│  │      │   └── ...
│  │      └── ...
│  ├── testing
│  │   └── frames
│  │       ├── ${video_1}$
│  │       │   ├── 000.jpg
│  │       │   ├── 001.jpg
│  │       │   └── ...
│  │       ├── ${video_2}$
│  │       │   ├── 000.jpg
│  │       │   └── ...
│  │       └── ...
│  └── annotation
│      ├── 01.npy
│      ├── 0.npy
│      └── ...
├── highway
│   ├── ...
└── ...

Evaluation

Please first download the pre-trained model

Dataset	Pretrained Model
UCSD Ped2
CUHK Avenue
ShanghaiTech
Drone-Anomaly

To evaluate a pretrained HSTforU on a dataset, run:

 python test.py \
    --cfg <path/to/config/file> \
    --pretrained </path/to/pre-trained/model> \
    [--batch-size <batch-size> --tag <job-tag>]

For example, to evaluate HSTforU on Ped2:

python test.py \
    --cfg config/scripts/ped2/ped2_pvt2_hst.yaml \
    --model-file output/HSTforU/ped2_pvt2_hst/ckpt_ped2.pth

Configuration

• We use YAML for configuration.
• We provide a couple preset configurations.
• Please refer to config.py for documentation on what each configuration does.

Citing

If you find our work useful, please consider citing:

@article{hstforu,
title = {HSTforU: Anomaly Detection in Aerial and Ground-based Videos with Hierarchical Spatio-Temporal Transformer for U-net},
author = {Le, Viet-Tuan and Jin, Hulin and Kim, Yong-Guk},
}

Contact

For any question, please file an issue or contact:

Viet-Tuan Le: [email protected]