| jupytext | kernelspec | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
\newcommand{\detectset}{\mathcal{D}}
\newcommand{\gauss}{\mathcal{N}}
\newcommand{\prob}{\mathbb{P}}
\newcommand{\esp}{\mathbb{E}}
\newcommand{\var}{\mathbb{V}}
\newcommand{\eqsp}{\,}
\newcommand{\Rset}{\mathbb{R}}
\newcommand{\filtdist}{\mathbb{Q}}
\newcommand{\transdist}{\mathbb{M}}
\newcommand{\SMCfiltdist}{\widehat{\filtdist}^{SMC}}
\newcommand{\rmd}{\mathrm{d}}
\newcommand{\predictdist}{\mathbb{L}}
\newcommand{\SMCpredictdist}{\widehat{\predictdist}^{SMC}}
\newcommand{\MCpredictdist}{\widehat{\predictdist}}
\newcommand{\likel}{\mathbb{G}}
\newcommand{\hatN}{\mathsf{\hat{N}}}
\newcommand{\N}{\mathsf{N}}
\newcommand{\Nmis}{\mathsf{\hat{N}_{mis}}}
\newcommand{\Nred}{\mathsf{\hat{N}_{red}}}
\newcommand{\Nfalse}{\mathsf{\hat{N}_{false}}}
\newcommand{\Ntrue}{\mathsf{\hat{N}_{true}}}
\newcommand{\gtlabels}{i \in[\![1, \N]\!]}
\newcommand{\predlabels}{j \in [\![1, \hatN]\!]}
\newcommand{\detre}{\mathsf{DetRe}}
\newcommand{\detpr}{\mathsf{DetPr}}
\newcommand{\assre}{\mathsf{AssRe}}
\newcommand{\asspr}{\mathsf{AssPr}}
\newcommand{\tp}{\mathsf{TP}}
\newcommand{\fp}{\mathsf{FP}}
\newcommand{\fn}{\mathsf{FN}}
\newcommand{\tpa}{\mathsf{TPA}}
\newcommand{\fpa}{\mathsf{FPA}}
\newcommand{\gtID}{\mathsf{gtID}}
\newcommand{\prID}{\mathsf{prID}}
\newcommand{\fna}{\mathsf{FNA}}
\newcommand{\countre}{\mathsf{CountRe}}
\newcommand{\countpr}{\mathsf{CountPr}}
\newcommand{\tempwindow}{[\![n-\lfloor \kappa/2 \rfloor, n+\lfloor \kappa/2 \rfloor]\!]}
+++
Litter is a known cause of degradation in marine environments and most of it travels in rivers before reaching the oceans. In this paper, we present a novel algorithm to assist waste monitoring along watercourses. While several attempts have been made to quantify litter using neural object detection in photographs of floating items, we tackle the more challenging task of counting directly in videos using boat-embedded cameras. We rely on multi-object tracking (MOT) but focus on the key pitfalls of false and redundant counts which arise in typical scenarios of poor detection performance. Our system only requires supervision at the image level and performs Bayesian filtering via a state space model based on optical flow. We present a new open image dataset gathered through a crowdsourced campaign and used to train a center-based anchor-free object detector. Realistic video footage assembled by water monitoring experts is annotated and provided for evaluation. Improvements in count quality are demonstrated against systems built from state-of-the-art multi-object trackers sharing the same detection capabilities. A precise error decomposition allows clear analysis and highlights the remaining challenges.
+++
Litter pollution concerns every part of the globe.
Each year, almost ten thousand million tons of plastic waste is generated, among which 80% ends up in landfills or in nature {cite}geyer2017, notably threatening all of the world’s oceans, seas and aquatic environments {cite}welden2020,gamage2020.
Plastic pollution is known to already impact more than 3763 marine species worldwide (see this detailed analysis) with risk of proliferation through the whole food chain.
This accumulation of waste is the endpoint of the largely misunderstood path of trash, mainly coming from land-based sources {cite}rochman2016, yet rivers have been identified as a major pathway for the introduction of waste into marine environments {cite}jambeck2015.
Therefore, field data on rivers and monitoring are strongly needed to assess the impact of measures that can be taken. The analysis of such field data over time is pivotal to understand the efficiency of the actions implemented such as choosing zero-waste alternatives to plastic, designing new products to be long-lasting or reusable, introducing policies to reduce over-packing.
Different methods have already been tested to monitor waste in rivers: litter collection and sorting on riverbanks {cite}Bruge2018, visual counting of drifting litter from bridges {cite}gonzales2021, floating booms {cite}gasperi2014 and nets {cite}moritt2014.
All are helpful to understand the origin and typology of litter pollution yet hardly compatible with long term monitoring at country scales.
Monitoring tools need to be reliable, easy to set up on various types of rivers, and should give an overview of plastic pollution during peak discharge to help locate hotspots and provide trends.
Newer studies suggest that plastic debris transport could be better understood by counting litter trapped on river banks, providing a good indication of the local macrolitter pollution especially after increased river discharge {cite}VanEmmerik2019,VanEmmerik2020.
Based on these findings, we propose a new method for litter monitoring which relies on videos of river banks directly captured from moving boats.
In this case, object detection with deep neural networks (DNNs) may be used, but new challenges arise.
First, available data is still scarce.
When considering entire portions of river banks from many different locations, the variety of scenes, viewing angles and/or light conditions is not well covered by existing plastic litter datasets like {cite}Proenca2020, where litter is usually captured from relatively close distances and many times in urban or domestic backgrounds. Therefore, achieving robust object detection across multiple conditions is still delicate.
Second, counting from videos is a different task than counting from independent images, because individual objects will typically appear in several consecutive frames, yet they must only be counted once. This last problem of association has been extensively studied for the multi-object tracking (MOT) task, which aims at recovering individual trajectories for objects in videos. When successful MOT is achieved, counting objects in videos is equivalent to counting the number of estimated trajectories. Deep learning has been increasingly used to improve MOT solutions {cite}Ciaparrone2020b. However, newer state-of-the-art techniques require increasingly heavy and costly supervision, typically all object positions provided at every frame. In addition, many successful techniques {cite}bergmann2019 can hardly be used in scenarios with abrupt and nonlinear camera motion. Finally, while research is still active to rigorously evaluate performance at multi-object tracking {cite}luiten2020, most but not all aspects of the latter may affect global video counts, which calls for a separate evaluation protocol dedicated to multi-object counting.
Our contribution can be summarized as follows.
- We provide a novel open-source image dataset of macro litter, which includes various objects seen from different rivers and different contexts. This dataset was produced with a new open-sourced platform for data gathering and annotation developed in conjunction with Surfrider Foundation Europe, continuously growing with more data.
- We propose a new algorithm specifically tailored to count in videos with fast camera movements. In a nutshell, DNN-based object detection is paired with a robust state space movement model which uses optical flow to perform Bayesian filtering, while confidence regions built on posterior predictive distributions are used for data association. This framework does not require video annotations at training time: the multi-object tracking module does not require supervision, only the DNN-based object detection does require annotated images. It also fully leverages optical flow estimates and the uncertainty provided by Bayesian predictions to recover object identities even when detection recall is low. Contrary to existing MOT solutions, this method ensures that tracks are stable enough to avoid repeated counting of the same object.
- We provide a set of video sequences where litter counts are known and depicted in real conditions. For these videos only, litter positions are manually annotated at every frame in order to carefully analyze performance. This allows us to build new informative count metrics. We compare the count performance of our method against other MOT-based alternatives.
A first visual illustration of the second claim is presented via the following code chunks: on three selected frames, we present a typical scenario where our strategy can avoid overcounting the same object (we depict internal workings of our solution against the end result of the competitors).
:tags: [hide-input]
from myst_nb import glue
import matplotlib
import matplotlib.pyplot as plt
import os
import pandas as pd
from surfnet.prepare_data import download_data
from surfnet.track import default_args as args
params = {'legend.fontsize': 'xx-large',
'axes.labelsize': 'xx-large',
'axes.titlesize':'xx-large',
'xtick.labelsize':'xx-large',
'ytick.labelsize':'xx-large'}
plt.rcParams.update(params)
# download frames and detections from a given deep detector model
download_data()
# prepare arguments
args.external_detections = True
args.data_dir = 'data/external_detections/part_1_segment_0'
args.output_dir = 'surfnet/results'
args.noise_covariances_path = 'surfnet/data/tracking_parameters'
args.confidence_threshold = 0.5
args.algorithm = 'EKF'
args.ratio = 4
args.display = 0
:tags: [hide-input]
import pickle
import numpy as np
from surfnet.tracking.utils import resize_external_detections, write_tracking_results_to_file
from surfnet.tools.video_readers import FramesWithInfo
from surfnet.tracking.trackers import get_tracker
from surfnet.track import track_video
# Initialize variances
transition_variance = np.load(os.path.join(args.noise_covariances_path, 'transition_variance.npy'))
observation_variance = np.load(os.path.join(args.noise_covariances_path, 'observation_variance.npy'))
# Get tracker algorithm
engine = get_tracker(args.algorithm)
# Open data: detections and frames
with open(os.path.join(args.data_dir, 'saved_detections.pickle'),'rb') as f:
detections = pickle.load(f)
with open(os.path.join(args.data_dir, 'saved_frames.pickle'),'rb') as f:
frames = pickle.load(f)
# Create frame reader and resize detections
reader = FramesWithInfo(frames)
detections = resize_external_detections(detections, args.ratio)
# Start tracking, storing intermediate tracklets
results, frame_to_trackers = track_video(reader, detections, args, engine,
transition_variance, observation_variance, return_trackers=True)
# Write final results
write_tracking_results_to_file(results, ratio_x=args.ratio, ratio_y=args.ratio, output_filename=args.output_dir)
:tags: [hide-input, remove-output]
from surfnet.track import build_image_trackers
# Choose a few indices to display (same for our algorithm and SORT)
idxs = [108, 112, 117]
considered_frames = [frames[i] for i in idxs]
considered_trackers = [frame_to_trackers[i] for i in idxs]
glue('demo_ours', build_image_trackers(considered_frames, considered_trackers, args, reader), display=False)
:name: "demo-ours"
*Our method*: one object (red dot) is correctly detected at every frame and given a consistent identity throughout the sequence with low location uncertainty (red ellipse). Next to it, a false positive detection is generated at the first frame (brown dot) but immediatly lost in the following frames: the associated uncertainty grows fast (brown ellipse). In our solution, this type of track will not be counted. A third correctly detected object (pink) appears in the third frame and begins a new track.
:tags: [hide-input]
## Tracker with SORT
from collections import defaultdict
import cv2
from sort.sort import track as sort_tracker
print('Tracking with SORT...')
print('--- Begin SORT internal logs')
sort_tracker(detections_dir='data/external_detections', output_dir='sort/results')
print('--- End')
def read_sort_output(filename):
""" Reads the output .txt of Sort (or other tracking algorithm)
"""
dict_frames = defaultdict(list)
with open(filename) as f:
for line in f:
items = line[:-1].split(",")
frame = int(items[0])
objnum = int(items[1])
x = float(items[2])
y = float(items[3])
dict_frames[int(items[0])].append((objnum, x, y))
return dict_frames
def build_image(frames, trackers, image_shape=(135,240), downsampling=2*4):
""" Builds a full image with consecutive frames and their displayed trackers
frames: a list of K np.array
trackers: a list of K trackers. Each tracker is a per frame list of tracked objects
"""
K = len(frames)
assert len(trackers) == K
font = cv2.FONT_HERSHEY_COMPLEX
output_img=np.zeros((image_shape[0], image_shape[1]*K, 3), dtype=np.uint8)
object_ids = []
for tracker in trackers:
for detection in tracker:
object_ids.append(detection[0])
min_object_id = min(object_ids)
for i in range(K):
frame = cv2.cvtColor(cv2.resize(frames[i], image_shape[::-1]), cv2.COLOR_BGR2RGB)
for detection in trackers[i]:
cv2.putText(frame, f'{detection[0]-min_object_id +1}', (int(detection[1]/downsampling)+10, int(detection[2]/downsampling)+10), font, 0.5, (255, 0, 0), 1, cv2.LINE_AA)
output_img[:,i*image_shape[1]:(i+1)*image_shape[1],:] = frame
return output_img
:tags: [hide-input, remove-output]
# open sort output
tracker_file = "sort/results/part_1_segment_0.txt"
frame_to_track = read_sort_output(tracker_file)
condisered_frames = [frames[idx] for idx in idxs]
considered_tracks = [frame_to_track[i] for i in idxs]
out_img = build_image(condisered_frames, considered_tracks)
plt.figure(figsize=(15,6))
plt.imshow(out_img)
plt.axis("off");
glue('sort_demo', plt.gcf(), display=False)
:name: "demo-sort"
*SORT*: the resulting count is also 2, but both counts arise from tracks generated by the same object, the latter not re-associated at all in the second frame. Additionally, the third object is discarded (in post-processing) by their strategy.
+++
Counting from images has been an ongoing challenge in computer vision.
Most works can be divided into (i) detection-based methods where objects are individually located for counting, (ii) density-based methods where counts are obtained by summing a predicted density map, and (iii) regression-based methods where counts are directly regressed from input images {cite}Chattopadhyay.
While some of these works tackled the problem of counting in wild scenes {cite}Arteta2016, most are focused on pedestrian and crowd counting.
Though several works {cite}wu2020fast,Xiong2017,Miao2019 showed the relevance of leveraging sequential inter-frame information to achieve better counts at every frame, none of these methods actually attempt to produce global video counts.
Automatic macro litter monitoring in rivers is still a relatively nascent initiative, yet there have already been several attempts at using DNN-based object recognition tools to count plastic trash.
Recently, {cite}Proenca2020 used a combination of two Convolutional Neural Networks (CNNs) to detect and quantify plastic litter using geospatial images from Cambodia.
In {cite}Wolf2020, reliable estimates of plastic density were obtained using Faster R-CNN {cite}ren2016faster on images extracted from bridge-mounted cameras.
For underwater waste monitoring, {cite}vanlieshout2020automated assembled a dataset with bounding box annotations, and showed promising performance with several object detectors.
They later turned to generative models to obtain more synthetic data from a small dataset {cite}Hong2020.
While proving the practicality of deep learning for automatic waste detection in various contexts, these works only provide counts for separate images of photographed litter.
To the best of our knowledge, no solution has been proposed to count litter directly in videos.
Multi-object tracking usually involves object detection, data association and track management, with a very large number of methods already existing before DNNs {cite}luo2021.
MOT approaches now mostly differ in the level of supervision they require for each step: until recently, most successful methods (like {cite}Bewley2016) have been detection-based, i.e.
involving only a DNN-based object detector trained at the image level and coupled with an unsupervised data association step.
In specific fields such as pedestrian tracking or autonomous driving, vast datasets now provide precise object localisation and identities throughout entire videos {cite}Caesar2020, Dendorfer2020.
Current state-of-the-art methods leverage this supervision via deep visual feature extraction {cite}Wojke2018,Zhanga or even self-attention {cite}Chu2021 and graph neural networks {cite}Wang2021.
For these applications, motion prediction may be required, yet well-trained appearance models are usually enough to deal with detection failures under simple motion, therefore the linear constant-velocity assumption often prevails ({cite}Ciaparrone2020b).
In the case of macrolitter monitoring, however, available image datasets are still orders of magnitude smaller, and annotated video datasets do not exist at all.
Even more so, real shooting conditions induce chaotic movements on the boat-embedded cameras.
A close work of ours is that of {cite}Fulton2018, who paired Kalman filtering with optical flow to yield fruit count estimates on entire video sequences captured by moving robots.
However, their video footage is captured at night with consistent lighting conditions, backgrounds are largely similar across sequences, and camera movements are less challenging.
In our application context, we find that using MOT for the task of counting objects requires a new movement model, to take into account missing detections and large camera movements.
+++
Our main dataset of annotated images is used to train the object detector. Then, only for evaluation purposes, we provide videos with annotated object positions and known global counts. Our motivation is to avoid relying on training data that requires this resource-consuming process.
With help from volunteers, we compile photographs of litter stranded on river banks after increased river discharge, shot directly from kayaks navigating at varying distances from the shore. Images span multiple rivers with various levels of water current, on different seasons, mostly in southwestern France. The resulting pictures depict trash items under the same conditions as the video footage we wish to count on, while spanning a wide variety of backgrounds, light conditions, viewing angles and picture quality.
For object detection applications, the images are annotated using a custom online platform where each object is located using a bounding box. In this work, we focus only on litter counting without classification, however the annotated objects are already classified into specific categories which are described in .
A few samples are depicted below:
:tags: [hide-input]
from PIL import Image, ExifTags
from pycocotools.coco import COCO
def draw_bbox(image, anns, ratio):
"""
Display the specified annotations.
"""
for ann in anns:
[bbox_x, bbox_y, bbox_w, bbox_h] = (ratio*np.array(ann['bbox'])).astype(int)
cv2.rectangle(image, (bbox_x,bbox_y),(bbox_x+bbox_w,bbox_y+bbox_h), color=(0,0,255),thickness=3)
return image
dir = 'surfnet/data/images'
ann_dir = os.path.join(dir,'annotations')
data_dir = os.path.join(dir,'images')
ann_file = os.path.join(ann_dir, 'subset_of_annotations.json')
coco = COCO(ann_file)
imgIds = np.array(coco.getImgIds())
print('{} images loaded'.format(len(imgIds)))
for imgId in imgIds:
plt.figure()
image = coco.loadImgs(ids=[imgId])[0]
try:
image = Image.open(os.path.join(data_dir,image['file_name']))
# Rotation of the picture in the Exif tags
for orientation in ExifTags.TAGS.keys():
if ExifTags.TAGS[orientation]=='Orientation':
break
exif = image._getexif()
if exif is not None:
if exif[orientation] == 3:
image=image.rotate(180, expand=True)
elif exif[orientation] == 6:
image=image.rotate(270, expand=True)
elif exif[orientation] == 8:
image=image.rotate(90, expand=True)
except (AttributeError, KeyError, IndexError):
# cases: image don't have getexif
pass
image = cv2.cvtColor(np.array(image.convert('RGB')), cv2.COLOR_RGB2BGR)
annIds = coco.getAnnIds(imgIds=[imgId])
anns = coco.loadAnns(ids=annIds)
h,w = image.shape[:-1]
target_h = 1080
ratio = target_h/h
target_w = int(ratio*w)
image = cv2.resize(image,(target_w,target_h))
image = draw_bbox(image,anns,ratio)
image = cv2.cvtColor(image,cv2.COLOR_BGR2RGB)
plt.imshow(image)
plt.axis('off')
For evaluation, an on-field study was conducted with 20 volunteers to manually count litter along three different riverbank sections in April 2021, on the Gave d'Oloron near Auterrive (Pyrénées-Atlantiques, France), using kayaks. The river sections, each 500 meters long, were precisely defined for their differences in background, vegetation, river current, light conditions and accessibility (see this section for aerial views of the shooting site and details on the river sections). In total, the three videos amount to 20 minutes of footage at 24 frames per second (fps) and a resolution of 1920x1080 pixels.
On video footage, we manually recovered all visible object trajectories on each river section using an online video annotation tool (more details here for the precise methodology). From that, we obtained a collection of distinct object tracks spanning the entire footage.
+++
Our counting method is divided into several interacting blocks. First, a detector outputs a set of predicted positions for objects in the current frame. The second block is a tracking module designing consistent trajectories of potential objects within the video. At each frame, a third block links the successive detections together using confidence regions provided by the tracking module, proposing distinct tracks for each object. A final postprocessing step only keeps the best tracks which are enumerated to yield the final count.
In most benchmarks, the prediction quality of object attributes like bounding boxes is often used to improve tracking.
For counting, however, point detection is theoretically enough and advantageous in many ways.
First, to build large datasets, a method which only requires the lightest annotation format may benefit from more data due to annotation ease.
Second, contrary to previous popular methods {cite}ren2016faster involving intricate mechanisms for bounding box prediction, center-based and anchor-free detectors {cite}Zhou2019, Law only use additional regression heads which can simply be removed for point detection.
Adding to all this, {cite}Zhanga highlight conceptual and experimental reasons to favor anchor-free detection in tracking-related tasks.
For these reasons, we use a stripped version of CenterNet {cite}Zhou2019 where offset and bounding box regression heads are discarded to output bare estimates of center positions on a coarse grid.
An encoder-decoder network takes an input image fisher2017.
In a video, for each frame $I_n \in [0,1]^{w \times h \times 3}$ (where $n$ indexes the frame number), the detector outputs a set $\detectset_n = {z_n^i}{1 \leq i \leq D_n}$ where each
(detector-training)=
Training the detector is done similarly as in {cite}Proenca2020.
For every image, the corresponding set
where
$$ \mathcal{L}(\hat{Y},Y) = -\sum_{x,y} \gamma_{xy}^\beta\left(1-\hat{p}{xy}\right)^\alpha \log{\left(\hat{p}{xy}\right)}, $$
where
$$ (\hat{p}{xy},\gamma{xy}) = \left{ \begin{array}{ll} (\hat{Y}{xy},1) & \mbox{if } Y{xy} = 1, \ (1 - \hat{Y}{xy},1 - Y{xy}) & \mbox{otherwise.} \end{array} \right. $$
(bayesian-tracking)=
Between two timesteps paragios2006:
$$ \widetilde{I}n[u] = \widetilde{I}{n-1}[u+\Delta_n(u)], $$
where, in our case, farneback2003two for details.
(state-space-model)=
Using optical flow as a building block, we posit a state space model where estimates of
:label: state-transition-eq
X_k = X_{k-1} + \Delta_k(\lfloor X_{k-1} \rfloor) + \eta_k
and
where $(\eta_k){k\geq 1}$ are i.i.d. centered Gaussian random variables with covariance matrix $Q$ independent of $(\varepsilon_k){k\geq 1}$ i.i.d. centered Gaussian random variables with covariance matrix
Denoting sarkka2013bayesian and references therein). Most SMC methods have been widely studied and shown to be very effective even in presence of strongly nonlinear dynamics and/or non-Gaussian noise, however such sample-based solutions are computationally intensive, especially in settings where many objects have to be tracked and false positive detections involve unnecessary sampling steps. On the other hand, UKF requires fewer samples and provides an intermediary solution in presence of mild nonlinearities. In our setting, we find that a linearisation of the model yields approximation which is computationally cheap and as robust on our data:
$$
X_k = X_{k-1} + \Delta_k(\lfloor \mu_{k-1} \rfloor) + \partial_X\Delta_k(\lfloor \mu_{k-1} \rfloor)(X_{k-1}-\mu_{k-1}) + \eta_k .
$$
where
This allows the implementation of Kalman updates on the linearised model, a technique named extended Kalman filtering (EKF). For a more complete presentation of Bayesian and Kalman filtering, please refer to this appendix. On the currently available data, we find that the optical flow estimates are very informative and accurate, making this approximation sufficient. For completeness, we present here an SMC-based solution and discuss the empirical differences and use-cases where the latter might be a more relevant choice.
In any case, the state space model naturally accounts for missing observations, as the contribution of
The full MOT algorithm consists of a set of single-object trackers following the previous model, but each provided with distinct observations at every frame. These separate filters provide track proposals for every object detected in the video.
(data-association)=
Throughout the video, depending on various conditions on the incoming detections, existing trackers must be updated (with or without a new observation) and others might need to be created. This setup requires a third party data association block to link the incoming detections with the correct filters.
At the frame
- For every detected object
$z_n^i \in \detectset_n$ and every filter$\ell$ , compute$P(i,\ell) = \prob(Z_n^\ell \in V_\delta(z_n^i)\mid Z^\ell_{1:n-1})$ where$V_\delta(z)$ is the neighborhood of$z$ defined as the squared area of width$2\delta$ centered on$z$ (see this appendix for exact computations). - Using the Hungarian algorithm {cite}
kuhn, compute the assignment between detections and filters with$P$ as cost function, but discarding associations$(i,\ell)$ having$P(i,\ell) < \rho$ . Formally,$\rho$ represents the level of a confidence region centered on detections and we use$\rho = 0.5$ . Denote$a_{\rho}$ the resulting assignment map defined as$a_{\rho}(i) = \ell$ if$z_n^i$ was associated with the$\ell$ -th filter, and$a_{\rho}(i) = 0$ if$z_n^i$ was not associated with any filter. - For
$1 \leq i \leq D_n$ , if$a_{\rho}(i) = \ell$ , use$z_n^i$ as a new observation to update the$\ell$ -th filter. If$a_{\rho}(i) = 0$ , create a new filter initialized from the prior distribution, i.e. sample the true location as a Gaussian random variable with mean$z_n^i$ and variance$R$ . - For all filters
$\ell'$ which were not provided a new observation, update only the predictive law of $X^{\ell'}{n}$ given $Z^{\ell'}{1:n-1}$.
In other words, we seek to associate filters and detections by maximising a global cost built from the predictive distributions of the available filters, but an association is only valid if its corresponding predictive probability is high enough.
Though the Hungarian algorithm is a very popular algorithm in MOT, it is often used with the Euclidean distance or an Intersection-over-Union (IoU) criterion.
Using confidence regions for the distributions of
A visual depiction of the entire pipeline (from detection to final association) is provided below. This way of combining a set of Bayesian filters with a data association step that resorts on the most likely hypothesis is a form of Global Nearest Neighbor (GNN) tracking. Another possibility is to perform multi-target filtering by including the data association step directly into the probabilistic model, as in {cite}mahler2003. A generalisation of single-target recursive Bayesian filtering, this class of methods is grounded in the point process literature and well motivated theoretically. In case of strong false positive detection rates, close and/or reappearing objects, practical benefits may be obtained from these solutions. Finally, note that another well-motivated choice for
---
name: diagram
---
Visual representation of the tracking pipeline.
At the end of the video, the previous process returns a set of candidate tracks.
For counting purposes, we find that simple heuristics can be further applied to filter out tracks that do not follow actual objects.
More precisely, we observe that tracks of real objects usually contain more (i) observations and (ii) streams of uninterrupted observations.
Denote by
+++
Counting in videos using embedded moving cameras is not a common task, and as such it requires a specific evaluation protocol to understand and compare the performance of competing methods. First, not all MOT metrics are relevant, even if some do provide insights to assist evaluation of count performance. Second, considering only raw counts on long videos gives little information on which of the final counts effectively arise from well detected objects.
Popular MOT benchmarks usually report several sets of metrics such as ClearMOT {cite}bernardin2008 or IDF1 {cite}RistaniSZCT16 which can account for different components of tracking performance.
Recently, {cite}luiten2020 built the so-called HOTA metrics that allow separate evaluation of detection and association using the Jaccard index.
The following components of their work are relevant to our task (we provide equation numbers in the original paper for formal definitions).
First, when considering all frames independently, traditional detection recall (
In classical object detection, those metrics are the main target. In our context, as the first step of the system, this framewise performance impacts the difficulty of counting. However, we must keep in mind that these metrics are computed framewise and might not guarantee anything at a video scale. The next points illustrate that remark.
- If both
$\detre$ and$\detpr$ are very high, objects are detected at nearly all frames and most detections come from actual objects. Therefore, robustness to missing observations is high, but even in this context computing associations may fail if camera movements are nontrivial. - For an ideal tracking algorithm which never counts individual objects twice and does not confuse separate objects in a video, a detector capturing each object for only one frame could theoretically be used.
Thus, low
$\detre$ could theoretically be compensated with robust tracking. - If our approach can rule out faulty tracks which do not follow actual objects, then good counts can still be obtained using a detector generating many false positives.
Again, this suggests that low
$\detpr$ may allow decent counting performance.
HOTA association metrics are built to measure tracking performance irrespective of the detection capabilities, by comparing predicted tracks against true object trajectories.
In our experiments, we compute the Association Recall (luiten2020, we denote with
See {cite}luiten2020 (fig. 2) for a clear illustration of these quantities.
In brief, a low
Nonetheless, association metrics are only computed for predicted tracks which can effectively be matched with ground truth tracks.
Consequently,
Denoting by
Define
Denote
-
$\Ntrue = \sum_{i=1}^{\N} 1_{|A_i| > 0}$ the number of ground truth objects successfully counted. -
$\Nred = \sum_{i=1}^{\N} |A_i| - \Ntrue$ the number of redundant counts per ground truth object. -
$\Nmis = \N - \Ntrue$ the number of ground truth objects that are never effectively counted. -
$\Nfalse = \sum_{j=1}^{\hatN} 1_{j \rightarrow \emptyset}$ the number of counts which cannot be associated with any ground truth object and are therefore considered as false counts.
Using these metrics provides a much better understanding of
$$
\hatN = \Ntrue + \Nred + \Nfalse,
$$
while
Conveniently, the quantities can be used to define the count precision (
which provide good summaries for the overall count quality, letting aside the tracking performance.
Note that these metrics and the associated decomposition are only defined if the previous assignment between predicted and ground truth tracks can be obtained.
In our case, predicted tracks never overlap with several ground truth tracks (because true objects are well separated), and therefore this assignment is straightforward. More involved metrics have been studied at the trajectory level (see for example {cite}garcia2020 and the references therein), though not specifically tailored to the restricted task of counting. For more complicated data, an adaptation of such contributions into proper counting metrics could be valuable.
Since the original validation set comprises only a few unequally long videos, only absolute results are available.
Splitting the original sequences into shorter independent sequences of equal length allows to compute basic statistics.
For any quantity
+++
We denote by
To demonstrate the benefits of our work, we select two multi-object trackers and build competing counting systems from them. Our first choice is SORT {cite}Bewley2016, which relies on Kalman filtering with velocity updated using the latest past estimates of object positions. Similar to our system, it only relies on image supervision for training, and though DeepSORT {cite}Wojke2018 is a more recent alternative with better performance, the associated deep appearance network cannot be used without additional video annotations. FairMOT {cite}Zhanga, a more recent alternative, is similarly intended for use with video supervision but allows self-supervised training using only an image dataset. Built as a new baseline for MOT, it combines linear constant-velocity Kalman filtering with visual features computed by an additional network branch and extracted at the position of the estimated object centers, as introduced in CenterTrack {cite}zhou2020. We choose FairMOT to compare our method to a solution based on deep visual feature extraction.
Similar to our work, FairMOT uses CenterNet for the detection part and the latter is therefore trained as in . We train it using hyperparameters from the original paper. The detection outputs are then shared between all counting methods, allowing fair comparison of counting performance given a fixed object detector. We run all experiments at 12fps, an intermediate framerate to capture all objects while reducing the computational burden.
+++
In the following section, we present the performance of the trained detector.
Having annotated all frames of the evaluation videos, we directly compute
:tags: [hide-input]
from IPython.display import display
import pandas as pd
fps = 12
fps = f'{fps}fps'
split = 'test'
long_segments_names = ['part_1_1',
'part_1_2',
'part_2',
'part_3']
indices_test = [0,7,9,13]
indices_val = [0,9,10,14]
indices_det = [0,17,24,38]
alpha_type = '___50'
def set_split(split):
if split == 'val':
indices = indices_val
elif split == 'test':
indices = indices_test
gt_dir_short = f'TrackEval/data/gt/surfrider_short_segments_{fps}'
eval_dir_short = f'TrackEval/data/trackers/surfrider_short_segments_{fps}'
if split is not None:
gt_dir_short += f'_{split}'
eval_dir_short += f'_{split}'
gt_dir_short += '/surfrider-test'
return indices, eval_dir_short, gt_dir_short
indices, eval_dir_short, gt_dir_short = set_split(split)
def get_det_values(index_start=0, index_stop=-1):
results_for_det = pd.read_csv(os.path.join(f'TrackEval/data/trackers/surfrider_short_segments_{fps}','surfrider-test','ours_EKF_1_kappa_1_tau_0','pedestrian_detailed.csv'))
results_det = results_for_det.loc[:,[f'DetRe{alpha_type}',f'DetPr{alpha_type}', f'HOTA_TP{alpha_type}',f'HOTA_FN{alpha_type}',f'HOTA_FP{alpha_type}']].iloc[index_start:index_stop]
results_det.columns = ['hota_det_re','hota_det_pr','hota_det_tp','hota_det_fn','hota_det_fp']
hota_det_re = results_det['hota_det_re']
hota_det_pr = results_det['hota_det_pr']
hota_det_tp = results_det['hota_det_tp']
hota_det_fn = results_det['hota_det_fn']
hota_det_fp = results_det['hota_det_fp']
denom_hota_det_re = hota_det_tp + hota_det_fn
denom_hota_det_pr = hota_det_tp + hota_det_fp
hota_det_re_cb = (hota_det_re * denom_hota_det_re).sum() / denom_hota_det_re.sum()
hota_det_pr_cb = (hota_det_pr * denom_hota_det_pr).sum() / denom_hota_det_pr.sum()
return [f'{100*hota_det_re_cb:.1f}', f'{100*hota_det_pr_cb:.1f}']
def get_table_det():
table_values = [get_det_values(index_start, index_stop) for (index_start, index_stop) in zip(indices_det[:-1],indices_det[1:])]
table_values.append(get_det_values())
return pd.DataFrame(table_values)
table_det = get_table_det()
table_det.columns = ['DetRe*','DetPr*']
table_det.index = ['S1','S2','S3','All']
display(table_det)
To fairly compare the three solutions, we calibrate the hyperparameters of our postprocessing block on the validation split and keep the values that minimize the overall count error
We report results using the count-related tracking metrics and count decompositions defined in the previous section. To provide a clear but thorough summary of the performance, we report
First, the higher values of AssRe confirm the robustness of our solution in assigning consistent tracks to individual objects. This is directly reflected into the count precision performance - with an overall value of
:tags: [hide-input]
def get_summary(results, index_start=0, index_stop=-1):
results = results.loc[:,[f'Correct_IDs{alpha_type}',f'Redundant_IDs{alpha_type}',f'False_IDs{alpha_type}',f'Missing_IDs{alpha_type}',f'Fused_IDs{alpha_type}', f'GT_IDs',f'HOTA_TP{alpha_type}',f'AssRe{alpha_type}']].iloc[index_start:index_stop]
results.columns = ['correct','redundant','false','missing','mingled','gt','hota_tp','ass_re']
ass_re = results['ass_re']
hota_tp = results['hota_tp']
ass_re_cb = (ass_re * hota_tp).sum() / hota_tp.sum()
correct = results['correct']
redundant = results['redundant']
false = results['false']
missing = results['missing']
# mingled = results['mingled']
# gt = results['gt']
# count_error = false + redundant - missing
summary = dict()
# summary['missing'], summary['missing_mean'], summary['missing_std'] = f'{missing.sum()}',f'{missing.mean():.1f}',f'{np.nan_to_num(missing.std()):.1f}'
# summary['false'], summary['false_mean'], summary['false_std'] = f'{false.sum()}', f'{false.mean():.1f}', f'{np.nan_to_num(false.std()):.1f}'
# summary['redundant'], summary['redundant_mean'], summary['redundant_std'] = f'{redundant.sum()}', f'{redundant.mean():.1f}', f'{np.nan_to_num(redundant.std()):.1f}'
# summary['gt'] = f'{gt.sum()}'
summary['ass_re_cb'] = f'{100*ass_re_cb:.1f}'
# summary['count_error'], summary['count_error_mean'], summary['count_error_std'] = f'{count_error.sum()}',f'{count_error.mean():.1f}',f'{np.nan_to_num(count_error.std()):.1f}'
count_re = correct.sum() / (correct + missing).sum()
summary['count_re_cb'] = f'{100*count_re:.1f}'
# count_re_mean = (correct / (correct + missing)).mean()
# summary['count_re_mean'] = f'{100*count_re_mean:.1f}'
count_re_std = (correct / (correct + missing)).std()
summary['count_re_std'] = f'{100*np.nan_to_num(count_re_std):.1f}'
count_pr = correct.sum() / (correct + false + redundant).sum()
summary['count_pr_cb'] = f'{100*count_pr:.1f}'
# count_pr_mean = (correct / (correct + false + redundant)).mean()
# summary['count_pr_mean'] = f'{100*count_pr_mean:.1f}'
count_pr_std = (correct / (correct + false + redundant)).std()
summary['count_pr_std'] = f'{100*np.nan_to_num(count_pr_std):.1f}'
return summary
def get_summaries(results, sequence_names):
summaries = dict()
for (sequence_name, index_start, index_stop) in zip(sequence_names, indices[:-1], indices[1:]):
summaries[sequence_name] = get_summary(results, index_start, index_stop)
summaries['All'] = get_summary(results)
return summaries
summaries = []
method_names = ['fairmot_kappa_7_tau_9', 'sort_kappa_7_tau_9','ours_EKF_1_kappa_7_tau_8']
pretty_method_names = ['FairMOT','SORT','Ours']
for tracker_name in method_names:
results = pd.read_csv(os.path.join(eval_dir_short,'surfrider-test',tracker_name,'pedestrian_detailed.csv'))
sequence_names = ['S1','S2','S3']
summaries.append(pd.DataFrame(get_summaries(results, sequence_names)).T)
fairmot_star, sort, ours = summaries
# nmis = '$\hat{\mathsf{N}}_{mis}$'
# mu_nmis ='$\hat{\mu}_{\hat{\mathsf{N}}_{mis}}$'
# sigma_nmis = '$\hat{\sigma}_{\hat{\mathsf{N}}_{mis}}$'
# nfalse = '$\hat{\mathsf{N}}_{false}$'
# mu_nfalse = '$\hat{\mu}_{\hat{\mathsf{N}}_{false}}$'
# sigma_nfalse = '$\hat{\sigma}_{\hat{\mathsf{N}}_{false}}$'
# nred = '$\hat{\mathsf{N}}_{red}$'
# mu_nred = '$\hat{\mu}_{\hat{\mathsf{N}}_{red}}$'
# sigma_nred = '$\hat{\sigma}_{\hat{\mathsf{N}}_{red}}$'
ass_re = '$\mathsf{AssRe}$'
count_re = '$\mathsf{CountRe}$'
count_pr = '$\mathsf{CountPr}$'
rows = ['fairmot_kappa_7_tau_9','sort_kappa_7_tau_9','ours_EKF_1_kappa_7_tau_8']
columns = ['ass_re','count_re', 'count_re_std', 'count_pr', 'count_pr_std']
for summary in summaries:
summary.columns = columns
results_S1 = pd.DataFrame([summary.iloc[0] for summary in summaries])
results_S1.index = method_names
results_S2 = pd.DataFrame([summary.iloc[1] for summary in summaries])
results_S2.index = method_names
results_S3 = pd.DataFrame([summary.iloc[2] for summary in summaries])
results_S3.index = method_names
results_All = pd.DataFrame([summary.iloc[3] for summary in summaries])
results_All.index = method_names
results = [results_S1, results_S2, results_S3, results_All]
results_S1.to_latex()
for sequence_name, result in zip(['S_1','S_2','S_3','All'], results):
for row in rows:
for col in columns:
glue(f'{sequence_name}_{row}_{col}', result.loc[row,col], display=False)
| FairMOT | {glue:text}S_1_fairmot_kappa_7_tau_9_ass_re
|
{glue:text}S_1_fairmot_kappa_7_tau_9_count_re
|
{glue:text}S_1_fairmot_kappa_7_tau_9_count_re_std
|
{glue:text}S_1_fairmot_kappa_7_tau_9_count_pr
|
{glue:text}S_1_fairmot_kappa_7_tau_9_count_pr_std
|
| Sort | {glue:text}S_1_sort_kappa_7_tau_9_ass_re
|
{glue:text}S_1_sort_kappa_7_tau_9_count_re
|
{glue:text}S_1_sort_kappa_7_tau_9_count_re_std
|
{glue:text}S_1_sort_kappa_7_tau_9_count_pr
|
{glue:text}S_1_sort_kappa_7_tau_9_count_pr_std
|
| Ours | {glue:text}S_1_ours_EKF_1_kappa_7_tau_8_ass_re
|
{glue:text}S_1_ours_EKF_1_kappa_7_tau_8_count_re
|
{glue:text}S_1_ours_EKF_1_kappa_7_tau_8_count_re_std
|
{glue:text}S_1_ours_EKF_1_kappa_7_tau_8_count_pr
|
{glue:text}S_1_ours_EKF_1_kappa_7_tau_8_count_pr_std
|
| FairMOT | {glue:text}S_2_fairmot_kappa_7_tau_9_ass_re
|
{glue:text}S_2_fairmot_kappa_7_tau_9_count_re
|
{glue:text}S_2_fairmot_kappa_7_tau_9_count_re_std
|
{glue:text}S_2_fairmot_kappa_7_tau_9_count_pr
|
{glue:text}S_2_fairmot_kappa_7_tau_9_count_pr_std
|
| Sort | {glue:text}S_2_sort_kappa_7_tau_9_ass_re
|
{glue:text}S_2_sort_kappa_7_tau_9_count_re
|
{glue:text}S_2_sort_kappa_7_tau_9_count_re_std
|
{glue:text}S_2_sort_kappa_7_tau_9_count_pr
|
{glue:text}S_2_sort_kappa_7_tau_9_count_pr_std
|
| Ours | {glue:text}S_2_ours_EKF_1_kappa_7_tau_8_ass_re
|
{glue:text}S_2_ours_EKF_1_kappa_7_tau_8_count_re
|
{glue:text}S_2_ours_EKF_1_kappa_7_tau_8_count_re_std
|
{glue:text}S_2_ours_EKF_1_kappa_7_tau_8_count_pr
|
{glue:text}S_2_ours_EKF_1_kappa_7_tau_8_count_pr_std
|
| FairMOT | {glue:text}S_3_fairmot_kappa_7_tau_9_ass_re
|
{glue:text}S_3_fairmot_kappa_7_tau_9_count_re
|
{glue:text}S_3_fairmot_kappa_7_tau_9_count_re_std
|
{glue:text}S_3_fairmot_kappa_7_tau_9_count_pr
|
{glue:text}S_3_fairmot_kappa_7_tau_9_count_pr_std
|
| Sort | {glue:text}S_3_sort_kappa_7_tau_9_ass_re
|
{glue:text}S_3_sort_kappa_7_tau_9_count_re
|
{glue:text}S_3_sort_kappa_7_tau_9_count_re_std
|
{glue:text}S_3_sort_kappa_7_tau_9_count_pr
|
{glue:text}S_3_sort_kappa_7_tau_9_count_pr_std
|
| Ours | {glue:text}S_3_ours_EKF_1_kappa_7_tau_8_ass_re
|
{glue:text}S_3_ours_EKF_1_kappa_7_tau_8_count_re
|
{glue:text}S_3_ours_EKF_1_kappa_7_tau_8_count_re_std
|
{glue:text}S_3_ours_EKF_1_kappa_7_tau_8_count_pr
|
{glue:text}S_3_ours_EKF_1_kappa_7_tau_8_count_pr_std
|
| FairMOT | {glue:text}All_fairmot_kappa_7_tau_9_ass_re
|
{glue:text}All_fairmot_kappa_7_tau_9_count_re
|
{glue:text}All_fairmot_kappa_7_tau_9_count_re_std
|
{glue:text}All_fairmot_kappa_7_tau_9_count_pr
|
{glue:text}All_fairmot_kappa_7_tau_9_count_pr_std
|
| Sort | {glue:text}All_sort_kappa_7_tau_9_ass_re
|
{glue:text}All_sort_kappa_7_tau_9_count_re
|
{glue:text}All_sort_kappa_7_tau_9_count_re_std
|
{glue:text}All_sort_kappa_7_tau_9_count_pr
|
{glue:text}All_sort_kappa_7_tau_9_count_pr_std
|
| Ours | {glue:text}All_ours_EKF_1_kappa_7_tau_8_ass_re
|
{glue:text}All_ours_EKF_1_kappa_7_tau_8_count_re
|
{glue:text}All_ours_EKF_1_kappa_7_tau_8_count_re_std
|
{glue:text}All_ours_EKF_1_kappa_7_tau_8_count_pr
|
{glue:text}All_ours_EKF_1_kappa_7_tau_8_count_pr_std
|
:tags: [hide-input]
set_split('test')
fig, axes = plt.subplots(1,3, figsize=(30,10), sharey=True)
for ax, title, tracker_name in zip(axes, pretty_method_names, method_names):
results = pd.read_csv(os.path.join(eval_dir_short,'surfrider-test',tracker_name,'pedestrian_detailed.csv'))
results = results.loc[:,[f'Redundant_IDs{alpha_type}',f'False_IDs{alpha_type}',f'Missing_IDs{alpha_type}']].iloc[:-1]
results.columns = ['redundant', 'false', 'missing']
results.loc[:,'missing'] = - results.loc[:,'missing']
results.columns = ['$\hat{\mathsf{N}}_{red}$', '$\hat{\mathsf{N}}_{false}$', '$-\hat{\mathsf{N}}_{mis}$']
results.plot(ax = ax, kind='bar', stacked=True, color=['orange', 'red', 'black'], title=title, xlabel='Sequence nb')
+++
We successfully tackled video object counting on river banks, in particular issues which could be addressed independently of detection quality. Moreover the methodology developed to assess count quality enables us to precisely highlight the challenges that pertain to video object counting on river banks. Conducted in coordination with Surfrider Foundation Europe, an NGO specialized on water preservation, our work marks an important milestone in a broader campaign for macrolitter monitoring and is already being used in a production version of a monitoring system. That said, large amounts of litter items are still not detected. Solving this problem is largely a question of augmenting the object detector training dataset through crowdsourced images. A specific annotation platform is online, thus the amount of annotated images is expected to continuously increase, while training is provided to volunteers collecting data on the field to ensure data quality. Finally, several expeditions on different rivers are already underway and new video footage is expected to be annotated in the near future for better evaluation. All data is made freely available. Future goals include downsizing the algorithm, a possibility given the architectural simplicity of anchor-free detection and the relatively low computational complexity of EKF. In a citizen science perspective, a fully embedded version for portable devices will allow a larger deployment. The resulting field data will help better understand litter origin, allowing to model and predict litter density in non surveyed areas. Correlations between macro litter density and environmental parameters will be studied (e.g., population density, catchment size, land use and hydromorphology). Finally, our work naturally benefits any extension of macrolitter monitoring in other areas (urban, coastal, etc) that may rely on a similar setup of moving cameras.
+++
(image-dataset-appendix)=
In this work, we do not seek to precisely predict the proportions of the different types of counted litter.
However, we build our dataset to allow classification tasks.
Though litter classifications built by experts already exist, most are based on semantic rather than visual features and do not particularly consider the problem of class imbalance, which makes statistical learning more delicate.
In conjunction with water pollution experts, we therefore define a custom macrolitter taxonomy which balances annotation ease and pragmatic decisions for computer vision applications.
This classification, depicted in {numref}trash-categories-image can be understood as follows.
-
We define a set of frequently observed classes that annotateors can choose from, divided into:
- Classes for rigid and easily recognisable items which are often observed and have definite shapes
- Classes for fragmented objects which are often found along river banks but whose aspects are more varied
-
We define two supplementary categories used whenever the annotater cannot classify the item they are observing in an image using classes given in 1.
- A first category is used whenever the item is clearly identifiable but its class is not proposed. This will ensure that our classification can be improved in the future, as images with items in this category will be checked regularly to decide whether a new class needs to be created.
- Another category is used whenever the annotater does not understand the item they are seeing. Images containing items denoted as such will not be used for applications involving classification.
---
height: 400px
name: trash-categories-image
---
Trash categories defined to facilitate porting to a counting system that allows trash identification
(video-dataset-appendix)=
In this section, we provide further details on the evaluation material.
{numref}river-sections shows the setup and positioning of the three river segments
- Segment 1: Medium current, high and dense vegetation not obstructing vision of the right riverbank from watercrafts, extra objects installed before the field experiment.
- Segment 2: High current, low and dense vegetation obstructing vision of the right riverbank from watercrafts.
- Segment 3: Medium current, high and little vegetation not obstructing vision of the left riverbank from watercrafts.
---
height: 400px
name: river-sections
---
Aerial view of the three river segments of the evaluation material
To annotate tracks on the evaluation sequences, we used the online tool "CVAT" which allows to locate bounding boxes on video frames and propagate them in time. The following items provide further details on the exact annotation process.
- Object tracks start whenever a litter item becomes fully visible and identifiable by the naked eye.
- Positions and sizes of objects are given at nearly every second of the video with automatic interpolation for frames in-between: this yields clean tracks with precise positions at 24fps.
- We do not provide inferred locations when an object is fully occluded, but tracks restart with the same identity whenever the object becomes visible again.
- Tracks stop whenever an object becomes indistinguishable and will not reappear again.
(tracking-module-appendix)=
(covariance-matrices)=
In our state space model,
- To estimate
$R$ , we computed a mean$L_2$ error between the known positions of objects and the associated predictions by the object detector, for images in our training dataset. - To estimate
$Q$ , we built a small synthetic dataset of consecutive frames taken from videos, where positions of objects in two consecutive frames are known. We computed a mean$L_2$ error between the known positions in the second frame and the positions estimated by shifting the positions in the first frame with the estimated optical flow values.
This led to
An important remark is that though we use these values in practice, we found that tracking results are largely unaffected by small variations of
(tau-kappa-appendix)=
An understanding of
In the following code cell, we plot the error decomposition of the counts for several values of
:tags: [hide-input]
set_split('val')
params = {'legend.fontsize': 'x-large',
'axes.labelsize': 'x-large',
'axes.titlesize':'x-large',
'xtick.labelsize':'x-large',
'ytick.labelsize':'x-large'}
plt.rcParams.update(params)
def hyperparameters(method_name, pretty_method_name):
tau_values = [i for i in range(1,10)]
kappa_values = [1,3,5,7]
fig, (ax0, ax1, ax2) = plt.subplots(3,1)
pretty_names=[f'$\kappa={kappa}$' for kappa in kappa_values]
n_count = {}
for kappa, pretty_name in zip(kappa_values, pretty_names):
tracker_names = [f'{method_name}_kappa_{kappa}_tau_{tau}' for tau in tau_values]
all_results = {tracker_name: pd.read_csv(os.path.join(eval_dir_short,'surfrider-test',tracker_name,'pedestrian_detailed.csv')).iloc[:-1] for tracker_name in tracker_names}
n_missing = []
n_false = []
n_redundant = []
for tracker_name, tracker_results in all_results.items():
missing = (tracker_results['GT_IDs'].sum() - tracker_results['Correct_IDs___50'].sum())
false = tracker_results['False_IDs___50'].sum()
redundant = tracker_results['Redundant_IDs___50'].sum()
n_missing.append(missing)
n_false.append(false)
n_redundant.append(redundant)
n_count[tracker_name] = missing + false + redundant
ax0.scatter(tau_values, n_missing)
ax0.plot(tau_values, n_missing, label=pretty_name, linestyle='dashed')
# ax0.set_xlabel('$\\tau$')
ax0.set_ylabel('$N_{mis}$')
ax1.scatter(tau_values, n_false)
ax1.plot(tau_values, n_false, linestyle='dashed')
# ax1.set_xlabel('$\\tau$')
ax1.set_ylabel('$N_{false}$')
ax2.scatter(tau_values, n_redundant)
ax2.plot(tau_values, n_redundant, linestyle='dashed')
ax2.set_xlabel('$\\tau$')
ax2.set_ylabel('$N_{red}$')
best_value = np.inf
best_key = ''
for k, v in n_count.items():
if v < best_value: best_key = k
best_value = v
best_key = best_key.split('kappa')[1]
best_kappa = int(best_key.split('_')[1])
best_tau = int(best_key.split('_')[-1])
print(f'Best parameters for {pretty_method_name}: (kappa, tau) = ({best_kappa}, {best_tau})')
handles, labels = ax0.get_legend_handles_labels()
fig.legend(handles, labels, loc='upper right')
plt.autoscale(True)
plt.tight_layout()
plt.show()
plt.close()
for method_name, pretty_method_name in zip(method_names, pretty_method_names):
hyperparameters(method_name.split('kappa')[0][:-1], pretty_method_name)
(bayesian-filtering)=
Considering a state space model with
- The predict step:
$p(x_{k+1}|z_{1:k}) = \int p(x_{k+1}|x_k)p(x_k|z_{1:k})\mathrm{d}x_k.$ - The update step:
$p(x_{k+1}|z_{1:k+1}) \propto p(z_{k+1} | x_{k+1})p(x_{k+1}|z_{1:k}).$
The recursions are intractable in most cases, but when the model is linear and Gaussian, i.e. such that:
with
-
$\mu_{k|k-1} = A_k\mu_{k-1} + a_k$ and$\Sigma_{k|k-1} = A_k \Sigma_{k-1} A_k^T + Q_k$ (Kalman predict step), -
$\mu_{k} = \mu_{k|k-1} + K_k\left[Z_k - (B_k\mu_{k|k-1} + b_k)\right]$ and$\Sigma_{k} = (I - K_kB_k)\Sigma_{k|k-1}$ (Kalman update step),
where
In the case of the linearized model in , EKF consists in applying these updates with:
(confidence-regions-appendix)=
In words,
When using EKF, this distribution is a multivariate Gaussian whose moments can be analytically obtained from the filtering mean and variance and the parameters of the linear model, i.e.
and
following the previously introduced notation. Note that given the values of
In
This allows easy computation of
(impact-algorithm-appendix)=
An advantage of the data association method proposed in is that it is very generic and does not constrain the tracking solution to any particular choice of filtering algorithm.
As for EKF, UKF implementations are already available to compute the distribution of
Denote
$$
\SMCfiltdist_k(\rmd x_k) = \sum_{i=1}^N w_k^i \delta_{X_k^i}(\rmd x_k) \eqsp.
$$
Contrary to EKF and UKF, the distribution $\mathbb{L}k$ of $Z_k$ given $Z{1:k-1}$ is not directly available but can be obtained via an additional Monte Carlo sampling step.
Marginalizing over
$$ \mathbb{L}k(\rmd z_k) = \int \int \likel_k(x_k, \rmd z_k)\transdist_k(x{k-1}, \rmd x_k)\filtdist_{k-1}(\rmd x_{k-1}) \eqsp. $$
Replacing
In our model, the state transition is Gaussian and therefore easy to sample from.
Thus an approximated predictive distribution
Since the observation likelihood is also Gaussian,
In theory, sampling-based methods like UKF and SMC are better suited for nonlinear state space models like the one we propose in .
However, we observe very few differences in count results when upgrading from EKF to UKF to SMC.
In practise, there is no difference at all between our EKF and UKF implementations, which show strictly identical values for
+++
:style: unsrt