Skip to content
forked from andreped/DSS

📳 From training ViTs to real-time deployment and integration in cross-platform mobile apps!

License

Notifications You must be signed in to change notification settings

SINTEFMedtek/DSS

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

drawing

DSS: Deep Sensor Systems

From training and deployment of ViTs to development of real-time cross-platform mobile apps

License GitHub Downloads DOI codecov

DSS was developed by SINTEF Medical Image Analysis with aim to integrate AIs into sensor systems.

This project serves as a demonstration on how to do it, and does not claim to be a generic framework.

Below there are described some of the key features of this project, but to see what else is possible, please, see the wiki.

Build Type Status
Test Training CI
Test Flutter CI
Build APK CI

When using this framework, it is a good idea to setup a virtual environment:

virtualenv -ppython3 venv --clear
source venv/bin/activate
pip install -r requirements.txt

The following dependencies will be installed:

  • pandas<=1.5.3
  • tensorflow<=2.12.0
  • tensorflow-addons<=0.19.0
  • tensorflow-datasets<=4.8.3

Tested with Python 3.7.9 on Win10, macOS, and Ubuntu Linux operating systems. Also tested with Python 3.10.4 on GitHub Codespaces.

Note that to activate the virtual environment on Windows instead run ./venv/Scripts/activate.

To train a model, simply run:

python main.py

The script supports multiple arguments. To see supported arguments, run python main.py -h.

To visualize training history, use TensorBoard (with example):

tensorboard --logdir ./output/logs/gesture_classifier_arch_rnn/

Example of training history for a Recurrent Neural Network (RNN) can be seen underneath:

The figure shows macro-averaged F1-score for each step during training, with black curve for training and blue curve for validation sets. Best model reached a macro-averaged F1 score of 99.66 % on the validation set, across all 20 classes.

Disclaimer: This model was only trained for testing purposes. The input features were stratified on sample-level and not patient-level, and thus validation performance will likely not represent true performance on new data. However, having a trained model enables us to test it in a Mobile app.

The current data used to train the AI model is the SmartWatch Gestures dataset, which is available in tensorflow-datasets. The dataset has the following structure:

FeaturesDict({
    'attempt': tf.uint8,
    'features': Sequence({
        'accel_x': tf.float64,
        'accel_y': tf.float64,
        'accel_z': tf.float64,
        'time_event': tf.uint64,
        'time_millis': tf.uint64,
        'time_nanos': tf.uint64,
    }),
    'gesture': ClassLabel(shape=(), dtype=tf.int64, num_classes=20),
    'participant': tf.uint8,
})

In order to be able to use the trained model in a mobile app, it is necessary to convert the model to a compatible format. TensorFlow Lite is an inference engine tailored for mobile devices. To convert the model to TF-Lite, simply run this command:

python dss/keras2tflite.py -m /path/to/pretrained/saved_model/ -o /path/to/save/converted/model.tflite

A simple Mobile app was developed in Flutter, which demonstrates the AI in action using the accelerometer data from the mobile phone in real time. The data can also be stored and deleted locally.

To use the app, you need an Android phone and have developer mode enabled (see here for how to enable it). Then simply download the APK from here, double-click to install, and use the app as you normally would.

Info on how the mobile app was developed (and how to make your own app), can be found in the wiki.

The training framework was mainly developed using Keras with TensorFlow backend.

The mobile app was developed using Flutter, which is a framework developed by Google. For the app, the following open packages were used (either MIT, BSD-2, or BSD-3 licensed):

If you found this project useful, please, consider citing it in your research article:

@software{andre_pedersen_2023_7701510,
  author       = {André Pedersen and Ute Spiske and Javier Pérez de Frutos},
  title        = {andreped/DSS: v0.2.0},
  month        = mar,
  year         = 2023,
  publisher    = {Zenodo},
  version      = {v0.2.0},
  doi          = {10.5281/zenodo.7701510},
  url          = {https://doi.org/10.5281/zenodo.7701510}
}

About

📳 From training ViTs to real-time deployment and integration in cross-platform mobile apps!

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Dart 76.5%
  • Python 20.5%
  • Batchfile 1.9%
  • Other 1.1%