Avatar

An application of Machine Learning on brainwave donations in order to remotely pilot a drone in real-time through brainwave reading.

Explore the docs »

Get help in the Wiki »

View Demo · Report Bug · Request Feature

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. Roadmap
5. Contributing
6. License
7. Contact
8. Acknowledgments

About The Project

This is an open source project from the Cloud Computing Club of Saint Cloud State University. Brainwaves are read using an OpenBCI headset. These brainwaves are sent to a server and randomly renamed and dated for privacy. After shuffling, the data is loaded into Spark where it is processed and a piloting prediction is made. A client-side request reads the most recent prediction and instructs the drone for its next action.

Institutional Review Board

Brainwave collection is considered human subject donation, and this project is compliant with the policies of the IRB .

(back to top)

Built With

(back to top)

Getting Started

There several components to the software. The server directory handles operations needed for storing data and generating ML models. The brainwave-prediction-app directory is operations to complete a live reading, receive a prediction, and then pilot the drone.

Prerequisites

Equipment

OpenBCI Headset with Cyton 16 Channel board

DJI Tello Edu Drone

Installation for Cloning ( Not Recommended )

Clone the repo (use sparse checkout if only wanting to run the client app for live demos)

git clone https://github.com/3C-SCSU/Avatar.git

Installation for Forking ( Recommended )

Forking

1. Navigate to the Fork Button:
At the top right corner of the repository page, you'll see a "Fork" button.

2. Click the Fork Button:
Click on the "Fork" button. GitHub will then create a copy of the repository in your account. All you have to do is name it something along the lines of Avatar-YourName-Forked

3. Wait:
Give it a moment. Depending on the size of the repository, it may take a few seconds to a minute. Once the process completes, you'll be redirected to the forked repository in your GitHub account.

4. Clone the Forked Repository:
Get the link to your newly forked repository and use the following cmd.

git clone <your-forked-url>

5. Configure upstream:
Upstream configuration allows you to pull changes from the main repository into your forked one. Navigate to your forked repository on your local machine, and then use the following command to list the currently configured remotes:

git remote -v

If you haven't added the upstream repository, you can add it with:

git remote add upstream https://github.com/3C-SCSU/Avatar

Fetch the changes from the upstream repository:

git fetch upstream

This will download all the changes from the upstream repository but won't merge them yet.

Switch to your fork's main branch (or whichever branch you want to merge the changes into):

git checkout main

(whatever branch you want typically main or your own branch)

Merge the changes from the upstream repository into your local branch:

git merge upstream/main

Again, replace main with the appropriate branch name if different.

Resolve any merge conflicts:

If there are any conflicts between your fork and the original repository, you'll need to resolve them manually. After resolving the conflicts, you would stage the changes with git add and then commit the changes with git commit.

Push the merged changes to your fork on GitHub:

git push origin main

Replace main with your branch name if different.

How to do a Pull Request

If you've made changes to your fork and want to contribute them back to the original repository, you would do so by creating a pull request. Here's how to create a pull request (often abbreviated as PR) from your fork to the original repository:

Make sure your fork is up to date: (follow instructions above)

Push your changes to your fork:

If you've made local changes, push them to your forked repository on GitHub:

git push origin <your-branch-name>

Replace with the name of your branch. Probably is main or your name.

Open a pull request via GitHub:

Navigate to the main page of your forked repository on GitHub. Just above the file list, click on the “New pull request” button. You'll be brought to the original repository's pull request page with a comparison between the original repo and your fork. By default, GitHub will try to compare the original repository's default branch (like main or master) to your fork's default branch. If that's not the branch you want to base your pull request on, use the branch dropdown menu to select the branch from your fork you'd like to propose changes from. Review the changes to ensure they are what you expect. Once you've reviewed the comparison and are ready, click on the “Create pull request” button. Fill out the pull request form by giving it a meaningful title and a description of the changes you made. This will help the maintainers of the original repo understand the purpose of your PR. Finally, click the “Create pull request” button.

After these steps, your forked repository should be updated with the latest changes from the original repository.

(back to top)

Usage

How to run a live demo

Steps:

1. On the VPS, install and launch the server.py file. It can be run from the command line with the command:

   python server.py

2. Using the OpenBCI software, test the connections on the headset for the person wearing it. Adjust as necessary. Try and make all connections “Not Railed”

3. Locally, launch the GUI application using either an IDE play button or from terminal

   python GUI.py

• Note the relative pathing to the images folder and launch with both a terminal and local files appropriately pathed
• If you receive a module error it can be installed using pip i.e.

  pip3 install <module name>

4. In the GUI select Brainwave Reading

5. Future implementations - select between Manual Control or AutoPilot

6. Connect the computer to the Drone’s wifi and press the Connect button in the GUI to connect to the drone.

7. Give the person wearing the headset a 5 second countdown and instructions on which drone action to think about.

8. Press Read my Mind to automatically initiate a 10 second reading. This triggers the brainflow1.py file and sends the data to the server.py endpoint. The results of the prediction will automatically be displayed in the output table of the GUI. (The endpoint IP may need to be manually configured until environment variables are set up to handle this operation)

9. If the prediction is correct, press the Execute button to have the drone perform the action. If the prediction is wrong, override the drone action by typing an alternative in the text input box and press Not what I was thinking…

10. At any time, press “Keep Drone Alive” to attempt to maintain the drone’s hovering while it waits for a prediction command. (Note: This isn’t working well and will need to be improved)

This section will be updated further as the project progresses.

Roadmap

• Read Brainwaves using OpenBCI software
  • Configure workstation Chromebooks for compatibility with BCI dongle
• Encrypted File Transfer from workstation to VPS
• Automated file shuffling
• Containerize VPS applications
• Implement K8s
• Connect Spark with GCP bucket
• Configure Spark with Delta Lake
• Create Application to Control Drone
  • GUI
  • Drone Control
  • Headset Data transferring
  • Connect with ML Predictions

See the open issues for a full list of proposed features (and known issues).

(back to top)

Contributing

University Contributions

If you have been informed that your submission for contribution was selected and approved by the Cloud Computing Club, follow the below steps to add it to the project and receive the bounty.

1. Create your Feature Branch (git checkout -b feature/AmazingFeature)
2. Name the feature branch using format feature-bounty ex. file_shuffler-bounty
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request and select a review from the list of Collaborators.
6. Update the Wiki with an explanation of your added feature.

External Contributions

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

Pull requests will be rieviewed and merged:

Community Members by Category

Members by promotion and merit voted in the weekly's #3C meeting:

1. Sporadic contributors: any developer who decides to start contributing to the project's code base.

2. Committers: contributors promoted above the so-called sporadic contributors. To become a committer, the person must get 20 patches approved. When becoming a committer, then the committer will review and support code submitted from sporadic contributors

3. Reviewers: are generally either the founding developers of the project - like initial administrator developers; or contributors who become committers, who then, after 50 patch contributions to the project's code base, are promoted to the status of reviewers. Reviewers will decide and review any conflict in code contributions to the project.

(back to top)

License

Distributed under the MIT License. See LICENSE for more information.

(back to top)

Contact

Cloud Computing Club [email protected]

Discord: https://discord.gg/mxbQ7HpPjq

Project Link: https://github.com/3C-SCSU/Avatar

(back to top)

Acknowledgments

• Dr. A. Cavalcanti - Faculty Advisor for #3C-SCSU

(back to top)