final-project-brainiac

BRAINIAC: ESE 5190 Team 03

* Team Members: Jake Donnini, Marcus Tsuei, Kasyap Chakravadhanula
* Github Repository URL: https://github.com/upenn-embedded/final-project-brainiac.git
* Github Pages Website URL: https://upenn-embedded.github.io/final-project-brainiac/
* Hardware: ATMega328PB, ESP32, Arduino Nano BLE 33, UpsideDown Labs Pill EXG Sensor, motorized chasis

1. Video

https://youtube.com/shorts/mY0LasGTrPM?feature=share

2. Images

[Insert final project images here]

3. Results

3.1 Software Requirements Specification (SRS) Results

• SRS01: The machine learning model shall recognize 4 different thoughts which correlate to the commands left, right, forward, and stop, with at least 70% model accuracy
  • Results: We tested the accuracy of the model on the validation set and got an accuracy of 83%
• SRS02: input/output delay shall not be more than 500ms to ensure a smooth thought-to-motion user experience.
  • Results: The input delay was less than a 1/10 of a second. We were able to verify by having new users successfully navigate around obstacles without any practice.
• SRS03: Drift in a straight line must be within 2 inches horizontally over a 0.5m vertical distance. * Results: Measured by constraining the vehicle to 2 inches horizontally and testing whether it could surpass 0.5m. We used a proportional control loop to ensure the motors moved at the target rpm rate

3.2 Hardware Requirements Specification (HRS) Results

Based on your quantified system performance, comment on how you achieved or fell short of your expected hardware requirements. You should be quantifying this, using measurement tools to collect data.

• HRS01: Motors shall encode their position and provide at least 0.25 m/s of translational velocity. Encoders shall provide at least 500 ticks per revolution
  • Results: We found that the car moved at about 0.5 m/s at 50% duty cycle travelling a distance slightly greater than 2m in 3.9s. Video here
• HRS02: Processor shall have at least an 8MHz clock speed to handle PWN, interrupts, and UART communication simultaneously. It must also have a least three independent PWM capable clocks, two pin change interrupts and UART capability.
  • Results: The Atmega328pb meets these requirements with a 16MHz clock speed and 4 PWM capable clocks, with over 2 pin change interrupts.
• HRS03: Microcontroller shall have at least 1MB of programmable memory and a clock speed of 64 MHz to run the model
  • Results: The BLE Nano 33 meets these requirements, as measured.
• HRS04: Wireless communication shall be able to communicate at a distance of 10m
  • Results: The range of the wireless communication exceeded 10m, tested by putting the transmitter and car in opposite corners of detkin and verifying it was still responsive – it was.

3.3 Machine Learning Implementation

4. Conclusion

Reflect on your project. Some questions to consider: What did you learn from it? What went well? What accomplishments are you proud of? What did you learn/gain from this experience? Did you have to change your approach? What could have been done differently? Did you encounter obstacles that you didn’t anticipate? What could be a next step for this project?

Our project overall went well. Some things we are proud of include the motor control that we wrote, including control in 3 directions with different PWM waves, reading encoder values at high rate, and using those encoder values to create a PID loop. We are also proud that we were able to build the full communication loop all the way from the instrumentation amplifier for the electrodes to the ATMega to control the motors, using different types of communication including UART and ESP-NOW. The overall loop from electrodes to motors was very responsive and worked at high range.

We learned several things, had difficulties, and also had to change our approach slightly over time.

First, we quickly realized that using 3 electrodes with a relatively low quality amplifier is not sufficient to read EEG signals. We thought that since we only needed a few different signals, this would be easy, but we realized that the skull acts as a filter so the EEG signals are extremely weak compared to noise from even small muscle movements. So, we shifted our approach to sensing these muscle movements instead, repurposing our electrodes as EMG sensors rather than EMG, which we still thought was pretty cool.

We also had difficulties with the motor control and PID loop. The way the board on the car is arranged is a bit confusing and we spent some time being stuck until we realized we were just connecting things to the wrong pins. Also, we severely underestimated the impact of print statements on the responsiveness of the loop. We spent quite a lot of time debugging why the response of the motors lagged so far behind the RPMs being read, and we realized that a big part of the issue was the 50ms delay we had after each print. So this was definitely a learning.

We also had a major difficulty in getting the ML model onto the BLE Nano, it was just going blank every time we tried to upload the model, and we realized it did not have enough memory so we had to pivot to a different chip but this was right before the deadline so we did not have time. Although this isnt in the scope of this class, in the future we will do more research and find hardware that goes beyond our anticipated need so we don’t have issues like this. The chip we selected was the default option for the Tensorflow lite library. We then pivoted to using the ESP32 Xiao board where the model was able the run. We implemneted a RTOS using freeRTOS to use the MCU’s two cores. One core to read the ADC for the EEG signals and the other to run two tasks. One task is to compute the spectrogram over 2 seconds of data and the other is to run inferences on the model. However, at the time of this report the model isn’t able to classify signals on the edge.

In the future, this project can be expanded in a few ways. A friend mentioned that the Bioengineering department has a full EEG cap with many more electrodes. Using this and training a model will likely enable us to actually create the brain controlled car. Also, we can add more functions to the car, for example reverse. Also, we can add intensities to the actions. For example, if we have identified the user wants to go forward, they can either do some other action to indicate going faster/slower, or the same action for more/less intensity.