FPGA Depth Estimation using a Camera Array

During the month of January 2022, I worked on developing a modular camera setup powered by an FPGA that can support variable offsets on the x and z positions of the camera. Using two offset cameras, the code is able to use color segmentation to determine an object of interest and estimate the distance away from it using the center offset. It powers a VGA display that showcases the two images, debug information and crosshairs that show the center of the object of interest. It was coded using Verik, precompiled into SystemVerilog and then synthesized using vivado before being uploaded onto a Xilinx A7. Here is a link to the GitHub code.

Eclipse - glasses that modulate epileptic triggers

In the Fall of 2021, with a team of product designers of different backgrounds, we went through the process of generating ideas, mockups, testing, user interviewing and finally fabrication and coming up with a plan to scale up. After mocking up different projects, we settled on developing a pair of glasses intended for people with photosensitivity and photosensitive epilepsy. The glasses have electrochromic lenses that darken automatically when a voltage is applied. Here’s a photo of me presenting at our product launch :)

We designed our custom PCB that houses an ATSAMD21G18A processor and programmed it to scan using RGB photodiodes the incoming light and predicting when an epileptic trigger occur. It then darkens and undarkens the lenses at a frequency that cancels out the trigger, allowing the user to still see and protecting them from the flashing light. I personally worked on the sensing and modulation code, prototyping on a Feather M0 dev board, designing user interaction, designing and performing EEG trials, coding in Atmel’s Microchip Studio and choosing the PCB & circuit components.

I was also responsible for documenting our entire process through multiple mediums including our instagram.

Teensy U2F Authenticator

As part of a computer systems security team, we designed and open sourced a homemade 2-factor authentication security key based on the FIDO alliance’s U2F specification. We designed it to require minimal hardware (a teensy 3.2 and a recommended button+resistor combo is all you need!). Since it works on generic teensy’s and all the code is open source, you can verify the security of the key.

I primarily worked on the communication scheme over RawHID and hardware interfacing and the authentication protocol on the key side. Here is a cool write-up on the security key and here is the repository you can use to make your own security key!

Glasses at a Picnic - Digital Music Instrument

Over Spring 2020, I took a digital instruments class where we got to design different musical instruments out of electronic components and programming them in Arduino C++, Python, PureData & Automatonism. For my final project, I built a Computer Vision based algorithm that performed real-time analysis on a video feed to determine the placement of different wine glasses that tagged with fiducial markers. The music generation part of the code has information on the position, roll, pitch and yaw of the glasses at all times. The code also had a live audio stream being frequency analyzed to detect the sounds of wine glasses clinking or resonating on their own. This is what the setup looked like with a mic on the bottom side of the table:

This all fed into a sound generating patch with different submodules that produced different unique sounds for every glass. The position of the glass would change the position of the audio in 3D space and change the “mood” of the glass. Resonances and clinking sounds would add effects or help transition “scenes.” Here is a link to a write-up for the instrument.

Non-Photorealistic Renderer - Convert images to paintings

During Spring 2020, I worked on a C++ project that processes images and converts them into detailed multi-layered paintings with the ability to interpolate and incorporate design styles from a reference image. Here are some example renderings:

The code would study the structure of an image using tools built from scratch, it would extract information such as the structure tensor and direction field (see all the details and more examples in this write-up). This helps capture details in an image and make the brush strokes look realistic and follow the seams and curves of an image.

The code also uses k-clustering to bin colors and produces gamma maps to change the feeling behind a photo, making them more dramatic and scenic. The code can also take two inputs and draw the first photo while using the style of another image. For example, feeding it a night photo of a desert and a day photo of some other setting, the color mapping can interpolate colors and produce a day photo of the desert. Here is a scene change example:

Computer Vision and LIDAR Based Obstacle Avoidance

In the spring of 2020, for Robotics: Systems and Science, we worked as a team to build a self- driving car to perform in two types of races. (1) A regular race with who reaches the finish line first where we know the race track configuration before hand and (2) an obstacle avoidance race where we don’t have a pre-determined map, and obstacles are placed randomly on the track. While the goal of (1) is to reach the finish line first, in (2), it is a timing and number of obstacles hit race. Here is the hardware that was on our car and what we simulated when we moved online.

This is a gif (🔉/dʒɪf/</a>) of the first iteration of our LIDAR-based code trying to path find!

(1) For the regular race, I worked with my team to program SLAM with LIDAR data for localization and then implemented path finding to find the optimal path. We then had a micro-strategy that controlled steering using PID controllers to control steering and the amount of acceleration the car is applying at all times.

(2) I was the one who primarily worked on the obstacle avoidance race. I chose to design a Computer Vision based algorithm over just using LIDAR (we were the only team to use CV!). The car used image segmentation and classification to build the navigation space and map roads vs. obstacles. It then would path find around them locally using the segmented image stream and use LIDAR data to help it when objects are too close or when distance information would be useful. Section 4 of the paper below details my work on the obstacle avoidance race!

Here is our final paper for the class

Quantum Assembly Dev Tools and Qiskit

I enjoy writing QASM (Quantum Assembly) code way more than I should. As a reason I try to contribute to Qiskit Terra and, over time, built multiple dev tools that are sought out for use in industry when working with QASM. The two main types of tools are syntax highlighters and circuit simulation tools.

I programmed extensions for major code editors and syntax highlighter engines that can highlight QASM code as per OpenQASM and cQASM standards.

I also built open source tools for quantum circuit designers in atom, where a live updated pane shows you circuit properties (such as circuit depth, width tensor factors) and simulation statistics (bell state counts and state vectors).