Please find our poster at: https://drive.google.com/file/d/1PMVkD6Rn18KJ0TKpRHflsXAusDpaIvGR/view?usp=sharing

Verifying and Validating the Matrix Voice

We recently took many samples of controlled sinusoid waves at fixed distances and multiple known angles from the Matrix Voice 8-Mic array. These samples will be passed through the beamformer to validate that the microphone array is properly recording consistent and reliable sounds.

Written By Aaron Spaulding

This week I updated the classification model from the baseline result. I tested multiple configurations changing the transfer learning approach, augmentations applied, and classification model architecture. Since I have many results I include just the worst and best performers.

Worst Model

This model was by far the worst performer with almost no predictive ability. The plot of accuracy vs. epochs shows how the model was unable to converge for either the test or train sets, never passing 20% accuracy.

The confusion table reveals more into this and shows how the model only predicts one type of bird, the cardinal.

Best Model

The best model I developed was developed using the “resnet_v2_152” as a base model with three dense layers for the classification model. The first two had sizes 64 and 32 with the ‘relu’ activation function and the final layer had size 19 (19 classes) with the ‘sigmoid’ function. For this, I used a learning rate of 0.001. My early stopping stopped training at 33 epochs after the model converged. To double-check I trained the model for 10 additional epochs however the model did not improve from before. The image below shows the model loss vs. epochs and may indicate some overfitting occurring. I also applied all the discussed augmentations and balances the train and test sets.

The confusion table also is much improved with many more predictions along the diagonal.

This model was able to achieve a final micro-F1 of 63.54. This is a fantastic result since this score on a recent bird classification challenge on Kaggle would be a top 2% submission.

Written By Aaron Spaulding

This week I received the initial results of our classification method. Most of the time this week was spent writing the code for the method and setting everything up to run properly. However, I was able to set a baseline result which we can improve on in the upcoming weeks!

Baseline Result

It is important to note that this is a baseline result. This means we use a simple model without applying augmentations or optimizations. This also means our next models should be improvements from this!

This baseline model was built with the “resnet_v2_152” as the base model for transfer learning. I used three dense layers as my classification model and saved 25% of the dataset for validation.

The figures below show the loss and accuracy as the model is trained. The model quickly overfits the training set and stops learning on the test set. This leaves us with accuracy just below 60%.

To also test performance I created a confusion table for all our species. It is clear that the model is not a strong performer. In addition, we can see that the classes are not balanced which may be contributing to the low accuracy score.

Written By Aaron Spaulding

During the past week, I continued working on the classification model that will be used to determine the species of a bird sound after detection. Most of the week was spent processing the clips we labeled over break and applying augmentations. The image below shows the process I created where I import the audio data for each clip, mix it with Gaussian noise, and then generate a spectrogram with 251×251 shape. These processed clips will be used to train and validate our initial classification models.

One example of this process is shown below for the northern cardinal (Cardinalis cardinalis). The first image shows a picture of the bird taken this winter and the second shows the fully processed spectrogram ready for training.

This week I also met with Paul to record the 24-hour dataset. We set the arrays and temperature sensor up to record in my yard. We were able to capture the full 24-hour dataset with birds and temperature!

Our 24 hour recording was a success! The next thing we need to do for this is manually label a sampling of the recording for validation of our beamformer and classifier outputs. For approximately one out of the 24 hours, I will listen to the recording and label the species of bird that is making each call (since I trimmed the audio clips for the training data, I am relatively familiar with the different bird calls and should be able to figure this out when cross referencing with labeled and verified bird calls on Xeno-Canto). Once we figure out the format that will be easiest for recording this information (taking into account time of bird call and species of bird), I will get started on this. With the data that we have, I will only be able to validate the classifier and not the beamformer. 

Our presentation went well! This week, I have been trying to fix the issue with the 2D beamformer code. We are trying to plot the beam angle versus the power of the beam output. I have tried converting the powers to decibels, normalizing, and checking to make sure we have consistent units, however the shape of the beam response is not changing. I am asking my teammates to take a look at this as I am not sure what to try next. I will also keep looking for ways to fix this. 

This is an example of the 10 kHz beam response output. Based on the large shape, by inspection this looks like what we would expect a 1 kHz beam response output to be.

This week is our design presentation! My teammates and I are on this, specifically making slides and practicing for Friday! 

This week, I got the beamformer array giving an output in 3D. While this is a good first step, the beams are not pointed in the directions that we expect them to, indicating that there is some issue. This week, I will focus on fixing this so that we can properly theoretically validate the beamformer. Unfortunately, we also found an issue this week with the 2D beamformer. Specifically, although the direction seems accurate, we expect that, as the frequency of the simulated sound source increases from 1 kHz to 10 kHz, there will be an increased number of beams that are smaller in width (one main beam with side and null lobes). However, with our simulations, we are not seeing this change in shape as the frequency increases. I would like to get this fixed before fixing the 3D issue. Hopefully it will be a quick fix!

One of the upgrades that our sponsor has asked that we add to our beamformer is to have a 3D beam response instead of the 2D responses we have been calculating thus far. This means that we should be taking a vertical angle into account as well as the azimuthal angle. To do this, I will be reconfiguring parts of our code to take a z-component into account, such as the microphone arrays and the time offsets. In addition to updating the beamformer, I also need to update the code that will simulate sine waves coming from a variety of angles for theoretical validation of the beamformer.