Zounds

Resonance Inference

Thu, 21 Nov 2024 12:00:00 +0000

In this mini-experiment, I further explore whether playable “instruments” can be extracted from a single audio recording by overfitting a simple model of resonating objects and a sparse control signal.

Iterative Decomposition Model V3

Thu, 21 Nov 2024 12:00:00 +0000

There’s a new demo page for v3 of the iterative audio decomposition model I’ve been working on.
The model is small and hasn’t had much (less than 24-hours) of training time so far, but doubles the duration of audio segments from the prior model.

Playable State-Space Model From a Single Audio Sample

Mon, 23 Sep 2024 12:00:00 +0000

This work attempts to reproduce a short segment of “natural” (i.e., produced by acoustic instruments or physical objects in the world) audio by decomposing it into two distinct pieces:

A state-space model simulating the resonances of the system
a sparse control signal, representing energy injected into the system.

The control signal can be thought of as roughly corresponding to a musical score, and the state-space model can be thought of as the dynamics/resonances of the musical instrument and the room in which it was played.

Gaussian/Gamma Splatting for Music

Thu, 01 Aug 2024 12:00:00 +0000

In this work, we apply a Gaussian Splatting-like approach to audio to produce a lossy, sparse, interpretable, and manipulatable representation of audio. We use a source-excitation model for each audio “atom”, implemented by convolving a burst of band-limited noise with a variable-length “resonance”, which is built using a number of exponentially decaying harmonics, meant to mimic the resonance of physical objects. Envelopes are built in both the time and frequency domain using gamma and/or gaussian distributions. Sixty-four atoms are randomly initialized and then fitted (3000 iterations) to a short segment of audio via a loss using multiple STFT resolutions. A sparse solution, with few active atoms, is encouraged by a second, weighted loss term.

Sparse Interpretable Audio Model V2

Thu, 29 Feb 2024 12:00:00 +0000

Happy Leap Day! I’ve just published some high-level details and sound reconstructions from a newly-trained model that decomposes musical audio into an easy-to-manipulate format:

event times and amplitudes
time and amplitude-agnostic event vectors that describe a musical event
A global context vector that determines room reverb

Different “axes” of the sound can be manipulated independently, and indivdual events can be played in isolation.

Sparse Interpretable Audio Model

Wed, 15 Nov 2023 12:00:00 +0000

This post covers a model I’ve recently developed that encodes audio as a high-dimensional and sparse tensor, inspired by algorithms such as matching pursuit and dictionary learning.

Its decoder borrows techniques such as waveguide synthesis and convolution-based reverb to pre-load inductive biases about the physics of sound into the model, hopefully allowing it to spend capacity elsewhere.

The goal is to arrive at a sparse, interpretable, and hopefully easy-to-manipulate representation of musical sound. All of the model training and inference code can be found in the matching-pursuit github repository, which has become my home-base for audio machine learning research.

The current set of experiments is performed using the MusicNet dataset.

⚠️ This post contains sounds that may be loud. Please be sure to start your headphones or speakers at a lower volume to avoid unpleasant surprises!

Motivations

I’ve formed several hunches/intuitions in my recent years working on machine learning models for audio synthesis that all find expression in the work being covering today.

Frame-Based Representations Are Limited

Almost all codecs and machine learning models that perform audio analysis and/or synthesis are frame-based; they chop audio up into generally fixed-size frames, without much regard for the content, encode each framem and represent audio as a sequence of these encodings.

The model discussed today takes some baby-steps away from a frame-based representation, and toward something that might be derived using matching pursuit or dictionary learning.

While a sequence of encoded frames may reproduce sound very faithfully, it is not easily interpretable; each frame is a mixture of physical or musical events that have happened before that moment, and continue to reverberate.

My sense is that a high-dimensional and very sparse representation maps more closely onto the way we think about sound, especially musical audio.

Audio Synthesis Models Must Learn Physics

While there are certainly classes of audio where the laws of physics are unimportant (synthesized sounds/music), those laws are integral to the sounds made by acoustic instruments, and even many synthesized instruments simply aim to mimic these sounds.

Do audio models waste capacity learning some of the invariant laws of sound from scratch? Could it be advantageous to bake some inductive biases around the physics of sound into the model from the outset, so that it can instead focus on understanding and extracting higher-level “events” from a given piece of audio?

Synthesis techniques such as waveguide synthesis and the use of long convolution kernels to model material transfer functions and room impulse responses suggest a different direction that might be fundamentally better (for audio) than the classic upsample-then-convolve-with-short-kernel approaches that have dominated both image and audio generation to-date.

Quick Tour

Before diving into the details, it might be good to get an intuitive feel for what the high-level components of the model are doing. We grab a random segment of audio from the MusicNet dataset…

from conjure import audio_conjure

@audio_conjure(conjure_storage)
def get_audio(identifier):
    from data.audioiter import AudioIterator
    from util import playable
    import zounds

    stream = AudioIterator(
        1,
        2**15,
        zounds.SR22050(),
        normalize=True,
        overfit=False,
        step_size=1,
        pattern='*.wav')
    chunk = next(iter(stream))
    chunk = chunk.to('cpu').view(1, 1, stream.n_samples)
    audio = playable(chunk, stream.samplerate)
    bio = audio.encode()
    return bio.read()

example_6 = get_audio('example_7c')
_ = get_audio.meta('example_7c')

You can click on the waveform to play the sound.

The Encoding

The network encodes the audio, and we end up with something like this.

⚠️ Because the representation is very high-dimensional, (4096, 128) to be exact, we downsample for easier viewing

from conjure import numpy_conjure, SupportedContentType

@numpy_conjure(
    conjure_storage,
    content_type=SupportedContentType.Spectrogram.value)
def get_encoding(identifier):
    from models.resonance import model
    import zounds
    from io import BytesIO
    import torch
    from torch.nn import functional as F

    n_samples = 2**15
    bio = BytesIO(get_audio(identifier))
    chunk = torch.from_numpy(zounds.AudioSamples.from_file(bio)).float()
    chunk = chunk[...,:n_samples].view(1, 1, n_samples)

    encoded = model.sparse_encode(chunk)

    # downsample for easier viewing
    encoded = encoded[:, None, :, :]
    encoded = F.max_pool2d(encoded, (8, 8), (8, 8))
    encoded = encoded.view(1, *encoded.shape[2:])
    encoded = encoded[0].T
    return encoded.data.cpu().numpy()

encoding_1 = get_encoding('example_7c')
_ = get_encoding.meta('example_7c')

The dark-blue portions represent zeros, and the yellow-orange portions represent events.

Reconstructed Audio

We can also perform a full pass through the network, both encoding and decoding. The reconstructions don’t sound amazing, but are clearly the same musical sequence as the original.

from conjure import audio_conjure

@audio_conjure(conjure_storage)
def reconstruct_audio(identifier):
    from models.resonance import model
    from util import playable
    from modules.normalization import max_norm
    import torch
    import zounds
    from io import BytesIO

    n_samples = 2**15
    bio = BytesIO(get_audio(identifier))
    chunk = torch.from_numpy(zounds.AudioSamples.from_file(bio)).float()
    chunk = chunk[...,:n_samples].view(1, 1, n_samples)
    recon, _, _ = model.forward(chunk)
    recon = torch.sum(recon, dim=1, keepdim=True)
    recon = max_norm(recon)
    recon = playable(recon, zounds.SR22050())
    encoded = recon.encode()
    return encoded.read()

result = reconstruct_audio('example_7c')
_ = reconstruct_audio.meta('example_7c')

Another Reconstruction Example

original:

result = get_audio('example_8c')
_ = get_audio.meta('example_8c')

reconstruction:

result = reconstruct_audio('example_8c')
_ = reconstruct_audio.meta('example_8c')

Model and Training Details

The model’s encoder portion is fairly standard and uninteresting, while the decoder includes some novel features. Once again, code for the experiment can be found on github.

Encoder

Starting with 32768 samples at 22050 hz, or about 1.5 seconds of audio, we first perform a short-time fourier transform with a window size of 2048 samples and a step size of 256 samples. This means that we begin with a representation of shape (batch, 1024, 128), where 1024 is the number of FFT coefficients and 128 the number of frames.

The spectrogram is then processed by a U-Net architecture with skip connections.

Finally, the reprsentation is projected into a high-dimensional space and sparsified by choosing the N top-k elements in the (batch, 4096, 128) tensor.

⚠️ The astute reader will notice that we’re working with the very frame-based representation we earlier warned against. There are still steps that need to be taken to fully free ourselves from the fixed-size grid.

Decoder

Beginning with our highly-sparse (batch, 4096, 128) representation, we “break apart” this tensor into N one-hot vectors. The current experiment sets N = 64.

Put another way, we now have N or 64 one-hot vectors, ideally representing “events” for each element in the batch of examples. Next, we embed these high-dimensional one-hot vectors into lower-dimensional, dense vectors, and generate the audio for each event independently from there.

First, we generate an impulse, which amounts to band-limited noise, representing the attack, breath, or some other injection of energy into a system.

Next we create a linear combination of a number of pre-generated resonance patterns, which amount to four different waveforms (sine, square, sawtooth, triangle) sampled at many different frequencies within the range of fundamental frequencies for most acoustic instruments, in this case, from 40 hz to 4000 hz. This represents the transfer function or impulse response of whatever system, or instrument, the impulse’s energy is injected into.

We also choose a decay value, which determines the envelope of the resonance and apply the envelope to the resonance we’ve generated.

Each impulse and resonance are then convolved to produce a single event.

The (batch, 4096, 128) encoding is also summed along its time-axis, such that it becomes (batch, 4096), and is then embedded into a lower-dimensional, dense vector. This vector is passed to a sub-network which chooses a linear combination of impulse responses for convolution-based reverb. Each event is also convolved with this room response kernel.

We end with a tensor of shape (batch, n_events, 32768), with each channel along the n_events dimension representing

Training

Instead of using MSE loss, each “event channel” loss is computed independently. There’s much more work to do to both refine this approach and understand better why it helps, but it does seem to encourage events to cover independent/orthogonal parts of the time-frequency plane. The model used to produce the encodings and audio in this post was trained for about 24 hours.

Random Generation

Finally, we can understand the model a little better (and have some fun) by generating random sparse tensors and listening to the results.

from conjure import audio_conjure

encoding_1 = get_encoding('example_7c')

@audio_conjure(conjure_storage)
def random_generation(identifier):
    import torch
    from models.resonance import model
    from modules import sparsify2
    from util import playable
    import zounds
    from torch.nn import functional as F

    n_samples = 2**15
    encoding = torch.zeros(1, 4096, 128).normal_(encoding_1.mean(), encoding_1.std())
    encoded, packed, one_hot = sparsify2(encoding, n_to_keep=64)
    audio, _ = model.generate(encoded, one_hot, packed)
    audio = torch.sum(audio, dim=1, keepdim=True)
    audio = playable(audio, zounds.SR22050())[..., :n_samples]
    bio = audio.encode()
    return bio.read()

result = random_generation('random_1d')
_ = random_generation.meta('random_1d')

Another random generation result:

result = random_generation('random_3d')
_ = random_generation.meta('random_3d')

Next Steps

A single, low-dimensional and dense “context” vector for each audio segment might allow similar sparse vectors to represent related musical sequences played on different instruments, or in different rooms
The “physical model” I’ve developed is very rudimentary. More study into existing physical modelling techniques might produce a far superior model

Thanks for Reading!

If you’d like to cite this article

@misc{vinyard2023audio,
  author = {Vinyard, John},
  title = {Sparse Interpetable Audio},
  url = {https://JohnVinyard.github.io/machine-learning/2023/11/15/sparse-physical-model.html},
  year = {2023}
}

A Music Vocoder Using Conditional Generative Adversarial Networks

Wed, 01 Apr 2020 12:00:00 +0000

The last couple of posts have been all about audio analysis and search but in this one, I’ll return to some work that gets me a little closer to my ultimate goal, which is building synthesizers with high-level parameters, allowing the production of audio ranging from convincingly-real renderings of traditional acoustic instruments to novel synthetic textures and sounds.

There’s clearly a lot of value in symbolic (usually MIDI-based) approaches to music generation but I feel there’s incredible promise and flexibility in producing audio directly. Some incredibly moving sequences of sound are also very difficult to score, and not every valid or valuable piece of music can be expressed as a collection of discrete notes or events.

All of the code for this, and future experiments involving music synthesis, can be found in this GitHub repo.

A Break in the Case

I had already been exploring the use of generative adversarial networks to produce very short and noisy snippets of Bach piano music when I read the WaveGAN paper, which was the first published application of GANs to raw audio that I’m aware of. This was a very exciting and insightful paper, but I got really excited last year when I read a pair of papers that had something very cool in common.

Those were:

Both papers focused primarily on speech (the MelGAN paper included piano samples as an aside) but the interesting feature they shared is that they were both training strongly-conditioned GANs using low-frequency audio features as input to the generator, which would then output high-frequency raw audio samples.

What really excited me about these papers is that they seemed to lay the foundation for a very flexible two-stage music synthesis pipeline. At its simplest, two trained networks would be required:

A strongly-conditioned generator that can produce convincing audio from low-frequency features
Another network, unconditioned or perhaps conditioned on yet higher-level features, that could produce plausible sequences of these low-frequency features

This post will focus on the first stage in that pipeline, with a subsequent post covering the generation of novel musical sequences.

Since my focus is music, I chose to use the MusicNet dataset, which consists of around 34 hours of classical music recordings. As my low-frequency conditioning feature, I chose to use 128-bin Mel-frequency spectrograms. Here are a couple examples of audio from the dataset, alongside the accompanying low-frequency Mel spectrograms:

First Attempt: Assessing the MelGAN for the Task

The first and most obvious approach would be to use the architecture described in the MelGAN paper, mostly as-is, and assess its performance on the task of producing musical signals.

The MelGAN paper focuses primarily on speech but you can hear some examples of both speech and music generation from the model on this page.

I had reason to be skeptical about the architecture’s performance on fairly diverse musical signals based on the piano examples included in that page, but decided starting with the MelGAN architecture would serve as a good baseline.

The MelGAN paper uses a single generator that makes use of dilated convolutions to greatly expand the receptive field of its layers, as well as multiple discriminators that operate at a few different time scales/sampling rates. Notably, the discriminators don’t just produce a single judgement per input example; they produce multiple, overlapping judgements in the time domain.

The generator is fully convolutional, meaning that it makes no assumptions about the input or output time dimension. This is a handy feature in a vocoder, since we can pass it a spectrogram of virtually any length at inference time.
During training, windows of 8192 samples at 22050hz (about a third of a second) are used.

In the usual GAN setting, the generator is only trying to minimize or maximize some scalar value that the discriminator computes to indicate how close to the real training data the synthetic data lies. The MelGAN paper adds to this a feature-matching loss, in which the generator tries to minimize the L1 distance between discriminator feature maps, or internal activations, for synthetic and real data.

It’s interesting to note that the discriminator does not have access to the conditioning information in the original MelGAN formulation; it is purely computing the distance between real and fake examples. When the generator loss is computed, discriminator feature maps are compared within the same batch, so that feature maps from real audio samples are being compared against feature maps from audio generated based on spectrograms derived from that same audio.

As you can hear, the generator captured the overall structure of the audio fairly well after around 12 hours of training, but produced reproductions that were somewhat metallic and noisy:

Real

Generated

Real

Generated

More generated samples can be heard here.

An Alternate Architecture Geared Toward Music

After evaluating the original MelGAN architecture’s performance on MusicNet, and trying quite a few alternative approaches, I settled on the some modifications to the original experiment that resulted in more pleasing/musical audio.

Over the course of several experiments, I found that giving the discriminator access to the conditioning data generally accelerated training and made for better end results. I also found that building some strong priors into both the generator and discriminator architectures seemed to help with producing crisper and more pleasant audio.

Before we dive in, code for the top-level parameters of this experiment can be found here.

A Conditioned Discriminator

First, several experiments seemed to indicate that allowing the discriminator access to the conditioning spectrograms accelerated training and made for better end results. It’d be worth an ablation study with my best-performing generator and discriminator to find out if this impression is true, but I haven’t yet performed one.

Strong Audio Priors

While one of the promises of deep learning is models that learn from scratch, there may be cases where aspects of human perception are difficult or impossible to learn without some strong guidance. With that in mind, and in hopes of accelerating training, my generator and discriminator architectures have in-built priors based on two facts:

Musical audio contains information at many different time-scales
Our perception of sound depends a great deal on a decomposition of audio into many frequency bands by the cochlea

Multi-Scale Generator and Discriminator

To address the problem of information at vastly different time scales, I saw an opportunity to try a multi-scale analysis and synthesis idea that mirrors multi-resolution wavelet transforms or the Non-stationary Gabor Transform.

In most raw audio-producing neural network architectures I’ve encountered thus far, all audio and thus all frequency bands are produced at the same sampling rate. I wanted to try an architecture where groups of frequency bands were produced at an appropriate sampling rate, such that low frequency bands could be faithfully rendered with significantly fewer samples and only up-sampled at the last possible moment. My hunch was that over-sampled filters producing lower frequencies were likely to produce a lot of artifacts.

More concretely, at 22050hz, we might represent one second of audio with the following five bands or channels:

N samples	Start Hz	End Hz
1378	0	689
2756	689	1378
5512	1378	2756
11025	2756	5512
22050	5512	11025

So, we might decompose the following audio samples at 22050hz…

…into the following five bands

0hz - 689hz

689hz - 1378hz

1378hz - 2756hz

2756hz - 5512hz

5512hz - 11025hz

For this experiment, the top-level generator was composed of five distinct frequency band generators that produced audio on each of the five time scales represented above. Each sub-generator is conditioned on the same input spectrogram.

Similarly, the discriminator is composed of five distinct frequency band discriminators that analyze audio on each of the five time scales represented above. Each of these also have access to the conditioning spectrogram.

Frozen Filter Bank Layers

To mimic the decomposition of audio into constituent frequency bands by the cochlea, each sub-generator and sub-discriminator includes a frozen layer (i.e., a layer with no learn-able parameters) representing a bank of finite impulse response filters. This is similar to an approach used in the paper Speaker Recognition from Raw Waveform with SincNet where the first layer of a speech recognition model is initialized as a bank of finite impulse response filters, parameterized by the filters’ center frequencies and bandwidths.

In this case, the discriminator for each time scale will have a frozen first layer initialized as a linear-spaced filter bank spanning the frequency interval [nyquist / 2, nyquist] except for the lowest-frequency time scale, whose filter bank will span [0, nyquist].

The generator for each time scale will also include one of these frozen filter bank layers, this time as its last layer. The generator will produce audio by performing a transposed convolution with this layer as its last step, essentially summing together the excitations from each filter bank channel.

During training, the discriminator views audio as N distinct bands at different time scales and the generator produces this same representation. Only during inference does each generated band need to be resampled to the desired sample rate (22050hz in our case) and summed together to produce the final output.

Conditioned Audio Examples

Now that we’ve covered the modifications to the generator and discriminator architectures, let’s listen to some samples generated by the new model:

Real

Generated

Real

Generated

Real

Generated

Real

Generated

Real

Generated

Real

Generated

These samples tend to be less noisy and metallic-sounding than those produced with the baseline MelGAN architecture. To my ear, they are more faithful to the originals, and more musically pleasant. They aren’t without their problems, however. There are some significant phase issues, possibly due to the fact that each generator band is produced independently. The model also seems to have trouble faithfully reproducing lower frequencies which I find somewhat puzzling and surprising.

You can hear more generated examples here.

Future Work

Overall, the results are promising and lay the groundwork for a first step in our two-stage music synthesis pipeline. Next steps will include:

Beginning to work on contioned or unconditioned Mel spectrogram generation using this trained model as the spectrogram-to-audio vocoder
Exploring strategies for mitigating phase issues in this model
Exploring strategies for reproducing low frequencies more faithfully
Performing a more principled comparison of the original MelGAN model with this one, including analysis of parameter counts, audio generation performance and a more careful side-by-side analysis of generated samples

Cochlea: A RESTful API for Annotating Audio

Tue, 22 Oct 2019 12:00:00 +0000

Cochlea is an early-stage, RESTful API that allows users to annotate audio files on the internet.
Segments or time intervals can be annotated with text tags or other arbitrary data. This may not sound very exciting on its own, but I believe that these simple primitives make possible incredibly diverse applications tailored to the needs of electronic musicians, sound designers and other folks interested in playing with sound.

Before starting to dream aloud about the endless possibilities, a little about how I arrived here…

False Starts

I’m interested in building tools that allow musicians, sound designers and machine learning researchers to explore libraries of audio samples in new and intuitive ways that go far beyond traditional tag-based text searches. Text can be a great starting point, but indexes based on perceptual audio similarity or other features such as pitch or timbre offer much more exciting possibilities.
Whereas text-based approaches require painstaking manual tagging of vast quantities of audio, indexes that organize sound based on features derived directly from audio samples make it feasible to imagine the entire internet as your sample library!

With this ideal in mind, I’ve started and discarded several audio similarity search applications due to overly-rigid approaches. I’ve often settled on a single feature or similarity metric I think will work well and based the entire application around it. Inevitably, the search works well in some contexts and not so well in others. In addition, I’ve failed time and again to make indexing new sounds painless and I’ve eschewed more basic but necessary features, like allowing users to tag audio and search across those tags.

The common theme in all these ventures has been a lack of flexibility due to assumptions I’ve baked in much too early in the process. The RESTful API I introduced above is one possible answer to this problem, providing a simple platform on which all sorts of diverse applications might be built.

Now, to dig into the details…

API Resources

The experimental Cochlea API consists of just three simple resources:

`/sounds`

sound resources are really just pointers to audio that’s hosted somewhere out there on the internet. While it’s not totally necessary, ideally the servers hosting the audio content will conform to a basic interface:

The servers should support byte-range requests, making partial downloads of the files possible
The servers should allow for cross-origin requests by setting appropriate CORS headers, making it possible for front-end applications to request the audio and play it using the Web Audio API

It’s also worth noting that sound identifiers are ordered according to the time they were created, which will come in handy when we discuss featurebot and aggregator users in a bit.

`/annotations`

annotations describe all or part of a sound using text tags or any other arbitrary piece of data. The Cochlea API natively supports the creation and storage of text tags, but other arbitrary data describing sound segments can be hosted elsewhere. We might create annotations tagging a segment of audio as containing female speech or create an annotation that highlights an interesting segment of a longer audio sequence. We might also compute dense numerical features from the raw audio samples (such as short-time Fourier transform data, chroma, or MFCC data) and store them as NumPy arrays in an S3 bucket.

Just as servers hosting audio data should conform to a particular interface, servers hosting dense features or other arbitrary annotation data should ideally support byte-range and CORS requests.

Like sound identifiers, annotation identifiers are ordered according to the time they were created, which will again come into play when we discuss featurebot and aggregator users a little later.

`/users`

The third and final resource type we’ll discuss is the user type. There are a few different types to cover, and I think that this is where the platform really starts to get interesting.

Humans

Humans are the first and most obvious user type. These users can read sound and annotation resources and can create annotation resources of their own, usually using textual tags added using some graphical user interface.

Datasets

dataset users represent some sound collection or repository on the internet.
Some examples might include well-known audio datasets used by the machine learning community, such as NSynth or MusicNet.
Since these datasets often include structured data about their audio files, dataset users will generally create both sounds and annotations. For example, the MusicNet dataset includes detailed information about each note played in each piece, include onset time, duration, instrument and pitch.
The NSynth dataset tags each note with certain characteristics such as acoustic, percussive or electronic.

Featurebots

featurebot users listen to some or all sound or annotation resources, optionally filtering by the user that created the resource, and compute features, such as onset times, chroma or MFCC features and create pointers to the computed/derived data using new annotation resources. A few example applications might include:

a bot that computes onset times using librosa’s onset detection functionality
a bot that computes short-time Fourier transform data for each sound
a bot that listens for annotations from the short-time Fourier transform bot and computes chroma or MFCC data, thus beginning to form a distributed computation graph that transforms the raw audio

Since sounds and annotations have identifiers ordered according to the time they were created, bots need only remember the last id they processed and poll against sound or annotation resources to continually compute new features for incoming resources.

Aggregators

aggregator users are similar to featurebot users in that they listen to a stream of some or all sound or annotation resources, but these users only have read access and generally create alternative indexes over resources, making them searchable in novel ways. A few example applications might include:

a bot that listens for any annotation with a tag and creates a more full-featured text search including fuzzy matching or using sound/music-related word embeddings for high-quality semantic searches
a bot that computes low-dimensional embeddings (using a technique similar to one I covered in an earlier post) from audio or other derived features, making visual exploration possible in a user interface

These indexes might be published via a private or public REST API, allowing their consumers to navigate some or all of the audio available via the Cochlea API in interesting ways.

An Example User Interface

An early, alpha-stage user interface built atop the Cochlea API and the concepts outlined above can be found here. All the code for the example app can be found in the Github repo. It pulls together data contributed by several user, featurebot and aggregator users into an interface for sound discovery. The users leveraged include:

dataset users that contribute sounds from several datasets, including NSynth and MusicNet
featurebot users that compute alternate visualizations of audio, including short-time fourier transforms, chroma and MFCC features.
an aggregator user that embeds short segments of audio onto a three-dimensional sphere based on perceptual similarity, allowing users to navigate sound “space” using a Google Maps-like interface.

The user interface is a single-page Vue.js application hosted using Amazon S3 and Cloudfront. It communicates with the Cochlea API as well as the search API hosted by the aggregator user.

The API and web app are invite-only (for now), but please reach out if you’re interested in giving them a spin (in exchange for some feedback, of course)!

Audio Query-By-Example via Unsupervised Embeddings

Fri, 22 Feb 2019 12:00:00 +0000

A couple of months ago, I gave a talk at the Austin Deep Learning Meetup about building Cochlea, a prototype audio similarity search engine. There was a lot to cover in an hour, some details were glossed over, and I’ve learned a few things since the talk, so I decided to write a blog post covering the process in a little more detail.

Motivations and First Steps

There are countless hours of audio out there on the internet, and much of it is either not indexed at all, or is searchable only via subjective, noisy and relatively low-bandwidth text descriptions and tags. What if it was possible to index all this audio data by perceptual similarity, allowing musicians and sound designers to navigate the data in an intuitive way, without depending on manual tagging?

The first challenge in building this kind of index is producing a representation of short audio segments that captures important perceptual qualities of the sound. This is, of course, subjective, and somewhat task-dependent (e.g., am I searching for audio with a similar timbre, or just the same pitch, or maybe a similar loudness envelope?), but in this case, we’d like to find some embedding space where elements are near one another if a human would likely assign the two sounds to the same class. For example, two segments of classical solo piano music should fall closer together in the space than a segment of solo classical piano music and a segment of rock music.

Above, you can see a t-SNE visualization of the 128-dimensional embeddings we end up learning in this experiment. What’s interesting here is that we’re plotting by tags or labels, but those won’t be used during training at all! We’ll train a neural network that separates audio classes fairly well in an unsupervised manner.

Unsupervised Learning of Semantic Audio Representations

There’s a great paper called Unsupervised Learning of Semantic Audio Representations from Dan Ellis’ research group at Google that develops the kind of representation we’re after by first noting a few things:

certain types of transformations (e.g. pitch shifts and time dilation/compression) don’t typically change the general sound class
sounds that are temporally proximal (occur near in time) tend to be assigned to the same sound class
mixtures of sounds inherit the sound classes of the constituent elements

They leverage these observations to train a neural network that produces dense, 128-dimensional embeddings of short (approximately one second) spectrograms such that perceptually similar sounds have a low cosine distance, or small angle on the unit (hyper)sphere in an unsupervised fashion, without requiring a large, hand-labeled audio dataset to get started.

Deformations

First, to prove to ourselves that the observations are valid, we can apply the deformations to some audio and listen to ensure that some kind of perceptual similarity is maintained.

Deformation	Audio
original
pitch shift up
pitch shift down
time dilation
time compression
additive noise
temporal proximity beginning
temporal proximity end

These all sound like they belong to the same, or a similar sound class, to my ears at least.

Triplet Loss

Getting a little more formal, we’d like to learn a function or mapping such that simple cosine distance in the target or embedding space corresponds to some of the highly complex geometric relationships between the anchor and deformed sounds represented in the audio we’ve just listened to.

For this experiment, that function, which we’ll call $g$, will take the form of a deep convolutional neural network with learn-able parameters:

\[g: \mathbb{R}^{F \times T} \rightarrow \mathbb{R}^{embedding\_dim}\]

The original space is expressed as having dimensionality $F \times T$ because the input representation used in the paper is a spectrogram, with the $^F$ and $^T$ representing the frequency and time dimensions, respectively. We’ll go into more detail about how this representation will be computed in the next section.

Our dataset will consist of $N$ “triplets” of data:

\[\tau = \{ t_i\}_{i=1}^N\]

Each triplet will look something like this:

\[t_i = (x_{a}^{(i)}, x_{p}^{(i)}, x_{n}^{(i)})\]

and each member of the triplet will be a spectogram:

\[x_a^{(i)}, x_p^{(i)}, x_n^{(i)} \in \mathbb{R}^{F \times T}\]

Each $x_a$ represents an anchor audio segment, each $x_p$ represents a positive example, i.e., the anchor with one of our audio deformations applied, or another audio segment that occurs near in time to the anchor, and each $x_n$ represents a negative example, which we’ll choose by picking any other audio segment from our dataset at random. Given a large enough dataset, our random strategy for choosing the negative example is probably fairly safe, but we will take some care to not accidentally choose negative examples that are actually more perceptually similar to the anchor than the positive example.

We’ll minimize a loss with respect to our network parameters that will try to push the anchor and positive examples closer together in our embedding space, while it also pushes our anchor and negative examples further apart, given some distance function $d$:

\[\mathcal{L}(\tau ) = \sum_{i=1}^{N}\left [ d(g(x_a^{(i)}), g(x_p^{(i)})) - d(g(x_a^{(i)}), g(x_n^{(i)})) + \delta \right]_+\]

Our network’s job is to ensure that the distance $d$ from anchor to positive examples is less than the distance from anchor to negative examples by some positive margin $\delta$. The paper’s authors used squared l2 distance as their $d$:

\[d(a, b) = \left \| a - b \right \|_2^2\]

While I used cosine distance:

\[d(a, b) = 1 - \frac{a \cdot b}{\left\| a \right\|_2 \left\| b \right\|_2}\]

The $_+$ here indicates the hinge loss, which really just means that our loss goes to zero if it’s less than this margin $\delta$. As usual, since we can’t optimize this term over all $N$ triplets, we’ll optimize the learn-able parameters using minibatches of data.
Additionally, we’ll see later that because of the way we’re sampling these triplets, our dataset is effectively unbounded.

Log-Scaled Mel Spectrograms

The paper’s authors compute their input spectrograms by:

using short-time Fourier transforms
discarding phase
mapping the resulting linear-spaced frequency bins onto an approximately log-spaced Mel scale
applying a logarithmic scaling to the magnitudes

This is a fairly standard pipeline for computing a perceptually-motivated audio representation. They go on to apply the various deformations mentioned above directly to this time-frequency representation. I was a little uneasy with this approach because:

mapping onto the Mel scale is done after the fact, from linear-spaced frequency bins that don’t have the frequency resolution we’d like in lower frequency ranges and are needlessly precise in higher frequency ranges
the representation isn’t invertible, as phase is discarded and many frequency bins in the higher ranges are averaged together, meaning that it’s impossible to know if the transformations actually sound plausible.

For this project, I decided to perform all deformations in the time domain, mostly to convince myself that all of them continued to sound plausible. I also opted to compute the time-frequency representation using a bank of log-spaced Morlet wavelets, so that perceptually-inspired frequency spacing and time/frequency resolution trade-offs could be accounted for from the outset, instead of being something of an afterthought. A subset of our bank of filters will look something like this:

This filter bank can be thought of as a 1D convolutional layer with frozen parameters. Visualizing all the channels and weights of our convolutional filter bank at once yields something like this:

One nice side effect of this decision is that now the computation of the time-frequency transform is performed as part of our neural network, or as part of our function $g$, by a convolutional layer with frozen parameters, meaning that our input representation is now just $\mathbb{R}^T$, with $^T$ being the number of audio samples in each segment. While the filter bank is not included in the network’s learn-able parameters for this initial experiment, a learn-able time-frequency transform is an open possibility in future iterations of this work.

The time-frequency representations we compute will look something like this:

You can see the implementation of our learn-able $g$ function here, which uses the zounds.learn.FilterBank module to compute the time frequency-representation before passing it on to a stack of convolutional layers. Also, this jupyter notebook offers a more in-depth exploration of the motivations behind using a bank of log-spaced Morlet wavelets to compute our log-frequency representation.

Within-Batch Semi-Hard Negative Mining

The paper’s authors also use a technique called “within-batch, semi-hard negative” mining to make training batches as difficult as possible for the network. Since our loss encourages our learn-able function to widen the gap between the anchor-to-positive and anchor-to-negative embedding distances, we’d like to arrange our training data in such a way that that gap between embeddings is narrowed, forcing our network to work harder to push anchor and negative embeddings further apart, and anchor and positive embeddings closer together. Picking the hardest negative example for each anchor requires that we compute the distance from every anchor embedding to every negative embedding. This is obviously prohibitively slow to do with our entire dataset, and again, isn’t really possible due to the fact that we’re building batches on the fly, so we’ll instead do it only within each batch.

You can see the results of this process on several batches of size eight in the graph below. After re-assigning the negative examples, the new anchor-to-negative distances (in red) for each example tend to be closer to the anchor-to-positive distances (in blue) than the original anchor-to-negative distances (in green), while still maintaining a positive margin. It’s worth noting that the technique I’m using to re-assign the negative examples allows for the same negative example to be assigned to multiple triplets if it meets the criteria we set out above.

There’s a little more detail about the algorithm I used to perform within-batch semi-hard negative mining in this jupyter notebook.

Training Data

The paper’s authors used Google’s AudioSet for their training data, but I chose to pull together around 70 hours of sound and music from various sources on the internet, including:

While AudioSet is fantastic and enormous, my ultimate goal is to index audio on the internet in the same way Google that indexes documents, so I wanted to get some practice scraping audio and related metadata from disparate sources on the internet as a part of this work.

Training

Now that all the details are ironed out, training the network is fairly straightforward and follows this procedure:

choose a minibatch of anchor examples at random
either apply a deformation (e.g., time stretch or pitch shift) or choose segments that occur within ten seconds of each anchor to produce positive examples
choose negative examples at random to complete the triplets
use our network $g$ in its current state to compute embeddings for the anchor, positive, and negative examples
perform within-batch semi-hard negative mining, re-assigning some or all of the negative embeddings to make each example in the batch “harder”, i.e., the loss greater
compute triplet loss, calculate gradients via backpropagation, and update our network’s weights

Building an Index with Random Projections

Naively, if we’d like to use our new embedding space to support query-by-example, it means computing the distance between a query embedding and every other segment in our database! A brute-force distance computation over every embedding obviously won’t scale well, but thankfully there are plenty of techniques available for performing approximate K-nearest neighbors search over high-dimensional data in sub-linear time.

One such approach that gets us log-ish retrieval times builds a binary tree from every embedding in our dataset by choosing a random hyperplane for each node in the tree and bisecting that node’s data based on which side of the hyperplane each data point lies. Erik Bernhardsson took this basic approach while working on music recommendations at Spotify, along with some clever tricks to improve accuracy, and open-sourced his work as a library called Annoy (Approximate Nearest Neighbors Oh Yeah).

Here’s a visualization of a single hyperplane tree built from our learned embeddings:

The first trick introduced is simply to search both of a node’s subtrees when a query vector lies sufficiently close to that node’s hyperplane. The intuition here is that good candidate vectors that lie just on the other side of the hyperplane “fence” would be missed, despite the fact that they’re nearby.

The second trick is to build multiple trees. All of the trees can be searched in parallel by building a priority queue of nodes to search, where node priority is determined by how far the query vector lies from that node’s hyperplane. As we observed when discussing the first trick, hyperplanes that are far from the query vector are more likely to catch nearby candidates, so we’ll prefer those nodes, searching until we’ve reached some threshold of candidates we’d like to consider.

Finally, we perform a brute-force distance search over our pool of narrowed candidates, which is typically orders of magnitude smaller than our total index size.

In the graph below, you can look at the speed/accuracy tradeoffs we can make for our search using from one to 64 trees. Accuracy here (ranging from 0-1) on the x-axis is defined as the overlap with results from our brute-force source-of-truth search. There are other parameters that can be tweaked, such as our threshold for searching both paths for a given node, and the size of the candidate pool over which we’ll perform our final brute-force search, but this holds those parameters constant, only varying the number of trees we’re using. These timings are for searches over around 60 hours of audio.

It’s worth noting that these timings are based on a pure-Python/Numpy implementation of the algorithm that I wrote for learning purposes; timings from Annoy’s C++ implementation would likely look better.

Conclusion and Next Steps

We’re off to a pretty good start. We’ve now got:

A way to learn a representation that captures perceptual similarity in an unsupervised way. The more data we can feed it, the better!
A pretty fast way to search

There are some interesting avenues to explore going forward, however:

Does vector arithmetic work for these embeddings just like it does for word embeddings (i.e., the classic king - man + woman = queen example)? Can we find a piano and violin duet by adding their “solo” embeddings together?
Could these embeddings be used in audio synthesis as conditioning vectors for a WaveNet-like model or a conditional WaveGAN?
Can we use these embeddings to predict tags for unlabelled audio for enhanced text search?
Is it possible to further embed our representations in a much lower two or three-dimensional space, making the navigation of audio segments visual and intuitive (our t-SNE visualization above points to “yes”)?

Resources

You can find slides from the original talk here, and watch a video of it on YouTube.
All the code for this experiment is on GitHub
Unsupervised Learning of Semantic Audio Representations is the paper on which this experiment is based
There’s a good blog post where Annoy’s author explains it in detail and an talk that covers the same material on YouTube.

Perceptual Audio Loss

Thu, 07 Jun 2018 12:00:00 +0000

Today, I perform a small experiment to investigate whether a carefully designed loss function can help a very low-capacity neural network “spend” that capacity only on perceptually relevant features. If we can design audio codecs like Ogg Vorbis that allocate bits according to perceptual relevance, then we should be able to design a loss function that penalizes perceptually relevant errors, and doesn’t bother much with those that fall near or below the threshold of human awareness.

Given enough capacity, data, and time, a model trained using mean squared error will eventually produce pleasing samples, but can we produce those same samples with less of all three?

I used zounds and pytorch to build a small experiment where I dip the first half of my pinkie toe into these waters.

The Experiment

In this experiment, I’ll train a “generator” network to transform a fixed noise vector (128-dimensional, drawn from a gaussian distribution with zero mean, unit variance, and a diagonal covariance matrix) into a single fixed audio sample of dimension 8192 (~.75 seconds at 11025hz sampling rate). I’ll perform this same experiment with five different audio samples, and two different loss functions:

The Perceptually-Inspired Loss

The PerceptualLoss is very rudimentary, but hopefully captures some characteristics of early stages in the human auditory processing pipeline, namely:

an FIR filter bank, whose filters’ center frequencies lie along the Bark scale
half-wave rectification (AKA ReLU)
a logarithmic, or decibel-like amplitude scaling

The network architecture, as well as the weight initialization scheme (but not the exact initialized weights) is held constant as we subjectively evaluate the performance of our two loss functions on five different audio files containing:

For each (audio sample, loss) pair, the generator network is given 1000 iterations to learn to transform the fixed noise vector into the given audio sample, and “checkpoint” audio samples are recorded from the network every 250 iterations.

Put another way, when over-fitting our low capacity network to a dataset of size one, what does the loss emphasize?

Inspiration and Previous Work

This approach was inspired by the Deep Image Prior paper. While that paper sought to understand biases inherent in neural network architectures by keeping the loss fixed, and varying network structure, this experiment holds the architecture fixed, and tries to understand the contribution of different losses.

Both Autoencoding beyond pixels using a learned similarity metric and Generating Images with Perceptual Similarity Metrics based on Deep Networks explore losses that go beyond simple per-pixel (or per-sample) metrics.

The Code

The code for this experiment can be found on GitHub.

The Results

In each section below, you can hear generations from the network every 250 iterations.

Richard Nixon Speaking

Original

Mean Squared Error

250 Iterations

500 Iterations

750 Iterations

1000 Iterations

Perceptual Loss

250 Iterations

500 Iterations

750 Iterations

1000 Iterations

Conclusions

For this sample, MSE at 250 iterations is noisy and unintelligible, while the perceptual loss is already starting to be intelligible at the same point. By the end of the experiment, the samples generated by the network trained with MSE are fairly intelligible, but completely missing the sibilant at the end of the word “practice”. The network trained with perceptual loss captures this plainly and simply.

Bach Piano

Original

Mean Squared Error

250 Iterations

500 Iterations

750 Iterations

1000 Iterations

Perceptual Loss

250 Iterations

500 Iterations

750 Iterations

1000 Iterations

Conclusions

Even at the final iteration, generations from the network trained with MSE are “blurry”, and noisy. Generations from the network trained with perceptual loss are also a bit noisy, but overall, the final result is significantly clearer.

Top Gun Soundtrack

Original

Mean Squared Error

250 Iterations

500 Iterations

750 Iterations

1000 Iterations

Perceptual Loss

250 Iterations

500 Iterations

750 Iterations

1000 Iterations

Conclusions

This sample has a pretty broad frequency range, and a lot’s going on: bass guitar, synth, female vocals, and a snare drum. At 250 iterations, generations from the network trained with MSE are a noisy, low frequency mess, while generations from the network trained with perceptual loss are beginning to be intelligible.

The network trained with MSE does OK by the final iteration, but the network trained with perceptual loss does a much better job of generating the crisp high end, especially notable in the crack of the snare drum.

Kevin Gates

Original

Mean Squared Error

250 Iterations

500 Iterations

750 Iterations

1000 Iterations

Perceptual Loss

250 Iterations

500 Iterations

750 Iterations

1000 Iterations

Conclusions

While this sample covers a frequencies similar to those from the “Top Gun” soundtrack, both networks seem to struggle a bit more. Unsurprisingly at this point, the network trained with perceptual loss has managed to capture the sharp attack of the snare drum, as well as make the words “I get…” at the end of the sample almost intelligibly. Generations from both networks are noisy, but the MSE generations are much more so.

Drumkit

Original

Mean Squared Error

250 Iterations

500 Iterations

750 Iterations

1000 Iterations

Perceptual Loss

250 Iterations

500 Iterations

750 Iterations

1000 Iterations

Conclusions

The difference between the two losses is not as apparent here, although there’s certainly more definition in the mid range of the tom from the network trained with perceptual loss. Interestingly, the perceptual loss network seems to plateau (or close to it) by iteration 250, while the MSE network starts pretty terribly and makes its way slowly to the same point.

Final Thoughts

While the results aren’t terribly dramatic, it does seem that the perceptually-inspired loss function has helped to use network capacity on more salient (from a human perspective) features of each sound. In general, this loss promoted:

crisper mid and high frequencies
less noise
quicker convergence on something intelligible

Obviously, my “perceptual model” is dead-simple, and certainly dead-wrong in many cases. Could we make even better use of model capacity by adding things that perceptual audio codecs use to save bits, like tonal and temporal masking? Our current model will penalize the generator for incorrectly producing a frequency that may be inaudible to the listener!

Addendum

The most realistic audio generation models in the recent past, namely, WaveNet and SampleRNN have some interesting (and surprising) properties:

they model audio at the PCM sample level, and not at the FFT frame level, which the vast majority of previous work has preferred
they are auto-regressive and recurrent models, respectively, ultimately meaning that each successive sample is conditioned on all previous samples. Sample generation is thus serial, and not parallelizeable (later work, especially on WaveNet makes this not entirely true)
the problem is modelled in both cases as a classification problem, and not a regression problem. Samples are modeled as discrete classes, and the relationships between those classes must be learned, from scratch, by the model

The question of whether it’s reasonable to expect to generate audio samples in parallel, as if they were all independent of one another is an interesting one, but isn’t really the topic of this post.

What’s most interesting in this context is the re-framing of the problem as a classification problem. This apparently has a couple benefits:

While using MSE to do regression assumes a gaussian probability distribution, this approach allows us to learn any arbitratily complex distribution
With MSE and naive per-raw-sample regression, it’s possible to do pretty well by modelling low frequencies (since that’s where most of the energy lies), and making a lot of noise. By framing the problem as classification, it becomes a lot harder for the model to get “partial credit” and coast along happily

This approach is very clever, but doesn’t totally sit right with me, because audio samples are continuous, so why can’t we model them as such? Is the problem a missing perceptual model that maps groups of “raw” audio samples into a more perceptually appropriate space? Would regression in that space work better?