Resonance Inference
Table of Contents
Resonance Inference
One path toward sparse representations of musical audio is to make assumptions about the
system or underlying process that produced the signal. Despite the prevalence of synthesizers and electronic music,
much of the audio and music we hear is produced by objects in the physical world resonating.
While a model that makes no assumptions would be more general, such a model might also
waste a lot of capacity learning about the physics of sound as well as the dynamics of a musical performance.
The event generator models I've used in my
iterative decomposition experiments use cascading convolutions to model
the resonances of musical instruments (and the rooms in which they are played) and my recent work on
playable state-space models also explored the modelling of physical resonance
using a fully-linear network.
This mini-experiment takes these ideas a bit further, overfitting a network to a single audio segment of ~6 seconds.
The "prior" here is that the signal was produced by injecting energy into a low-dimensional "control plane", which is
connected to multiple layers of resonances, gains and non-linearities. Ideally, this model might be able to capture
some of the nuances of physical resonance that the state-space model could not, and could lead to highly sparse "scores"
which can be seen when looking at the "Control Signal" section of each example.
Statefulness
One thing that the fully event-based approach misses is that resonating instruments/objects are stateful; If I pluck
an already-vibrating string, I'll get a different sound than if I had plucked a string at rest. This model's RNN-like
structure makes it at least possible to capture those kinds of dynamics. It is, of course, a tradeoff, and I am
thinking about ways to integrate this model as an event generator in the iterative decomposition framework.
Object Deformations
Event generators in the iterative decomposition models interpolate between
several different convolutions to mimic the deformation of resonating objects. For example, I might pluck a string
and then bend it, causing the resonant properties to change. This model does not yet incorporate deformations, but might
do so more efficiently than the long-convolution models, since we operate in a recursive, frame-by-frame fashion.
Model and training code is here!
Cite this Article
Examples
Example 1
Original
Reconstruction
Control Signal
The "control signal" has 860 non-zero elements and a sparsity of 10.50%.
The control signal has a total of
512 x 16 = 8192 elements
Random control signal
We produce a random, sparse control plane, interfaced with the overfit resonance model.
Example 2
Original
Reconstruction
Control Signal
The "control signal" has 693 non-zero elements and a sparsity of 8.46%.
The control signal has a total of
512 x 16 = 8192 elements
Random control signal
We produce a random, sparse control plane, interfaced with the overfit resonance model.
Example 3
Original
Reconstruction
Control Signal
The "control signal" has 853 non-zero elements and a sparsity of 10.41%.
The control signal has a total of
512 x 16 = 8192 elements
Random control signal
We produce a random, sparse control plane, interfaced with the overfit resonance model.
Example 4
Original
Reconstruction
Control Signal
The "control signal" has 604 non-zero elements and a sparsity of 7.37%.
The control signal has a total of
512 x 16 = 8192 elements
Random control signal
We produce a random, sparse control plane, interfaced with the overfit resonance model.
from typing import Dict
import torch
from torch.optim import Adam
from conjure import CompositeComponent, Logger, conjure_article, AudioComponent, S3Collection, ImageComponent, \
TextComponent, CitationComponent
from data import get_one_audio_segment
from freqdomain import construct_experiment_model, reconstruction_loss, sparsity_loss
from modules.infoloss import CorrelationLoss
from util import device
remote_collection_name = 'freq-domain-resonance-demo'
def to_numpy(x: torch.Tensor):
return x.data.cpu().numpy()
def reconstruction_section(logger: Logger, n_iterations: int) -> CompositeComponent:
n_samples = 2 ** 17
samplerate = 22050
model = construct_experiment_model(n_samples)
optim = Adam(model.parameters(), lr=1e-3)
#loss_model = CorrelationLoss(n_elements=2048).to(device)
target = get_one_audio_segment(n_samples, samplerate, device=device)
target = target.view(1, 1, n_samples).to(device)
for i in range(n_iterations):
optim.zero_grad()
recon, control_signal = model.forward()
recon_loss = reconstruction_loss(recon, target)
recon_loss = recon_loss #+ loss_model.forward(target, recon)
loss = recon_loss + sparsity_loss(control_signal)
loss.backward()
print(i, loss.item())
optim.step()
with torch.no_grad():
final_recon, final_control_signal = model.forward()
rnd, _ = model.random()
nonzero = model.nonzero_count
sparsity = model.sparsity
markdown_text = f'''
The "control signal" has {nonzero} non-zero elements and a sparsity of {(100 * sparsity):.2f}%.
The control signal has a total of
`{model.n_frames} x {model.control_plane_dim} = {model.n_frames * model.control_plane_dim} elements`
'''
_, orig = logger.log_sound('original', target)
_, recon = logger.log_sound('recon', final_recon)
_, random = logger.log_sound('random', rnd)
_, cp = logger.log_matrix_with_cmap('cp', final_control_signal[0], cmap='hot')
orig_audio_component = AudioComponent(orig.public_uri, height=200)
recon_audio_component = AudioComponent(recon.public_uri, height=200)
random_audio_component = AudioComponent(random.public_uri, height=200)
control_plane_component = ImageComponent(cp.public_uri, height=200)
text_component = TextComponent(markdown_text)
composite = CompositeComponent(
original=orig_audio_component,
reconstruction=recon_audio_component,
random=random_audio_component,
cp=control_plane_component,
sparsity_text=text_component,
)
return composite
def demo_page_dict() -> Dict[str, any]:
remote = S3Collection(
remote_collection_name, is_public=True, cors_enabled=True)
logger = Logger(remote)
n_iterations = 1000
example_1 = reconstruction_section(logger, n_iterations=n_iterations)
example_2 = reconstruction_section(logger, n_iterations=n_iterations)
example_3 = reconstruction_section(logger, n_iterations=n_iterations)
example_4 = reconstruction_section(logger, n_iterations=n_iterations)
citation = CitationComponent(
tag='johnvinyardresonanceinference2024',
author='John Vinyard',
url='https://blog.cochlea.xyz/resonance-inference.html',
header='Resonance Inference',
year='2024')
return dict(
example_1=example_1,
example_2=example_2,
example_3=example_3,
example_4=example_4,
citation=citation
)
def generate_demo_page():
display = demo_page_dict()
conjure_article(
__file__,
'html',
title='Resonance Inference',
**display)
if __name__ == '__main__':
remote = S3Collection(
remote_collection_name, is_public=True, cors_enabled=True)
remote.destroy(prefix=b'')
generate_demo_page()
Back to Top