artwork

Four Synesthetic Neural Networks

Details

• Artist: Ian Patrick Cunningham
• Year: 2018
• Exhibition: Kinesis 2018
• Medium: Interactive installation
• Url: https://www.instagram.com/p/BqAw2vMjePs/
• Location: Saint Louis, Missouri, USA

Description

AI art installation

Installation was 4 projectors playing video generated by neural networks, synced to ~30 minutes of original music.

Each projection is comprised of two separate neural networks, spliced into one.

The first neural network was Google's NSynth (a Recurrent Autoencoder), part of their Magenta project: https://magenta.tensorflow.org/nsynth

The other neural network was a Convolutional Autoencoder. It was based partially on a traditional Convolutional Autoencoder, with one neural network having the addition of multiple latent spaces at different resolutions throughout the network.

Both neural networks were trained independently.

When generating the video, music was sent through the first half of Nsynth, which encoded it into (N, 16) dimensions, N being a number of timesteps, depending on the duration of the input audio. These (N, 16) encodings were then sent through the second half of the Convolutional Autoencoder, which decoded the encodings into images.

It should be noted that the Convolutional Autoencoders were trained at different resolution input/outputs: 64x64 and 512x512. The 512x512 autoencoder had multiple latent spaces, while the 64x64 autoencoders did not.