Science
How to convert emotions into music
Developments in science and research have resulted in exciting innovations that allow us to augment people's abilities in radical new ways using neural networking and machine learning.
Thanks to feedback loops that provide data to reinforce learning and artificial intelligence that is trained to recognise patterns, we now have the neccessary tools to make it possible to elegantly interface with machines in natural and intuitive new ways that constantly evolve, refine and improve the more they are used.
The PhotoSYNTH, and other AMIC instruments, are developed from the ground up to be accessible and inclusive, with the support, training and feedback from hundreds of Disabled and differently-abled people. This ensures that every step is a step forwards, towards improvement.
By tracking your motions and emotions in real-time, the PhotoSYNTH converts your everyday movements into music and captures the subtleties and nuances of your expressions and performance.
An expressive musical instrument is at the zenith of complicated fidelity, some are incredibly complicated machines that require time to master. The same techniques used to simplify the complex act of making music can be used for simplifying the control of other things.
Holographic PhotoSYNTH, AMIC, London UK.
Digitising the human form
Finding a purpose for your reflection
The Photo
To help a computer understand humans, we must first allow them to see us by giving them eyes, and for that we use a digital video camera. This will be our primary control mechanism.
Using a machine learning algorithm trained on millions of face photographs taken from many directions, the computer can examine the image for features it may recognise by comparing them against the photos in its memory. A recognisable nose, a mouth, an eye...
Using these facial landmarks as references, the user's orientation, expressions and mood can be digitally captured and represented in 3D space which can quite accurately recreate the shape of the face with all its contours and character.
"Computer vision"
How to read a human being
Computers are very good at recognising patterns. If you keep showing a computer ducks, eventually it will realise what ducks have in common - a bill, wings, webbed feat, a distinct waddle and some quacks. With practice it becomes an expert it duck detection.
The same can be true for recognising human characteristics such as moods and emotions. Being happy presents many clues and indications when studied, most obvious is smiling, but joy also brings wide eyes, crows feet, and delight raises eyebrows. Frowns and furrowed brows indicate signs of melancholy and malaise.
Both happiness and sadness are easy to spot. Like ducks.
A well-trained machine can read you like a book...
But can it play you like a musical instrument? π¦
If it looks like a duck, and quacks like a duck, we have at least to consider the possibility that we have a small aquatic bird of the family Anatidae on our hands
Douglas AdamsDirk Gently's Holistic Detective Agency
Technology :
Imagine a world where rather than having to learn a musical instrument, the musical instrument could learn how to play you...
By using motion capture and augmented reality, heads are converted into musical instruments where smiles create musical notes, movements make melodies and facial expressions shape the sound with audio filters.
If there was a magical music spell, what musical skills would you wish for?
Any sufficiently advanced technology is indistinguishable from magic.
Arthur C Clarke
The Shape of Sound
Unique Sonic characteristics - The Sound of Shapes
Each sound has a distinct shape, similar to a fingerprint or snowflake. This affects how it is perceived by your ears and how the sound travels and bounces off things.
Sound travels at the same speeds as gases do, because that is how they travel, they are gas powered.
Instead, variation in sound comes from their shape, their repetition and their strength. Tiny fragments of sound when repeated hundreds of times a second can create a musical sound similar to what an instrument might make. We call these sound shapes timbres. Timbres can be made from any fast repeating sound including noises.
The frequency of the timbre, the rate that the noise rapidly repeats, dictates its unique sonic pattern and musical tone, which forms a note. Each musical note is determined by the amount of times that the pattern repeats in a fixed time period, often refered to as its vibration, wavelength or frequency.
Different shapes affect the air as they travel as well as any physical matter encountered on the way, including vibrating the little hammers inside your ears, where these sonic bubbles tap dance the music directly into the brain to decode like morse code.
Slow repeating sounds (low frequency) are perceived as deep or bassy, whist the faster they vibrate, the more high pitched they sound.
Play with the keyboard below to see how the shape varies between high notes and low notes..
Percussion
Sounds have shapes
The sonic characteristics of Sound
As sound moves it leaves remnants of itself on things it touches, like sonic snail trails, vibrating and shedding frequencies and reverberations.
These collisions alter the acoustics and harmonics of the sound and add tone and character.
When many notes exist in the same space they join together and form a new sound. Their waves double up or cancel out, creating entirely new waves and sounds.
Certain frequencies when played together, join up and form a chord, and if the notes are harmonic, and complement each other, can trigger strong emotional feelings in the person hearing them.
There are many different types of chords, but the most common ones in western music are the major and minor chords. Major chords sound happy, bright and cheerful, whereas minor chords can sound sad and somber yet compelling.
There is much research into the topic of how sounds affect humans, but it is commonly thought that many of the emotional triggers caused by sound are rooted in our historic relationship with other people and animals. From the calming songs of the birds, or the purring of the pussycat, to the cries of a baby or the squeal of an animal in distress.
Many sounds have historical contexts that served an important role in our evolution as survival skills. We may never find out the true origins of these musical-emotional relationships, but they appear to be deeply entrenched and universal.
Evocative sounds include the growl of a bear, the roar of a lion, bird song, the patter of rain and the clap of thunder. These all form part of the ecological audio landscape that has surrounded us for millenia, speaking to us long before we had a language.
The dulset tones of "mumese" are universal, a high pithed, softer sounding way to communicate with children and cute animals. This inate skill also appears to be universal. Smaller things make higher pitched sounds, so we adopt these tones in an effort to communicate with them.
Music is an unconscious exercise in metaphysics in which the mind is unaware that it is philosophizing.
Arthur SchopenhauerThe World as Will and Representation
Shapes & Patterns
Chords & Patterns
When a musical note is played, the air pulses in patterns which is heard in our ears in the form of vibrations.
All that music is, is repetition on many levels.
Repetition forms music through many orders of magnitude, from the microspcopic vibrating patterns that form timbres, that at certain vibrations sound like notes, to the repetition of notes that form melodies, all the way up to the macroscopic patterns of repetition where the percussion forms rhythms and grooves.
Our ears respond to repeating waves, and our brains yearn for patterns to discover, and information to decode.
Repetition serves as the handprint for human intent.
Elizabeth Hellmuth Margulis
Certain combinations of sounds when played together are refered to as a chord.
Throughout time people have searched for chords to represent emotions for their melodies.
Below are some of the most famous chords that all have a mathematical connection. See if you can figure them out by observing the keyboard below during the next section.
Music is an unconscious exercise in mathematics in which the mind is unaware that it is calculating.
Gottfried Wilhelm Leibniz
Accessible Music Instrument Collection, AMIC
The PhotoSYNTH:Sympathetic Synthesizer and the Holographic PhotoSYNTH_4D_ are available at the AMIC.
-
A ground-breaking national collection, representing state-of-the-art of accessible instrument design.
-
A world-class resource for Disabled people to explore technological possibilities for making music.
-
A focal point for the development of cutting-edge accessible instrument technology.
Accessible Musical Instrument Collection
Access to the right musical instruments and technology can be powerful and transformative.
As well as establishing a collection of existing accessible tech, the AMIC will commission and support the creation of the next generation of accessible instruments. The Collection will be as varied as possible, from high-end commercial instruments through to DIY experimental prototypes and bespoke instruments.
We will continue to build the collection as new instruments come to light, working with makers, hackers and music tech companies to make sure it reflects the most up to date technology. AMIC will be housed in an accessible space at Drake Musicβs HQ and will be available to visit. Disabled musicians will be able to visit the collection, test out different instruments using an accessible workstation and (hopefully) find one which suits them. Makers can come and be inspired or see what has already been built. Academics can call upon it for research and teachers can explore options for their students.
The story so far...
-
Holographic PhotoSYNTH at the Electromagnetic Fields Festival
Photograph by Alexander Baxevanis CC2.0 -
Zoom Stream -
Gawain Hewitt playing with the PhotoSYNTH II -
Holographic PhotoSYNTH 4D being played at Electromagnetic Fields
History
-
Gawain Hewitt playing with PhotoSYNTH II -
Media City Installation -
DMLab Demonstration