Smile Powered Pop Music with the PhotoSYNTH and friends!

Royal Society Summer Science Exhibition 2025

(this is a musical website)

Skip directly to the main content.

Smile Powered Music!

Converting happiness into music

MUSICALSUPERPOWERSFORYOUANDYOURFRIENDS!β€’SMILEPOWEREDSYNTHESIZERβ€’INSTANT
Instant Musical Superpowers, for you and your friends!

Drake Music presents, the PhotoSYNTH, the happiest way to make music!

The smile powered synthesizer that grants you and your friends radical new musical superpowers.

This magic mirror tracks your movements, moods and expressions and converts them into music β€” giving your reflection a musical purpose!

It is fun and easy to make a musical masterpiece, especially if you collaborate with friends.

Rather than playing a musical instrument, this musical instrument plays you!

Designed from the ground up to be the most accessible musical instrument possible with the sole intention of empowering everybody to make lovely music! Express yourself!

Magic Mirror on the wall,
who's the fairest of them all?

Without any training you can instantly make great music using movements familiar to you. Easy... Like a Sunday morning!

Created alongside Drake Music Charity, specialists in accessible technology, and with the help & guidance of hundreds of differently-abled people, this mixed-reality pop music machine will put a smile on your face and make it musical =D

The PhotoSYNTH is a
  • Collaborative
  • Multiplayer
  • Mixed Reality
  • Sympathetic Synthesizer
  • Musical Instrument
  • Science & Research Project

Summer Science Exhibition 2025

Ever dreamed of being more musical?
Well alright... Hold tight!

The Summer Science Exhibition is a free six-day festival showcasing the science shaping our future and will be held at the Royal Society in London from to .

Come and join us for a fun musical celebration of inclusive technology, play with some of the newest instruments from the Accessible Music Instrument Collection and learn all about the research and science that made them possible.

Hearing loop enabled venue. The exhibition will feature a series of interactive installations, hands-on demonstrations, musical performances and talks.

Located in the heart of London

Between Buckingham Palace and Trafalgar Square, the Royal Society is located in a fully accessible building that is wheelchair friendly and has hearing loops enabled, quiet spaces and a cafe.

Show full facilities
Canteen on site. Wheelchair accessible venue. Hearing loop enabled venue. Accessible floors. Working dogs welcome. Quiet room.
Available facilities
Please contact the Royal Society via email if you have any special requirements.
Wheelchair accessible There is both a wheelchair ramp and a wheelchair lift at the entrance. The interior is wheelchair accessible with large lifts and a powerchair elevator.
Lifts, elevators and stairs Lifts are available for access to the basement level or first floor. There is step-free access.
Hearing loop Frequencies may change depending on which exhibition and will be displayed on signs around the event.
Signage There will be well lit signage in simple language to guide your day as well as floor plans and an information pack.
Guide dogs The building welcomes helpful animals and will provide water in bowls by the entrance and faces a small park where you can take your dog.
Accessible toilet There is an accessible toilet on the ground floor behind the reception area and an ambulant toilet in the basement. There are baby changing facilities in all the toilets in the basement and in the accessible toilet behind reception. Please note that the baby change facilities in the accessible toilets should only be used by parents/carers that specifically require the use of an accessible toilet, as baby changes facilities are available elsewhere.
Quiet space There is a quiet room on the ground floor next to the marble hall. Please ask the reception desk if you need access to this space.
View Larger Map
The Royal Society is located at :
6-9 Carlton House Terrace, London, SW1Y 5AG, United Kingdom
Getting there

Underground stations: Charing Cross and Piccadilly Circus Bus routes: 12, 13, 15 and 453. Coach station: Connections from Victoria coach station

Learn more at the official Royal Society Summer Exhibition website

Science

How to convert emotions into music

Developments in science and research have resulted in exciting innovations that allow us to augment people's abilities in radical new ways using neural networking and machine learning.

Thanks to feedback loops that provide data to reinforce learning and artificial intelligence that is trained to recognise patterns, we now have the neccessary tools to make it possible to elegantly interface with machines in natural and intuitive new ways that constantly evolve, refine and improve the more they are used.

The PhotoSYNTH, and other AMIC instruments, are developed from the ground up to be accessible and inclusive, with the support, training and feedback from hundreds of disabled and differently-abled people. This ensures that every step is a step forwards, towards improvement.

By tracking your motions and emotions in real-time, the PhotoSYNTH converts your everyday movements into music and captures the subtleties and nuances of your expressions and performance.

An expressive musical instrument is at the zenith of complicated fidelity, some are incredibly complicated machines that require time to master. The same techniques used to simplify the complex act of making music can be used for simplifying the control of other things.

The Holographic PhotoSYNTH being presented at the AMIC, Accessible Musical Instrument Collection.
Every articulation can become an audio revelation!
Holographic PhotoSYNTH, AMIC, London UK.

Digitising the human form

Finding a purpose for your reflection

The PhotoSYNTH is a mixed-reality audio synthesizer that uses real-time motion capture with facial-landmark tracking and emotion detection to enhance your movements, and the movements of your friends, with radical new musical superpowers.

To help a computer understand humans, we must first allow them to see us by giving them eyes, and for that we use a digital video camera. This will be our primary control mechanism.

The Photosynth, smile powered music.
Face is observed by the camera and digitised

Using a machine learning algorithm trained on millions of face photographs taken from many directions, the computer can examine the image for features it may recognise by comparing them against the photos in its memory. A recognisable nose, a mouth, an eye...

Facial landmarks are detected and tracked

Using these facial landmarks as references, the user's orientation, expressions and mood can be digitally captured and represented in 3D space which can quite accurately recreate the shape of the face with all its contours and character.


"Computer vision"

How to read a human being

Computers are very good at recognising patterns. If you keep showing a computer ducks, eventually it will realise what ducks have in common - a bill, wings, webbed feat, a distinct waddle and some quacks. With practice it becomes an expert it duck detection.

The same can be true for recognising human characteristics such as moods and emotions. Being happy presents many clues and indications when studied, most obvious is smiling, but joy also brings wide eyes, crows feet, and delight raises eyebrows. Frowns and furrowed brows indicate signs of melancholy and malaise.

Both happiness and sadness are easy to spot. Like ducks.

A well-trained machine can read you like a book...

Mood recognised

But can it play you like a musical instrument? πŸ¦†

If it looks like a duck, and quacks like a duck, we have at least to consider the possibility that we have a small aquatic bird of the family Anatidae on our hands

Douglas AdamsDirk Gently's Holistic Detective Agency


Technology :

Imagine a world where rather than having to learn a musical instrument, the musical instrument could learn how to play you...

By using motion capture and augmented reality, heads are converted into musical instruments where smiles are converted into musical notes, movements into melodies and facial expressions into audio filters and controls.

If there was a magical music spell, what musical skills would you wish for?

Mood has been recognised
Mood has been recognised

Any sufficiently advanced technology is indistinguishable from magic.

Arthur C Clarke

The Shape of Sound

Unique Sonic characteristics - The Sound of Shapes

Each sound has a distinct shape, similar to a fingerprint or snowflake. This affects how it is perceived by your ears and how the sound travels and bounces off things.

Sound travels at the same speeds as gases do, because that is how they travel, they are gas powered.

Instead, variation in sound comes from their shape, their repetition and their strength. Tiny fragments of sound when repeated hundreds of times a second can create a musical sound similar to what an instrument might make. We call these sound shapes timbres. Timbres can be made from any fast repeating sound including noises.

Different shapes affect the air as they travel as well as any physical matter encountered on the way, including vibrating the little hammers inside your ears, where these sonic bubbles tap the music like morse code directly into the brain to decode,

The frequency of the timbre, the rate that the noise rapidly repeats, dictates its unique sonic pattern and musical tone, which forms a note. Each musical note is determined by the amount of times that the pattern repeats in a fixed time period, often refered to as its vibration, wavelength or frequency.

Slow repeating sounds (low frequency) are perceived as deep or bassy, whist the faster they vibrate, the more high pitched they sound.

Play with the keyboard below to see how the shape varies between high notes and low notes..

Bouba : Lovely sounds

Round sounds such as the curvey Sine wave sound soft, familiar and comfortable. Beautiful forms that look smooth and predictable usually also sound nice.

The lack of detail in a shape makes them sound clean and pleasant.

Layers of waves create new shapes and waveforms, compounding into new sounds.

Often discordant and unpleasant sound is described as noise. Noise can be very powerful when used as the seed for synthesis. We can use it to drive anger into our sound.

Kiki : Angry sounds

Sharp, angular sounds, such as the Sawtooth wave, sound fierce and aggressive.

Unpredictable and eratic, they sound uncomfortable and unpleasant.

The more discordant, loud and random the sound, the more angry it appears to be.

Play around with the instrument selector below to see how each one looks and sounds.

Sound Shapes, also known as timbres
Song Controls

Each Sound Has a Shape

Sonic characteristics of Sound

As sound moves it leaves remnants of itself on things it touches, like sonic snail trails, vibrating and shedding frequencies and reverberations.

These collisions alter the acoustics and harmonics of the sound and add tone and character.

When many notes exist in the same space they join together and form a new sound. Their waves double up or cancel out, creating entirely new

Certain frequencies when played together, join up and form a chord, and if the notes are harmonic, and complement each other, can trigger strong emotional feelings in the person hearing them.

There are many different types of chords, but the most common ones in western music are the major and minor chords. Major chords sound happy, bright and cheerful, whereas minor chords can sound sad and somber yet compelling.

Emotional Sounds using Chords

There is much research into the topic of how sounds affect humans, but it is commonly thought that many of the emotional triggers caused by sound are rooted in our historic relationship with other people and animals. From the calming songs of the birds, or the purring of the pussycat, to the cries of a baby or the squeal of an animal in distress.

Many sounds have historical contexts that served an important role in our evolution as survival skills. We may never find out the true origins of these musical-emotional relationships, but they appear to be deeply entrenched and universal.

Evocative sounds include, the growl of a bear, the roar of a lion, bird song, the patter of rain and the clap of thunder. These all form part of the ecological audio landscape that has surrounded us for millenia, speaking to us long before we had a language.

The dulset tones of "mumese" are universal, a high pithed, softer sounding way to communicate with children and cute animals. This inate skill also appears to be universal. Smaller things make higher pitched sounds, so we adopt these tones in an effort to communicate with them.

Music is an unconscious exercise in metaphysics in which the mind is unaware that it is philosophizing.

Arthur SchopenhauerThe World as Will and Representation

Shapes & Patterns

Chords & Patterns

When a musical note is played, the air pulses in patterns which is heard in our ears in the form of vibrations.

All that music is, is repetition on many levels.

Repetition forms music through many orders of magnitude, from the microspcopic vibrating patterns that form timbres, that at certain vibrations sound like notes, to the repetition of notes that form melodies, all the way up to the macroscopic patterns of repetition where the percussion forms rhythms and grooves.

Our ears respond to repeating waves, and our brains yearn for patterns to discover and information to decode.

Repetition serves as the handprint for human intent.

Elizabeth Hellmuth Margulis

Certain combinations of sounds when played together are refered to as a chord.

Throughout time people have searched for chords to represent emotions for their melodies.

Below are some of the most famous chords that all have a mathematical connection. See if you can figure them out by observing the keyboard below during the next section.

Music is an unconscious exercise in mathematics in which the mind is unaware that it is calculating.

Gottfried Wilhelm Leibniz

Bookmark the Circular Synth
Emotional Circle of Fifths Synthesizer😌 Tone Frequency Timbre
Circle of Fifths
Return to the circular synthesizer
Frequency
Mood
Choose an Emotion
Return to the circular synthesizer

Research

Everyday People

Everybody is different and unique. Some people have incredible memories and can vividly remember past events, whilst some people forget where they put their house keys every day. Some people remain laser focussed while others' brains begin to womble off.

Everybody has their own talents and their limitations.

Everybody is differently abled.

The diversity in human abilities has supported humanity by approaching problems from different perspectives and finding creative new ways of solving them.

Diversity is a human strength, and differences are powerful.

If everybody did the same things in the same fashion, everything would be a sea of uneventful nothingness, devoid of character, charm and appeal, where nothing new or exciting ever happens.

Some people have a natural talent for music whilst others struggle to play a note.

We should celebrate our differences through the universal language of music and dance!

Could technology help us to create a world where everyone can jam together?...

What a wonderful world it would be!

One area that has seen much research is using neural networks and machine learning to attempt to determine intentions from emotions and motions in realime, in order to augment existing skills with new musical abilities and super-powers.

By piggybacking onto intrinsic movements, that come naturally, a performance can be both expressive and communicative, where the dance leads the music. This could be as simple as binding musical actions to hand movements, or as sophisticated as a collaborative multiplayer face controlled pop-music factory that learns how to play you!

Music is an unconscious exercise in communication in which the mind is unaware that it is being guided.

Zenon Olenski

#DMLab

#Drake Music Laboratory, research and development arm

A Celebration of diversity & Inclusion.

Over 14 years, working directly with researchers, scientists, technologists, disabled artists and even people with no musical abilities, #DMLab has been researching ways to grant musical superpowers through research and technology.

The DMLab workshop originally started in London over a decade ago and still runs today, but there is also now a Manchester branch. Bringing together people with technical knowledge and people with disabilities, and different abilities, to work together to research and develop new ways of interacting with technology in order to make music.

If you would like to participate in this research or attend lab or AMIC, get in touch!

#DMLab Events and Workshops details
London Venue
DMLab RichMix
Address
35-47 Bethnal Green Rd , Shoreditch , London , E1 6LA United Kingdom
Manchester Venue
School of Digital Arts SODA
Address
35-47 Bethnal Green Rd , Shoreditch , London , E1 6LA United Kingdom
Telephone
020 7739 5444
Email
Social Media

Accessible Music Instrument Collection, AMIC

  • A ground-breaking national collection, representing state-of-the-art of accessible instrument design.

  • A world-class resource for disabled people to explore technological possibilities for making music.

  • A focal point for the development of cutting-edge accessible instrument technology.

John Kelly playing some accessible musical instruments.
Rock n roll legend, John Kelly on stage playing some accessible musical instruments
Accessible Musical Instrument Collection

Access to the right musical instruments and technology can be powerful and transformative.

As well as establishing a collection of existing accessible tech, the AMIC will commission and support the creation of the next generation of accessible instruments. The Collection will be as varied as possible, from high-end commercial instruments through to DIY experimental prototypes and bespoke instruments.

We will continue to build the collection as new instruments come to light, working with makers, hackers and music tech companies to make sure it reflects the most up to date technology. AMIC will be housed in an accessible space at Drake Music’s HQ and will be available to visit. Disabled musicians will be able to visit the collection, test out different instruments using an accessible workstation and (hopefully) find one which suits them. Makers can come and be inspired or see what has already been built. Academics can call upon it for research and teachers can explore options for their students.

Learn more on the AMIC Website

Frequently Asked Questions, FAQ

Answers to common questions

What else can we expect

Apart from showcasing some exciting music technology from the AMIC, including the PhotoSYNTH and mi mu gloves, you might see some members of the London Philharmonic Orchestra, perhaps even a pop star or two! It always pays to expect the unexpected though, and there will be more surprises announced closer to the event.

What are Mi Mu gloves?

Mi Mu gloves are special handworn controllers that allow you to interface with music technology using your hands and arms. See their official website

Where can I learn more about the research?

There are a number of research papers written about this topic including :

Cool website! How did you make it?

This website was designed and created by designerzen of Awesomething social enterprise. Hand coded in HMTL, CSS and Javascript, using Web-Audio and the Web-MIDI APIs and Canvas for some of the visuals. The source code is available on github

Back to Top