Exploring Synesthesia Through LEGO and Python
Written on
For my inaugural piece in 2023, I aimed for something both unusual and relatable. I set myself the challenge of merging four primary subjects I often write about: technology, coding, accessibility, and LEGO. This proved to be quite a challenge, but I remembered an idea I had back in the spring of 2020—a concept I found intriguing but had never shared.
It's also time to confess that my previous article on the LEGO smart lock was inspired by this earlier project, with a significant portion of the code being reused. Reusability is a great asset! Now, let's not keep you in suspense—let's explore a topic that has intrigued me for almost twenty years.
In 2020, my goal was to showcase synesthesia in a simple yet playful manner. After a few hours of experimentation and coding, I succeeded.
How To Create A Smart Lock Using Python And EV3dev
This section provides high-level instructions for constructing a functional smart lock using LEGO’s EV3 brick along with straightforward Python code.
What is synesthesia?
A brief introduction to this perceptual phenomenon will enhance this narrative. My first encounter with synesthesia occurred while watching the NBC series Heroes. One character, Emma Coolidge—who is also deaf—experiences a type of heightened synesthesia while playing the cello (or possibly the double bass). Looking back, I believe this was when the importance of accessibility began to resonate with me. This scene remains one of the most impactful moments in television for me. I encourage you to watch it, along with another scene that illustrates this phenomenon.
> Synesthesia (American English) or synaesthesia (British English) is a perceptual phenomenon where stimulation of one sensory pathway leads to involuntary experiences in another. Individuals with a lifelong history of such experiences are referred to as synesthetes. — Wikipedia
An abundance of literature exists on this topic, so if you share my fascination, I highly recommend delving deeper after reading this article.
Prerequisites for the synesthesia demonstration
To create this demonstration, you will need a few essential components; however, it's significantly simpler than the smart lock project, making it more approachable if the latter felt daunting.
- An EV3 brick. This is crucial; it must be the version from set #31313 or the official educational edition set #45544. Only one smart brick is necessary for this prototype. You can purchase it separately, but availability may be limited.
- Ideally, a light/color sensor is required. This will detect various colors from your chosen source and enable us to translate them into sound.
- Basic Python knowledge. I may write a companion piece introducing Python basics soon, but for now, resources like Codecademy and the many links in this FreeCodeCamp article can help.
- A computer. This could be a simple device like a Raspberry Pi or something more premium like a Mac. A Windows machine will work perfectly as well. I used my 2019 Intel MacBook Air for this project.
- A microSD or microSDHC card (2 GB or larger, but not exceeding 32 GB due to the EV3 brick's limitations).
- Regarding software, you will need two things: VSCode and EV3dev. Additionally, install the ev3dev-browser plugin for seamless project uploads to the brick.
As you can see, the requirements are quite minimal, and the beauty of this project is that you don't have to construct anything out of LEGO bricks apart from the smart brick and the light/color sensor, making it straightforward.
The synesthesia “engine”
While it's common to associate specific colors with particular sounds, my demonstration takes a different approach. The project is simple and requires minimal coding effort. The synesthesia "engine" is essentially just a set of if-else statements, but as Apple has shown us, marketing is key—so if I label it an "engine," you can bet it sounds impressive!
from ev3dev2.sensor import INPUT_1 from ev3dev2.sensor.lego import ColorSensor from ev3dev2.sound import Sound
spkr = Sound() xylophoneSensor = ColorSensor(INPUT_1)
spkr.speak('Starting synesthesia engine.') timeRemaining = 100
while time > 0:
# xylophone sensor operates on input port 1
if xylophoneSensor.color_name == 'Blue':
spkr.play_file('/home/robot/synesthesia/xylophone/a1.wav')
time = time - 1
elif xylophoneSensor.color_name == 'Green':
spkr.play_file('/home/robot/synesthesia/xylophone/b1.wav')
time = time - 1
elif xylophoneSensor.color_name == 'Yellow':
spkr.play_file('/home/robot/synesthesia/xylophone/c1.wav')
time = time - 1
elif xylophoneSensor.color_name == 'Red':
spkr.play_file('/home/robot/synesthesia/xylophone/d1.wav')
time = time - 1
elif xylophoneSensor.color_name == 'Brown':
spkr.play_file('/home/robot/synesthesia/xylophone/e1.wav')
time = time - 1
elif xylophoneSensor.color_name == 'Black':
spkr.play_file('/home/robot/synesthesia/xylophone/f1.wav')
time = time - 1
elif xylophoneSensor.color_name == 'White':
spkr.play_file('/home/robot/synesthesia/xylophone/g1.wav')
time = time - 1
Upon examining the code, you’ll notice the operations are quite straightforward. As the program initiates, the smart brick announces that the engine has started, at which point it awaits color input. For each color detected by the sensor, a different xylophone note is played. The instrument can vary, provided you have the necessary .wav files. It’s worth mentioning that the sensor can also detect the absence of color, but I found that this added complexity, so I limited it to seven notes.
You can view a demonstration of this code on my Instagram or TikTok. I would appreciate it if you checked it out and followed me there as I share interesting LEGO and EV3 programming content not found in my articles.
In fact, you can significantly enhance this demo by adding one or two more light/color sensors. This allows for color combinations to produce full octaves and even polyphonic sounds. If you want to modulate the sound, consider using an ultrasonic or infrared sensor, or a touch sensor to create additional effects or switch between instruments. Ultimately, your imagination is the only limit.
Synesthesia is an incredibly engaging method to introduce someone to diversity and accessibility, showcasing just how differently we can experience the same world.
Attila Vago — Software Engineer improving the world one line of code at a time. A lifelong enthusiast, I write about coding, web accessibility, LEGO, and more. Enjoy craft beer! Read my Hello story here! Subscribe and/or become a member for more stories about LEGO, technology, coding, and accessibility! For those who don't read regularly, I also explore random topics and writing.