SENSORY SUBSTITUTION, CROSS-MODAL PROCESSING AND AUGMENTED REALITY

In Consciousness, Psyence, Psymbiosis by Nick Chan

Sensory Substitution generally refers to transforming the modalities of one sense into another, carrying over most of the primary functions and usefulness of the original sense. This finds its main utility in devices for people with sensory impairments, especially the deaf and blind.
Sensory Substitution Devices(SSDs) are devices that perform such functions. Examples would include devices that enable the blind to ‘see using sound’, the deaf to ‘hear using touch’ , the color-blind to perceive hues using tones.
SSDs are remarkable in that they provide a non-surgical non-invasive means of ‘sensory-prosthesis’ that has been proven to be on par with, sometimes even surpassing, other more costly alternatives.
In the past, the requirement for a separate laptop, software, battery pack, miniature camera and headphones meant a bulky and expensive device. But the fact that we carry smart-phones means that many SSDs are not potentially-available, but already in our pockets. The software, especially the better ones, are all free.
For the sensory-impaired, the blind and deaf especially, there is already an implicit need for anything that can extend their senses to allow for things as basic as crossing the street safely. Things such as cochlea implants that help the deaf can cost tens of thousands of dollars, putting them squarely in the hands of the rich. Bionic eyes are an even more specialized case, being far more expensive and experimental.

Traditional implants generally do not require any training to use, but are expensive. SSDs while cheap, require investment of effort and their efficacy usually sets in after around 3 to 4 months of sustained use/practice. Despite their growing pervasiveness and low cost, the use of SSDs faces major hurdles in terms of training and time. But it can also relate to culture. In cultures where the sensory-impaired are more marginalized and the support-structures less robust, a blind person for example, may have other more pressing needs like earning a living and thus not give SSDs much thought. But wouldn’t it be a waste if that person were to be able to tangibly benefit from a device that costs almost next to nothing?
To my knowledge, there is insufficient awareness of SSDs in Singapore. Unlike the Guide-Dogs Association for the Blind, there are no training specialists or advocacy groups for their use. Which is strange because a cursory search on the Internet would show that SSDs have far greater prominence in the Neuroscience and Psychology communities, as well as the Tech community, since they relate directly to augmented reality and wearable tech.
Training specialists for SSDs do not have to be blind themselves, they just need to understand the principles behind their use and be able to act as a facilitators to the blind user.

We often come across questions of how to monetize something that is low cost or free. Why not a startup then, that is focused on SSD training and use for the blind and deaf? This startup need not, and probably should not, only be focused on the sensory-impaired, though that should be the main vision. Leveraging on other novel uses of SSDs and the implications that come in the wake of research into them will result in engagement with different groups, be they artists, scientists or hackers. This in turn will raise awareness even further and enhance the original vision of helping the sensory-impaired.

We often see demonstrations of this-or-that tech at trade shows and conferences all around. SSDs are no different.

SSDs are often simple in design, but their underlying principles and possible implications run deeper than we likely expect.
SSDs are not just for the ‘impaired’. They are also great open-ended tools for anyone adventurous enough to figure out a use for them. SSDs and the research that accompanies them coalesce at the further edges of augmented reality. We’re far more used to hearing of Google-glass like augmentation, the seamless placement of informational layers co-tangent to the everyday fabric. SSDs work at the level of the perceptual interface, our senses, their relations and how the brain makes sense of them. Augmentation in this realm can mean that one hears color or sees sound, in a very literal sense. While SSDs are interfacing tools that get better like other tools, they are unique in that they have the potential to redefine what it means to interface at all.

…some users are exceeding the level of visual performance often achieved by invasive techniques to restore vision, such as stem cell implants and retinal prostheses. This is partly because, even after surgery, many formerly blind people have such impaired vision that they are unable to make out anything more than abstract images. But it is also down to the remarkable results produced by the soundscapes. Some long-term users claim they have images in their minds that are somewhat akin to normal sight as their brains become rewired to “see” without using their eyes.a team of psychologists and computer scientists at the University of Bath, researching The vOICe

As a musician myself, I do wonder what it might mean and value(and fun) it might add for my creative and perceptual process if I managed to induce artificial-synesthesia as a result of SSD use. It’s not as far out as we might expect, and really, isn’t it simply ‘hacking’, plain and simple? Just that we are doing it with our perceptions and senses. The area here lies not in the fusion of body and machine, but rather of software and brain process.

SSDs require the investment of time, work, patience, learning and self-questioning to render their use effective, because there is a language that one needs to both invent and learn at the same time. There are no shortcuts. But if one reaches that point of finding a use, the rewards are numerous, and while uses and outcomes are unique to individual contexts, the process of getting there is always one of active engagement, participation, learning and earning the learning. Simply because SSDs will not work otherwise, and appear as no more interesting than your pocket calculator.
It’s akin to the difference between building, say, your own Arduino based device that answers your own question/needs, vs picking a pre-assembled finalized one up from a catalog, while telling yourself that your aim is to learn soldering, putting together basic circuits and getting a feel of the process.

Can engaging the world with a sensory substitution device give rise to new aesthetic experiences? For Neil Harbisson, such was the case, which grew out of a need.


A SSD made for blind people, it works by scanning the visual-field with a regular camera and translating it to complex and/or simple ‘soundscapes’ conveyed through earphones.

Available completely free on the Android platform, there is also a free ‘Learning Edition’ for Windows. The website is full of information suitable for everyone from the blind, to the curious hobbyist, to the full-blown cognitive neuroscientist. Creator Peter Meijer has stated that it has always been free as the priority is to aid the blind and spread its use.
He also encourages its use by sighted people. Perhaps most interesting is that certain blind-since-birth users have not only been able to do functional things like recognize objects and improve their general quality of life and mobility, but to even ‘enjoy the scenery’ when taking walks, with this going as far as one user taking up Photography as a hobby.
I personally am able to be blindfolded and use this SSD and pick up a dark object placed anywhere in a white room of regular dimensions and stable lighting. which is trivial and mostly a process of rational deduction. What’s not trivial is that blind users are reporting ‘visual sensations’ such as depth, perspective, and other factors related to the modality of a sense they were born without. This goes against certain things we assume about the brain.

You can place a cube in a blind person’s hands and explain 3-dimensions and he can understand through touch. Length, Breadth, Width. Now think of telling the same person about the same cube as a drawing on paper(2D). There is hardly any bridge of ‘understanding’ to be built.

The vOICe’s website states that the “…psychophysical or psychological research community has not yet found conclusive answers with regards to the potential of this approach for the blind, and the relevant limitations in human auditory perception and learning abilities for comprehension and development of visual qualia (sensations) therefore remain largely unknown or anecdotal, although the training effort is expected to be significant while involving perceptual recalibration for accurate sensorimotor feedback.”

However, neuroscientist Dr. Amir Amedi from the The Hebrew University of Jerusalem successfully conducted experiments that have begun to indicate that the brain doesn’t really discern between information from the different senses. Rather, it uses whatever information it can get and constructs a relevant ‘picture’ with which the organism can use to interact with and navigate through the world.

I have used The vOICe for some of my artistic work, shown in a separate section.

Amazing fact: For a piece of software that has:
stereoscopic capabilities,
text to speech,
sonification of GUIs,
3D ‘threat detection and collsion modes,
a talking color identifier and much much more, it stands at a whopping size of 500KB.

http://brain.huji.ac.il/default.asp

Dr Amir Amedi has an interesting lab exploring not just SSDs, but multi-sensory integration on a broader scale.
He released his own SSD in the last few years. Also mainly for the blind, this one is available free from the Apple Store and it’s only for iOS. It takes a slightly different approach from The vOICe, EyeMusic conveys color information by using different musical instruments for each of the five colors: white, blue, red, green, yellow; Black is represented by silence. This one seems to take advantage of mnemonics, the melodic scale and the timbrality of common music instruments to correlate sound and vision.
EyeMusic is freely available for download with additional instructions at http://brain.huji.ac.il/EM.htm

“When the researchers mapped the brain activity of the participants, they found something astonishing. The generally accepted model of the brain contains regions devoted to each sense, such as the sight-centric visual cortex. Researchers had long believed that if those regions aren’t used for their intended sense, they are repurposed for other uses; for example, the visual cortex of someone blind from birth could be used to help boost her hearing. But Amedi and his colleagues found that the area of the visual cortex responsible for recognizing body shapes in sighted people—called the extrastriate body area—lit up with activity in the study participants when they were interpreting the human silhouettes.
Amedi says the traditional sensory-organized brain model can’t explain this activity; after all, the subjects only heard the information, and scientists believed that the body-recognizing area shouldn’t have fully developed without visual experiences during development. Neuroscientist Ella Striem-Amit of Harvard University, who co-authored the paper, thinks it’s time for a new model. “The brain, it turns out, is a task machine, not a sensory machine,” she says. “You get areas that process body shapes with whatever input you give them—the visual cortex doesn’t just process visual information.”
Ione Fine, a neuroscientist at the University of Washington, Seattle, who studies brain changes and did not work on the project, says Amedi’s work is the best evidence yet for functional constancy—the idea that areas of the brain do the same job even with different kinds of input. “The idea that the organization of blind people’s brains is a direct analog to the organization of sighted people’s brains is an extreme one—it has an elegance you rarely actually see in practice,” she says. “If this hypothesis is true, and this is strong evidence that it is, it means we have a deep insight into the brain.” In an alternative task-oriented brain model, parts of the brain responsible for similar tasks—such as speech, reading, and language—would be closely linked together.”

David Eagleman is a neuroscientist at Baylor College of Medicine. The mission of his lab reads,

The long range goal of our lab is to understand how the brain constructs perception, how different brains do so differently, and how this matters for society. To that end, our four main prongs involve time perception, sensory substitutionsynesthesia, and neurolaw.

His device is called the VEST, the Versatile Extra-Sensory Transducer  and unlike the other two, his is a tactile-to-audio SSD that enables the deaf to hear through vibrations they feel from a suit they wear, which has embedded microphones and actuators.

A direct demonstration occurs at 12:15 in this video, and it speaks volumes about the plasticity of our brains to adapt information for different purposes.

His Kickstarter video for the VEST describes the technology and rationale behind it.

Tools aid our work, but they often change our working process, Photoshop being a good example. Sometimes they redefine our output, which feeds back into the process and redefines the tool itself. This is especially significant for tools that are used for purposes outside their ‘intended’ function, a broom, or even boxing gloves being used as a paintbrush are good examples.

While no different from adopting a new tool for one’s purpose, the uniquely cross-modal nature of SSDs means that all sorts of unexpected and novel surprises arise from their use. Reversal of cause-and-effect, and uncommon correlations between things once thought unrelated, are the most immediately noticeable. Putting the ‘cart before the horse’ is an effect commonly found in the realm of SSDs, and such effects are already exploited by many artists to varying depths of amusement and profundity. Perhaps the bigger challenge lies in answering questions such as, ‘Is it possible to do away with the cart, the horse, or throw the baby out with the bathwater altogether?’

Color-blind artist Neil Harbisson taking a sonic-portrait of Apple co-founder Steve Wozniak.

This is a sono-chromatic portrait of the song ‘Baby’ by Justin Beiber.


Here are two experiments I conducted to explore their use. The time taken for each from execution to completion of final product was no more than 30 minutes in both cases.

Both were made using The vOICe SSD as the primary tool, with the usual audio/visual software playing a supporting role. It involves mostly interlacing and edge detection visually. Not much was done to the sound. It sounds typically less ‘musical’, but there’s a strange timbral correlation between the roughness and graininess of the sounds and the image, with depth expressing itself in a way different from the more linear dimensionality of a Kinect point-cloud.

This next example was done in a shorter time than the first and involved mostly bi-directional horizontal scanning visually. I spent more time putting together the sounds. The audio/visual process was also separated at parts here, while in the first example everything was one seamless process with hardly any distinction made between audio and visual. And while this one probably ‘makes more sense’, I preferred and derived a lot more out of the first example.

 

In early 2014, I was commissioned by FuturePerfect Gallery to create a piece for TRANSMISSION, an art exhibition in Bangkok organized by the Jim Thompson Foundation. My only brief was that my piece had to be sonic in nature and aligned to the show’s theme, which revolved around the cultural identity of Thailand.

As every other artist was making a physical artifact to express this, my piece titled ‘Omnis’, used The vOICe SSD to sonify their pieces and represent them in sound. Infra-red sensors were placed at key-points in the gallery, such that human traffic would trigger a shifting soundscape of textures at different parts of the gallery, the sonic textures representative of works located at other parts of the gallery. While it’s hard to capture such installations on record, here is a video of the sonification process of a sculpture, along with some of the works I turned into sound.


The shifting polygons are images of pieces of the sculpture being projected back onto the sculpture to create variations in light. With The vOICe being put in ‘threat detection mode’, it recognizes this as movement, which resulted in variation in the soundscape, despite the fact that both the camera and the sculpture were stationary.
IMG_3633 IMG_3571 IMG_3609
IMG_3540 IMG_3567 IMG_3568 IMG_3584 IMG_3615
IMG_3577

 

About the Author
Nick Chan

Nick Chan

Facebook Twitter

Nick divides time between music and sound-stuff, in both work and leisure capacity. As a budding game designer, he has not released anything.

Share This Post