Abstract

This study concerns the human experience and interrelation between synesthesia, memory and art. The aim is to outline an extended notion of synesthesia. The approach is from an interdisciplinary perspective, embedded in a context, where the senses are considered as interrelated aspects of human bodily engagement with the world. The questions I would like to focus on are: How are synesthetes 'people of the future'? What's the 'address' of the senses? What's the role of images vs. the construction of meaning, and how images are 'nomadic' in our visual memory? I want to draw some parallels between synesthesia, memory and art, towards a better understanding of the brain's natural synesthetic abilities. I will end this lecture with some remarks on five of my recent paintings.


1. Synesthesia and the future of the senses

"A strong imagination creates its own reality." —Michel de Montaigne (1)

My thesis is that all human art has a synesthetic origin, specifying the interconnectedness of existence. Synesthesia has a long history. Synesthesia is also know as 'sense transfer'. 'Cross-modal plasticity' is one of the most natural elements for all human artistic forms of creation. Synesthesia and art coexist (think with the senses, feel with the mind). Synesthesia is what makes art possible. Synesthesia and cross-modal plasticity is fundamental to human experience. Nothing in this world exists, isolated from everything else. In life everything overlaps and merges (a great undivided multiplicity is always at work). Through the interaction of our senses we are constantly engaged in a dialogue with the world. Perception is a synesthetical gestalt.

The extended notion of synesthesia consists of three connected fields:

1. Natural synesthesia - the multi-sensory experiences of synesthetes

2. Synesthesia in art - the synesthetic fields of experience revealed by the arts

3. Tele-synesthesia - the synesthetic dimensions of multimedia and telematics (2)

What connects synesthesia, memory and art? What are the corresponding concepts? What are the futuristic implications of synesthesia?

Experiencing synesthesia, memory and art, happens in an idiosyncratic way. (3) We all have our own reality. The unique reality in every individual is based on how interactions between perceptions, desires, and expectations are formed in our neural networks. To deal with the mind-brain interaction one needs to consider the physical processes in human brains. The source of the brain’s great plasticity is called synaptic transmission. This experience-dependent plasticity, or neuroplasticity, is the ability of the brain to reorganise neural pathways based on new experiences. Neuro-imagining shows that different neurons process information at different rates: the perception of colour occurs before that of form, which in turn occurs before that of motion. Neuro-realism is based on the fact that the brain (just like the body) can't lie. The mind is what the brain does.


1.2. Neuro-privacy and brain scanning

The following are the various types of brain scan systems being used today which have accelerated our understanding of the brain and the localisation of the sense and senses.

Brain scanning technology:

EEG - electroencephalograpy - the electrical patterns created by the rhythmic oscillations of neurons.

CAT - computerized axial tomography - to generate cross-sections or 3-dimensional images of the brain.

PET - positron emission tomography - it identifies the brain areas that are working hardest.

fMRI - (functional) magnetic resonance imaging - can produce very clear and detailed pictures of brain structures.

MEG - magnetoncephalography - it picks up signals from neuronal oscillation.

With these advanced techniques it is possible to monitor simultaneously the activity of many neurons within complex neural networks during discrete behaviors. By scanning and neuro-imaging, we can see into the workings of a human brain:

Brain activity in a fake orgasm

A simple brain scan can spot whether a woman is faking an orgasm or not. Researchers at the University of Groningen in the Netherlands have used scans to show that different areas of the brain are stimulated during an orgasm but are not activated when a woman fakes it. Neuroscientist Dr. Gert Holstege found that when a woman is faking a part of the brain under conscious control lights up, while real orgasms occur subconsciously.

Cognitive neuroscience has come to be viewed as the flagship of cognitive sciences and is transforming our understanding of the nature of the mind. As a result of the new scanning techniques, new academic disciplines and research fields on 'mapping the mind' and mind-brain science are taking form:

Neuroesthetics founded by Prof. Semir Zeki, University College London. Neuroaesthetics is attemping to understand how art arouses aesthetic experience by starting from the basis of emotional and sensory experience. Going deeper into molecular neurobiology, neuro-aesthetics aims to identify the molecules in the brain that lie behind its emotional contemplation of art.

Neuroethics is the ethics of neuroscience. Neuroethics encompasses a wide array of ethical issues emerging from different branches of clinical neuroscience (neurology, psychiatry, psychopharmacology) and basic neuroscience (cognitive neuroscience, affective neuroscience).

Neuroarthistory (world art studies) founded by Prof. John Onians, University of East Anglia, Norwich, to investigate what happens inside artists' brains.

Neuroimaging includes the use of various techniques to either directly or indirectly image the structure, function/pharmacology of the brain. It is a relatively new discipline within medicine and neuroscience.

Neuro-linguistic programming (NLP) is based on the idea that with our senses we are only able to perceive a small part of the world. Our view of the world is filtered by our experience, beliefs, values and assumptions. We act and feel based on our perception of the world rather than the real world.

Neurophilosophy explores the relevance of neuroscientific data to arguments in philosophy of mind (it is one of the fastest growing subfields in contemporary philosophy).

Because a brain scan can tell when your paying attention to something, the scientific research on brain chemistry and neuro-anatomy has opened up new ethical responsibilities in the field of neuroscience vs. our neuro-privacy. An academic discipline of 'neurolaw' and 'neurolawyers' is flourishing. The critical questions are: "How are individual identities effected by brain scanning technology?", and "how will we integrate new neurotechnologies into our lives and social practices?". The following key terms indicate some major topics of a neurosociety: neural coding and information processing —embodiment of memory —remapping of reality —personal coding —privacy of experience.

Ludwig Wittgenstein distinguishes two senses of the word private as it is normally used: privacy of knowledge and privacy of possession. Something is private to me in the first sense if only I can know it; it is private to me in the second sense if only I can have it.


1.3. The Synesthetic brain: to be a synesthete

Synesthesia is a neurological phenomenon, where sensory modes blend. A synesthete's mind has the remarkable plasticity to 'taste' shapes, 'hear' colours, or 'feel' sounds. Synesthesia is genetic, synesthetes are born with it. Research suggests that it is relatively rare: it is thought that around one person in every 100 or 200 has a form of synesthesia. There are at least 50 different types of synesthesia, involving various combinations of senses both as the triggering stimulus and the secondary response (in extreme rare cases, all five senses are blended together). A returning question is: "How it must be to have synesthesia —to be a synesthete?". Synesthetes are living in a reality that others cannot experience. Most of the synesthetes I know are more "plugged in" to their environment than other people. It can be said that the brain of a synesthete is even more unique than that of a non-synesthete.

Dr. Simon Baron-Cohen, synesthesia researcher at the University of Cambridge says: "If you ask synesthetes if they wish to be rid of it, they almost always say no. For them, it feels like that's what normal experience is like. To have that taken away would make them feel like they were being deprived of one sense."

Uniqueness makes each of us very special, given that no one was or will ever be like us. Many writers, visual artists and musicians are synesthetes. At birth we have a raw but integrated (whole) brain to a level where synesthesia is common. We are all synesthetic to some degree, but to be a natural-born synestheste, and to experience the richness of synesthesia is indeed an extraordinary gift. The passive-active structure of world disclosure is for each synesthete a personal, complex, detailed and precise experience. In a sense, synesthetes are the people of the future, 'Homo Futuris', because with the futuristic, telematic extension of the human senses, everything will become more and more synesthetic —the senses will become 'interactive tele-senses'. (4)

For example, the future libraries will become brain banks instead of book banks, synesthetic archives: an interactive multimedia integration of the visual, the kinetic, the haptic, the sonic, and the telematic. To demonstrate the state-of-the-art of the "Multi-Touch Interaction Research" hereby a short film by Jefferson Y. Han, Consultant Department of Computer Science, Courant Institute of Mathematical Sciences, New York University.

"Multi-Touch Interaction Research"
Demo Reel Jan-2006, 03m 31s (5)

These are bi-manual, multi-point, and multi-user interactions on a graphical display surface. Multi-touch sensing enables a user to interact with a system with more than one finger at a time, as in chording and bi-manual operations. Such sensing devices are inherently also able to accommodate multiple users simultaneously, which is especially useful for larger interaction scenarios such as interactive walls and tabletops.


2. Memory, an interactive storage bank of virtual possessions

"Art can be viewed as parallel to memory. Many of the same terms are used to describe the two. For example: art and memory both employ and integrate the senses, both are representations, and both refer to a sense of timelessness. Art can evoke memory and vice versa. ... There are many aspects of memory that, like art, are not quantifiable." —Andrea Polli (6)

2.1. Seeing, hearing, touching

"Through vibration comes motion. Through motion comes colour. Through colour comes tone." —Pythagoras

To understand the wonderful complexity of our human body we have to look at the prehistory of the mind, and our connection with nature in a neurobiological sense. First of all: the human brain is wired for survival. Humans are the result of a stage of evolutionary transformations, which has lasted over two million years. Since our prehistoric ancestors developed a bigger brain, we have slowly learned how to extend our senses and how to create culture through innovations. When men created the first images, he made an important evolutionary step forward. Think about it as the invention of external memories: to store memories in the outside world instead of in their own brain (cave paintings, figurines, calendars, etc). It was also the creation of a synesthetical and metaphorical connection between time and space. Mental representations where transformed into physical ones. These representations 'detached' from external reality where fundamental for creating the kind of mind that we now have. Today, we use computers with multi-dimensional associative search engine, having an external memory.

Our brains form a million new connections for every second of our lives, to make up our perceptions of the world. The external world is a physical and objective reality. The internal world is a mental and subjective reality. Key questions are: "How does our inner world influence our outer world? where does the body ends, and the world begins?" We perceive with our whole being —we see the world with our entire body sensorium. Perception is the initial point where mind and environment interact. As in seeing, we tend to hear what we expect, to hear what our memories and conditions tell us to hear. Consider the synchronisation between eyes and ear, the virtues of seeing and hearing complement each other. While the eye sees only 180 degrees, the ear hears a panoramic 360. For example: sight isolates, sound incorporates. In other words, vision is a dissecting sense; it comes from one direction at the time. Sound is a unifying sense; when we hear, we gather sound from every direction at the time.

Seeing and hearing are remote senses. They tell us about distant parts of our environment by receiving vibes and waves. Seeing is also touching at distance, for example, the haptic look is a form of tactile vision; here the eye is used as an organ of touch. Haptic images are invitations to come as close as possible to the image. The word haptic is derived from the Greek term "hapthai", meaning touch —the earliest sense to develop in the foetus. Our need to touch is passionate. Touch gives an immediate communication with internal, or external bodies. Our skin is what stands between us and the world. Skin is a language; skin records the passage of layers of reality.

Like an onion with many layers

In our knowledge, in how we explore our world, and universe, we can go wider and deeper every day. It's like peeling away layers of an onion, you have to go through one layer at a time.

2.2. Tasting, smelling, remembering

"Nothing fixes a thing so intensely in the memory as the wish to forget it." —Michel de Montaigne

Taste is a contact sense. Smell is a distance sense. Tongue and nose, they both involve a physical/chemical, intimate contact. Taste comes from the Latin word taxare, meaning "to touch" or "to feel". Of all our senses, smell is the most primal, evocative and immediate; it acts upon our body within fractions of a second, before we are aware of it. Smell can affect our mood, our emotions and the way we behave. Take cold sweat for example, which can create the smell of fear. We are also more influenced by the erotic qualities of smell than we realise. Smell has a direct access to our natural alarm system, the amygdala. The amygdala is a component of the limbic system. In evolutionary terms, the limbic system is amongst the oldest parts of the brain. Up to seventy-five percent of what we perceive as taste actually comes from our sense of smell. Taste and smell are closely coupled to memory.

The main questions are:

How does the brain transform experience into memory? and consequently how does the brain encode a memory?

Memory is one of the mind's most crucial functions

Memory provides a bridge between past and present

Memory is a structure that parallels visual fragmentation

We know that synesthesia underscores all memory processes. We also know that a richer synesthetic capacity often means a stronger, richer memory. A strong imagination and memory always employs synesthetic imagery and a metaphoric reading of life. Imagination works on memory like a flame on paper, making it glow. To some degree, the unifying function of consciousness depends on our ability to remember. As a dynamic system shaped by selection and suppression, memory provides the continuity with the immediate preceding conscious experience. New memories are powerfully influenced by old knowledge. The past imposes itself on the future. The dominant metaphor for memory retrieval is association. Remembering happens by sensory correspondences, similarities and synesthetics. Today, the term 'synesthetics' means also a new approach towards the development of the whole person with emphasis on perceptual awareness, psychological growth, self-actualization, nonverbal learning, and creative behaviour.

Researchers argue that "sleep orchestrates the strengthening of memories", but also that sleep may be involved in "erasing memories from the immediate and distant past". In both cases it seems to be an ordering, a personal coding of one's stream of experiences. When a lived experience has been connected with a strong emotion, the chance is higher that it will be fused into our synesthetic memory. Flashbulb memory (the recollection of lifetime episodes via important events) is one example of this.

Andy Warhol said, "Of the five senses, smell has the closest thing to the full power of the past. Smell really is transporting. Seeing, hearing, touching, tasting are just not as powerful if you want your whole being to go back for a second to something". (7)

Through the power of smell, the past imposes itself on the present. This type of memory is called an episodic memory (the recollection of events). It evokes a specific experience that occurs instantly, for example, when being exposed to odour. Another type of memory associated with smell is semantic memory (the memory of facts and concepts). Semantic memory is connected with conscious thought, for example, being asked to think of what coffee or chocolate smells like. From personal memory to collective memory, from short- or long-term memory, remembering is an active and selective process of the (re)construction of the past. Memory is the capacity to bring elements of an experience from one moment in time to another. It is the unique property of life forms.


3. The synesthetic nature of painting

"Art makes us more aware of the synesthetic aspects of experience." —Dr. Hugo Heyrman

Art and its visual codes are human strategies of seeing and tactics of negotiating meaning. Visual thinking depends on the ability to decode and understand signification. The following five approaches are a personal source of ideas and remarks:

"Visual perception is always saturated with experiences from other senses. My paintings are about the physical and tactile dimensions of synesthesia. I want to discover new visual configurations in what I see and hear, feel and smell, taste and touch."

"The pictorial transposition of ways of seeing is the main theme of my work. Painting cannot take the place of reality or refer to it since there is no original reality. The paintings are orientation points, reflections based on synesthetic experiences, self-made realities; they are images of an image of an image, of an image."

"Art can touch us deeply when it says something specific about the time and space we live in. With an exploratory approach to the blank canvas, I want to turn matter, surface, texture and colour into imagination. When I paint, I enter the mental dimensions of 'image space', and/or 'memory space'. A painting becomes interactive when it looks back at you, at this moment the image becomes cinematic and activates our own individual films."

"I paint the existential tension between ideas and images; an appeal to several senses at once. I bring the visual and the conceptual, synesthetical closer together. I paint how I want to remember what I saw. These 'meta images' are external memories, representations of mental image experiences."

"It is my intention to give the term 'reality' a new meaning by creating synesthetic metaphors. The main elements of these mind maps are: tactility, physiology, and the process of image formation."

Art exist as experience. The viewer remakes the meaning of the artist. Perception is the cognitive act of projecting or creating meaning and pretending that it belongs to what is perceived. Meaning is an activity. The creation of meaning is above all embedded in human relationships and always culturally bound. In a media-dominated zap culture, the slow attention of a painting can act as a contemplative counterforce.

4. Five paintings from the "City Life & Body Language" series (8)

"We interpret one sense by another." —William Hazlitt

3. Woman from Brazzaville
	Statement "The painting shows the face of a young African woman. I was intrigued by the surface of her face. She frontally stares towards us, directly returning the viewer's gaze. She is using the viewer as a mirror. It's all about eye contact. A face can be a weapon, a mask or a shield. Proudly she confirms the evolutionary "Out of Africa" hypothesis. On her face we see the history of the sun. Between her dark eyes a background-colour of red copper is glowing through the layers of paint. Beyond her doll-like beauty and her red glossy lips, an individual character shines through."
Dr. Hugo Heyrman, Woman from Brazzaville, 2006, acrylic paint on canvas (50 x 50 cm.)

4. Enigma
	Statement "Here I painted an image of an anonymous beggar. He is sitting on the street curb, just on the rupture between things. He is pausing between two begging sessions, in a foetal posture. His body seems to be abandoned by his mind. He is a man of sorrow, living on the margins, lost in the labyrinth of life, disconnected from the world. The other is also the unknown, the enigma."
Dr. Hugo Heyrman, Enigma, 2006, acrylic paint on paper (21 x 29,7 cm.)