cellF

About




spacer 1

CellF is the world’s first neural synthesiser. Its “brain” is made of biological neural networks that grow in a Petri dish and controls in real time it’s “body” that is made of an array of analogue modular synthesizers that work in synergy with it and play with human musicians. It is a completely autonomous instrument that consists of a neural network that is bio-engineered from my own cells that control a custom-built synthesizer. There is no programming or computers involved, only biological matter and analogue circuits; a ‘wet-analogue’ instrument.

There is a surprising similarity in the way neural networks and analogue modular synthesizers function, in that for both, voltages are passed through components to produce data or sound. The neural interface we developed juxtaposes these two networks and in a sense creates a continuum that creates one unified network. With CellF, the musician and musical instrument become one entity to create a cybernetic musician, a rock star in a petri dish.

Premiering on October fourth, 2015, in Perth, Western Australia, CellF performed a live set with Tokyo based Australian musician Darren Moore. Sound from Darren’s drums was fed as electrical stimulations into CellF’s neural network which then responded through the modular synthesizers, its ‘body’, to create an improvised post-human sound piece. During the performance there was a clear sense of communication and responsiveness between the 2 musicians.  CellF represents a radical new way to think about what a musical instrument can be and how music can be made.

cellF was inspired by an ultimately narcissistic desire to re-embody myself. However, when thinking about what kind of body I would give myself, I chose not to follow the idea of working within a humanist paradigm. I decided to portray one of my adolescent dreams – to be a rock star. The long-standing passion for music, combined with sixteen years of research exploring artistic embodiments of bio-engineered ‘brains’, laid the foundation for creating this cybernetic self-portrait.

cellF is the result of a collaborative effort involving a creative team consisting of designer and new media artist Nathan Thompson,  electrical engineer and synthesiser builder Dr. Andrew Fitch,  musician Dr. Darren Moore, Stem cell scientist Dr. Michael Edel, Neuro-scientist Dr. Stuart Hodgetts, and Neuro-Engineer Dr. Douglas Bakkum. A collaborative project where each member played an important role in shaping the final outcome

The development of cellF posed enormous technological challenges. Establishing unique publishable biological protocols for differentiation and electrophysiology, developing a custom made all analogue neural interface that allows interaction with the neurons through simulations, building the biological lab that is embedded into the sculptural object and contains DIY high precision tissue culture incubator and a DIY certified class 2 laminar flow biological safety cabinet to work with human genetic modified material and more…

In 2012 I was awarded a Creative Australia Fellowship from the Australia Council of the Arts that resulted in four years of research towards cellF. I had a biopsy taken from my arm and cultivated my skin cells. Using Induced Pluripotent Stem cell technology, I transformed my skin cells into stem cells. When these began to differentiate they were pushed down the neuronal lineage until they became neural stem cells, which were then fully differentiated into neural networks over a Multi-Electrode Array dish to become – “my external brain”.

These neural networks contained approximately 100,000 cells. Human brains contain approximately 100 billion neurons, interconnected via trillions of synapses. The ‘brain’ used to control cellF is essentially a symbolic one to entice the viewer to consider future possibilities that these technologies can present.  But these neural networks do produce a tremendous amount of data, respond to stimuli, exhibit properties of plasticity and are subject to a lifespan.

The MEA dishes that host my external brain consist of a grid of 8×8 electrodes. These electrodes can record the electric signals that the neurons produce and at the same time send stimulations to the neurons – essentially a read-and-write interface to the “brain”.

Human musicians are invited to play with cellF in special one-off shows. The human-made music is fed to the neurons as stimulation, and the neurons respond by controlling the analogue synthesizers, and together they perform live, reflexive and improvised sound pieces or “jam sessions” that are not entirely human.

cellF sonifies the activity of the neurons in an engaging way whilst maintaining the integrity of the signals that the neural network produced. It reflects the complex nature and the spatialized aspects of the neural activity. During the performance the sound is spatialized in the space to 16 speakers. The spatialization is controlled by the neural network and reflects the spatial pockets of activity within the petri dish. Walking around the performance space offers the sensation of walking through my external brain in real time.

We plan to collaborate with other musicians and various musical ensembles and we wonder how different musical styles might influence cellF ‘s functional plasticity or ability to play. In the human brain music is known to entrain neural activity, and early music training in children alters brain structure and function. As a therapy in adults, music also enhances activity in brain circuitries after stroke or in neurodegenerative diseases such as Parkinson’s. In the long term, cellF provides a unique opportunity to understand how coherence and plasticity in neural circuits can be induced by rhythmic (and perhaps frequency) dependent inputs, with potential translational benefits.

CellF addresses my ‘interest in problematising new bio-technologies and contextualizing them within an artistic framework’. It started with a new materialist question underpinned by the belief that artistic practice can act as a vector for thought: What is the potential for artworks using biological and robotic technologies to evoke responses in regards to shifting perceptions surrounding understandings of “life” and the materiality of the human body?

Download cellF’s PDF – a summery of cellF

15-11-03-Cellf-Illustration-web

Performances

@ 'NeoLife', Perth

@ 'The Patient', Sydney

@ MONA (MOFO Festival), Hobart

@ House of World Culture, Berlin

@ ARS Electronica 2017

@ Gallery Kapelica, Slovenia

@ Todays Art festival

The Process

Harvesting fibroblasts

The starting point of this project was obtaining fibroblasts from my body so that I could grow them and then reprogram them to become stem cells. I had a biopsy taken from my arm and then took the flesh back to Symbiotica’s labs in the University of Western Australia (UWA). I collaborated with Dr. Mark Fear and Samantha Valvis from the Burn Injury Research Unit of the faculty of Medicine and together we processed the flesh in the tissue culture lab and grew my skin cells (Fibroblasts) in tissue culture flasks. I ended up with 30 million cells, divided into 30 vials, which we then cryogenically froze. I sent half of them to be reprogrammed in the Pluripotency Laboratory at the University of Barcelona

 

Hospital waiting room - Waiting for the Biopsy 

biopsy taken from my arm

biopsy taken from my arm

Biopsy examined in the car park 

processing the flesh in the tissue culture lab and to grow my skin cells

processing the flesh in the tissue culture lab and to grow my skin cells

processing the flesh in the tissue culture lab and to grow my skin cells

processing the flesh in the tissue culture lab and to grow my skin cells

My Fibroblasts in tissue culture

Time Lapse of the Fibroblasts growing

spacer 1

Cellular Reprogramming

I shipped 10 vials of my fibroblasts (approximately 10,000,000 cells) to the Pluripotency Laboratory, University of Barcelona. Due to the negligence of the shipping company, 10 million of my precious cells, bits and pieces of me, were destroyed. I shipped another 10 frozen vials and they arrived safely to the lab in Barcelona, where I secured an artist residency. I collaborated with Dr Mike Edel and together we managed to reprogram the cells successfully to stem cells using iPS technologies.

The iPSc technology was pioneered by Professor Shinya Yamanaka who showed that the introduction of four specific genes could convert adult cells into pluripotent stem cells. In layman’s terms, the iPSc method transforms adult specialised cells into a form that is equivalent to stem cells, which are capable of becoming any other type of cell in the body (skin, liver, muscle, neuron, etc.). The process involves re-programming their ‘software’ (genome), and coaxing them back into their embryonic state. In Barcelona we reprogrammed my fibroblasts to become pluripotent stem cells.

After I left Barcelona Mike Edel continued the work, “pushed” the stem cells down the neuronal lineage and further differentiated the pluripotent stem cells into neural stem cell (NSCs). At this point he froze them down and shipped them back to UWA.

 

Barcelona University

Dr Mike Edel

Working in the Lab

Working in the Lab

Working in the Lab

my pluripotent stem cells

my pluripotent stem cells

my pluripotent stem cells

my pluripotent stem cells

my pluripotent stem cells

My pluripotent stem cells

My pluripotent stem cells

My pluripotent stem cells

My pluripotent stem cells

Characterization of the stem cells 

Differentiated Neurons 

Differentiated Neurons 

Frozen Vials with Neural Stem Cells sent back to UWA

spacer 1

Neurons on MEA

The next phase of the project focused on establishing protocols to successfully and efficiently differentiate my Neural Stem Cells to neurons on a Multi Electrode Array (MEA) dishes and induce their electro-physiological activity.

the MEA dishes that host my “external brain” (neural networks) consist of a grid of electrodes that can record the electric signals that the neurons produce and at the same time send stimulation to the neurons – essentially a read-and-write interface to the neural network.

This phase posed enormous challenges as there are not a lot of labs in the world that have established protocols that I could follow. I collaborated with Dr. Stuart Hodgetts (Head of the Spinal Cord Repair Laboratory in UWA) and Dr. Cornelia M Hooper and together we established new protocols to achieve our goals. I can now successful differentiate my Neural Stem Cells on the MEA and around day 8-10 (from the initiation of differentiation) the neurons produce action potentials and the networks produces data. These protocols are still new and i am now trying to optimize them so that I could predict and control the process while being away from the lab – in the field conditions, a gallery or museum.

 

My Neurons differentiating at day 8

My Neurons differentiating on an MEA

My Neurons differentiating on an MEA

My Neurons differentiating on an MEA

My Neurons differentiating on an MEA

My Neural Stem Cells differentiating - day 3 

My Neurons differentiating on an MEA

My Neurons differentiating on an MEA

My Neurons differentiating on an MEA

Raw neural data in real time - neurons on electrode 41 quite active

Action potentials accumilating

My iPS derived Neurons

My iPS derived Neurons

My iPS derived Neurons

My iPS derived Neurons

 

MEA dish

MEA dish

MEA dishes with bits of me in the incubator

Neuronal data

Neuronal data

spacer 1

Neural Network – Analogue Synth Interface

One important component is the electrophysiology system that consists of amplifiers that are connected to the Multi Electrode Array dishes that hosts the neural network. These dishes consist of a grid of 8×8 electrodes that can record the electric signals that the neurons produce and at the same time send stimulations to the neurons – essentially a “read” and “write” interface to my external “brain”. The neural networks grow on top of the electrodes so they are in contact with them. The signals (in micro volts) are then transferred to the amplifiers and from there we direct them to the analogue synth interface. I am collaborating with Dr. Stuart Hodgetts, Dr Douglas Bakkum (neuro-engineer) and Dr Andrew Fitch to optimize the system and make sense out of the vast data sets that we receive from the cultures. We also spent a lot of time exploring how to convert the external sounds (music played by humans) collected in the gallery into meaningful stimulations to the neurons so that they can react and control the synths with integrity.

The analogue synthesizers were custom designed and made especially for this project by Andrew Fitch. Andrew is an electric engineer, an analogue synthesizer builder and he owns the analogue  modular synthesizers brand nonlinearcircuits. It was he that suggested to keep the whole project analogue. With cellF there is no programming, no computers, no zeros or ones; just biological matter and analogue circuits – a “wet-alogue instrument

Andrew also developed the neural interface hardware. Its the black box that you can see in the image below. This interface amplifies and control the output signals from the neurons and also the inputs (stimulations) to the neurons. We allowed 2 types of outputs. The actions potentials of the neurons (the filtered data) and the neural noise (the raw data). We allowed 16 stimulation inputs to the neurons. This way we could spatialize the input we send to the neurons according to various music/sound events.

“The FriGate module” allowed us to control and manage the stimulations that are sent to the neurons vie the electrodes of the MEA. This module was developed by Nathan Thompson and it was added to the Synths panels. Up to 8 microphones could be connected to this module. These microphones pick up the sound produced by the human musicians, analysed (via hardware) and sent to the neurons as stimulations if they passed a set threshold. This way the human musicians could decide when so they send cues to the neurons via stimulations similar to improve musicians that communicate between themselves while playing via visual or sonic cues.

The In-built “Matrix Mixer” allowed us to spatialize the sound to up to 16 speakers. This spatialization reflects the respective pockets of neural activity within the dish. Therefor, during a performance, the audience is invited to walk around the space as if they walk inside my external brain.

synth - devon2

I also worked closely with Dr. Darren Moore (musician and music technologist) on the sound aesthetics, sound-mapping schemes and the correlation to the neural activity, investigating the dynamics of the special organization and ever changing temporal behavior of the neural networks using scientific as well as artistic strategies such as sound.

 

Neural Network-MEA-Synth Interface

Neural Network-MEA-Synth Interface

MEA dish in the amplifier

Neural Network-MEA-Synth Interface

Andrew telling us how it is...

Neural Network-MEA-Synth Interface

My external brain is working !!!

MEA dish in the amplifier

Raw neural data in real time - neurons on electrode 41 quite active

Action potentials accumulating

Darren doing his magic

Andrew doing his magic

 spacer 1

Sculptural Object and the Bio-lab

Nathan Thompson designed and constructed the Sculptural object that includes the Bio-lab as well as the electronics. CellF has been designed to be a sound-producing ‘body’ comprised of the neural networks and an array of analogue modular synthesizers. The aesthetics of the synthesizer, which are similar to that of an electrophysiological laboratory, fits my vision perfectly.

Both in sound production process and neuronal cellular investigation, the amplification, alignment of signals in order to increase their facility is arguably one of the most important element in cellF. With this as a fertile bed of inspiration, the starting point for the design were the noise machines of the futurist, Russolo, in the early 1900’s.  The design also took shape around the amplification chamber that blooms outwards from an endless internal loop. Informed by natural forms such as a floral stamen inversely curving, or a wasp’s self-inflicting sting, we decided on a design that folds back and engulfs itself, shielded from intrusive irregular states while projecting its internal Lukasiewicz-like logic.

Aside from the complex brain – modular synthesizer interface that is at the heart of this project, other aspects have been equally as important for the success of the work as a whole. For example, the environment where the neurons are maintained needs to be controlled within tight tolerance. A constant 37 degrees C at 5% CO2 is an optimum habitat to ensure the culture remains active and healthy. Although we looked for an ‘off the shelf’ incubator, none were of satisfactory size, so we decided to build our own with the help of Andrew Pelling from Pelling Labs.

A Class 2 sterile hood was also an essential element of cellF, both for worker safety and culture integrity. Sterile hoods offer an environment where cell culture work can be carried out without contamination. They come in various standard ratings and because we are working with human cells we are governed under the most stringent guidelines. As sterile hoods tend to be large, heavy and extremely expensive we were forced to build our own hood that complements cellF’s aesthetic concerns as well as adhere to health and safety guidelines. In-cooperating the sterile hood to the final object will allow me to maintain the cultures in sterile conditions (mainly feed them) the gallery or performance spaces.

The Sculptural object of cellF embeds a fully equipped, high precision biological lab that allows following the most complex biological protocols that are needed for the differentiation of the stem cells and the maintenance of the MEA neuronal cultures in field conditions.

Finally, the analogue interface with my external brain was designed to be carried out at the mouth of the piece, what Nathan describes as ‘the station of genesis but also its sonitus ultimum.’ This encapsulating vessel positions and accelerates the non-binary values and with each pass, with each stimulation pulse, particles charge and axons grow. Interjecting and slicing this stream is the aerofoil control surface, a tactile plane situating the performer/observer at the very apex of wetware < > hardware collaboration, whilst laboratory procedures are carried out buffered by this internal flow path in a biologically sealed cavity. New cell cultures are prepared under the vibrationary influence of current active neurons at the interface, and an evolving dialogue of neuro-acoustic chatter infuses the protocol whilst synaptic memories pass along and through to future generations.

 

Sound Prototypes

It is interesting to contextualise the work from a musical perspective and in conversation, Moore referred to several examples. The futurist, Russolo, in the early 1900’s, wrote about the art of noise and was interested in expanding the sonic palette to include noise and noise-making machines; conceptually ahead of its time and not fully realised by others until the 1950’s and 60’s when synthesisers became more commonly used in music. John Cage’s 4’33”, also known as ‘the silent piece’, was an important work in the conceptual development in the field of experimental sound-art; it emphasised the noise of the environment around the performance and the non-musical aspects around the music. David Tudor, in the 1990’s, combined the engineering of electronics with the inspiration of biology and developed a synthesiser that was controlled by an artificial computer coded ‘brain’, not made from biological matter, but closely resembling one in its activity and intention and used it to composed and play a series of works titled Neural Synthesis Nos. 6-9. In other words, Tudor’s artificial neural network simulated the way real biological neural networks operate using a computer code and wired this to a synthesiser to create sound.  cellF builds on these precedents, and in particular it takes Tudor’s vision a step further from using an artificial neural network and making use of a real biological neural network to play electronic music.

 

Experimenting with 7 electrode interface (out of 60) and various sound aesthetics. 

First time we connected the neurons to a synthesizer. We used only 1 electrode (out of 60)

Another very early prototype.

spacer 1

Collaborators

Andrew Fitch

Dr. Andrew Fitch completed a B.Eng (Hons) in 2010 and his PhD in 2015, at The University of Western Australia. His main research focused on memristor-based chaotic circuits and has published a book, journal articles and book chapters describing the results of this work.
He has been building analogue synthesizers since 1998 and currently runs Nonlinearcircuits, producing a wide variety of unique designs intended for use in experimental music.
When not building synthesizers, Andrew enjoys bushwalking with his dogs and reading books by A.E. van Vogt. http://nonlinearcircuits.blogspot.com.au/

Darren Moore

Darren Moore is an Australian drummer, electronic music based in Tokyo, Japan. He performs regularly on drum set and modular synthesizer in predominantly in the fields of jazz, experimental and improvised music styles. Darren has performed extensively in Australia, South-East Asia, Japan and Europe. He completed his Doctorate of Musical Arts at Griffith University in Brisbane Australia in 2013 which looked at the adaptation of Carnatic Indian rhythms to drum set.

Douglas Bakkum

Douglas Bakkum is a scientist and group leader in the Department of Biosystems Science and Engineering at ETH Zurich, Switzerland. He received a Ph.D. in Neuroengineering from the Georgia Institute of Technology and Emory University in Atlanta, GA, USA. His interdisciplinary background includes training in neuroscience, mechanical engineering, robotics, and artificial intelligence. He currently leads projects to find the fundamental rules neurons use to communicate with each other and how such rules can scale to produce learning, memory, and creativity.

Mike Edel

Dr. Michael Edel has specialised his university studies in the basic and fundamental principles of Anatomy (achieving Honours), Embryology and Human Physiology, completing his PhD in Pathology in the year 2000 at the University of Western Australia, Perth, Australia. He moved to Barcelona in 2004 and works with his wife, Dr. Ana Belen Alvarez Palomo, to develop new clinical grade cell reprogramming technology to make stem cells to study and treat human disease. Dr. Michael Edel is currently a Group Leader/Ramon y Cajal Investigator (RYC-2010-06512)  and accredited Associate Professor teaching and research based at the University of Barcelona, Faculty of Medicine. He is a senior researcher in the field of IPSCs, hESC, gene regulation, epigenetics, cell cycle, differentiation to neurons or cardiac muscle cells for the use of such cells as a cell replacement therapy. He has 38 publications to date, a book on the state of the art of iPSC technology (2015), ten years post-doctoral experience and five years as group leader. He is Affiliated as Senior Research Fellow at the University of Western Australia, and the University of Sydney, Faculty of Medicine, and Visiting International Research Fellow at the Victor Chang Cardiac Research Institute, Sydney, Australia.  He has been awarded over 850,000 Euros in project grants the past 8 years as chief investigator and has a NH&MRC UWA near miss grant to develop a project on bioengineering cardiac muscle from iPSC to treat heart disease.. He has been invited as a guest speaker as part of the distinguished faculty or co-chair for over 20 congresses the past ten years.  For more information see his Lab Web Page:   http://pluripotencylaboratory.wordpress.com/

Nathan Thompson

Nathan Thompson is a multi-disciplinary artist exploring the possibilities of man/machine interaction and the hidden corners that arise from this relationship. Mostly he implements machine/robots that play along the blurred edge of the interactive while showing independent thought, only slightly tethered to the audiences actions. His work often questions the role of humans in the natural landscape and through these investigations, builds a greater understanding of our inhabited space. His machines are self built, analogue and lifelike in their behavior, using custom electronic Neural-type networks that are on their own, very simple but when piggybacked, multiplied and fed back into the stream, display behavior remarkably organic. The interpretations of this evolving machine language lead to an installation unique to every setting… the participant, oftentimes, both controls and is controlled by the mechanics of these unique lifelike constructions.

Stuart Hodgetts

Associate Professor. Stuart Hodgetts (SH) is currently Director of the Spinal Cord Repair Laboratory in the School of Anatomy, Physiology & Human Biology, at the University of Western Australia (UWA). He has extensive knowledge and expertise in cell based transplantation therapies and has been devoted to this research since 1998. Previously, SH has worked at the University of Essex, UK where he obtained his Ph.D. did his first postdoctoral appointment, before working overseas at the Oklahoma Medical Research Foundation, USA. SH returned to Australia and joined UWA in 1998, conducting research in cell-based transplantation for neuromuscular diseases such as muscular dystrophy, and since 2004 in the repair of spinal cord following injury using stem cells.  He is developing technologies for the use of Induced Pluripotent Stem Cells (iPSCs) for use in spinal cord injury, and is also using this in a number of art/science collaborations.  In particular as part of a collaboration with Guy Ben Ary (also at UWA). Since 2005, SH has been involved in the generation of nearly $3 million in research funds and has published in many highly ranked peer reviewed journals. As a member of Academic staff he teaches undergraduates and postgraduates (currently supervision of 5 PhD students). SH is an active member of committees at the School and Faculty levels and is also Chair of the Animal Users Group at UWA.  In addition, he has had a long standing relationship with Symbiotica (ANHB, UWA), involving collaboration with many artists and residents.  A long standing advocate of this cross-disciplinary research, he is currently Symbiotica’s Scientific Consultant and Adviser.  Over the years SH has been an invited speaker for his Science/Art work at; ANZCAART, The Body, Art and Bioethics Conference, The 6th European Meeting of the Society for Literature, Science & the Arts, and has conducted a Tissue Engineering workshop in the UK as well as exhibited works in collaboration with Symbiotica at ISEA.  SH has also recently been involved in funding successes together with Guy Ben-Ary and Ionat Zurr from Symbiotica.

Support

 

SymbioticA, The Centre for Excellence in Biological Arts

The University of Western Australia

Government of Western Australia, Department for Culture and the Arts 

Lasalle_Logo_1 gray

Australian Council for the Arts