The Franklin Institute Science Wired Huffington Post Philly.com NSF VIZZIES Nature Colossal UpWorthy

Self Reflected

"Self Reflected" Artwork by Greg Dunn and Brian Edwards. Video by Will Drinker. Music by David Haldeman.

What:

Self Reflected is your brain perceiving itself. It is perhaps the most fundamental self portrait ever created, a hyperdetailed animated representation of human consciousness designed to mirror the functioning of the viewer’s own mind in the very moment that you are observing the piece. Self Reflected asks the question whether the brain is uniquely tuned to appreciate its own fractal-like anatomy and elegant, wavelike electrical activity as a consequence of those traits underlying its own construction. It is a work of neonaturalism, inspired by the cutting edge of neuroscience and engineering to expand our understanding of the natural world.

Much of Science is understood and taught through a process called reductionism. We take something very complicated and reduce it down to the composition of many small mechanisms, each of which is easily understood. Toward these ends, we understand the neuron very well. However, some things, such as consciousness, are irreducible. We have not been able to find consciousness through a detailed understanding of either one neuron or even the combination of many. However, we have a sense that it comes from the intricate combination of order, randomness, and chaos that is continually happening within our brain when we combine a sufficiently large number of neurons, although we still don't and possibly never will understand it.

In many cases, art is a superior communicator of complex and nuanced ideas. Self Reflected was not created to simplify the brain’s functionality for easier consumption, but rather to depict it as close to its native complexity as possible so that the viewer comes away with a visceral and emotional understanding of its beauty. Though the neuroscience of the piece was painstakingly researched to give the piece a level of reality not seen on this scale before, Self Reflected’s deeper meaning is to elevate the understanding of the average person to the exquisite machine that most defines our humanity.

This piece resides on permanent loan with the Franklin Institute in Philadelphia in their Your Brain Exhibit.

How:

Like any large engineering project, the creation of Self Reflected needed to be broken down into bite-sized pieces. The division for this piece was fundamentally drawn in terms of scale and dimension. Neurons consist of several different parts. We consider the dendritic arbor and soma as the cell body, and treat it separately from the axon. The cell bodies span several microns in the human brain, but the axons can stretch out several centimeters or longer. We made the decision to handle the cell bodies “by hand” while the axons were handled by a computer. Of course, they must ultimately form complete neurons, so they both must be engineered to interface together. What’s more, when they interface, they must form real neural pathways that will be depicted through time. Therefore, this project can be thought of as the collection and integration of three types of information: microscopic data, macroscopic data, and behavioral data.

Microetchings are 2D objects and the brain is 3D. While microetchings can contain one extra dimension, which we could have used to denote depth, we decided early in this work to use the extra dimension to denote time. Therefore, the brain must be reduced to a 2D plane so that we can put it on our microetching canvas. We chose to depict an oblique slice that contains important brain structures such as the cerebellum, frontal cortex, and brain stem. This was largely an aesthetic decision, but one which provided a nice starting point for an otherwise intractable problem. Within this slice, we divided the brain up into about one hundred different functional regions of which we could then ask the following questions.

Like any large engineering project, the creation of Self Reflected needed to be broken down into bite-sized pieces. The division for this piece was fundamentally drawn in terms of scale and dimension. Neurons consist of several different parts. We consider the dendritic arbor and soma as the cell body, and treat it separately from the axon. The cell bodies span several microns in the human brain, but the axons can stretch out several inches or longer. We made the decision to handle the cell bodies “by hand” while the axons were handled by a computer. Of course, they must ultimately form complete neurons, so they both must be engineered to interface together. What’s more, when they interface, they must form real neural pathways that will be depicted through time. Therefore, this project can be thought of as the collection and integration of three types of information: microscopic data, macroscopic data, and behavioral data.

What does it look like on the microscopic scale? What is the morphology of each neuron and what is the resulting texture from their placement? This forms the gray matter of the brain.
What other regions does it connect to and what do those connections look like? This forms the basis of the white matter of the brain.
How are those connections utilized through time for some prototypical thoughts? This informs our choreography.

Each of these questions was extensively researched and a dossier was created for each region.

Microscopic Data Preparation

A handful of the several hundred vectorized and processed cell body.

On the microscopic scale, each one of these functional regions was extensively researched to understand the neuron morphology and distribution or the neurons. This involved a lot of literature research and collection of micrographs. The information gathered into a wiki and we developed a library of neuron images. These neurons were used as models for Greg’s trademark ink blowing technique. Hundreds of neurons (sans axon) were created by hand. These were scanned and vectorized. Each of these neurons contained a soma from which the axon was expected to emerge. To help the computer understand what part of the drawing was the soma, a “key” was cut out of the vectorized shape at this location. The “key” consisted of ten straight lines in a row. Straight lines are exceedingly rare in biology so this was easy for the computer to identify later.

We used Adobe Illustrator to form vectorial brushes of these cell bodies. These brushes were used to paint each functional brain region. Large scale photos were used as a guide for the overall placement of the regions while micrographs were used to attempt to match texture. Each brain region was saved separately and the file was then processed to extract the location of the soma of each neuron. This yielded a data block in which each neuron has been located and numbered region by region. This data was set aside for later.

DSI track data forming long range connections between different regions.

Macroscopic Data Preparation

Scientists have a technique that they use to understand how the brain is connected. The process is called Diffusion Spectrum Imaging(DSI) and is a variation on the more commonly known MRI medical imaging technique. If we imagine the long range white matter in the brain a fibrous rope, then it follows that water will more easily seep along the length of this rope than it will across it. The data extracted in the DSI technique indicates point by point which direction water most easily travels. Fibers can be inferred starting at a random location and making thousands of small steps wherein on each step you choose a direction to walk based on the direction water most easily diffuses.

Our friend, John Pyles, works in a lab that performs this type of research and was scanned in an MRI to yield the DSI data for his brain. We examined all of the connections between each of the regions we had designated earlier. This formed a sparse matrix of track sets.

Behavioral Data Preparation

One of the dozen scripts describing the interaction between the different brain regions and long range connections.

Through the use of Functional MRI and various other probing techniques, scientists have a rough idea on what regions of the brain are used in what kind of thoughts. Through a series of interviews with various experts, Greg gathered this information and distilled it into a highly formulated script suitable for a computer. Each line of the script started with a named activated collection of neurons and routed them to a region using a named a DSI track set. The connected neurons would become “activated” themselves and could then be used for a subsequent step In the neural pathway. Each step had about 25 different parameters that could be set including velocity, size, roughness, beginning count, end count, efferent connections, densities, and many others. There were about 25 such neural pathways, each with upwards of 25 steps.

Algorithmic Processing

A rather involved computer code read each script and used it to construct a causal network of neurons within the brains. This was done in the following steps:

Each neural pathway imported and interpreted to determine the neuron counts required of each region and perform basic checks for feasibility.
The list of all available neurons was consulted. For each step of the choreography, a population of efferent neurons was chosen, labeled, and removed for the list of available neurons. While not purely biological, within a microetching a neuron cannot fire more than once. In the case of long range connections, efferent populations were heuristically chosen based on the positions of the afferent activated populations, track set data, and a combination of parameters that effected a stochastic drawing. Short range connections did no utilize track set data but was otherwise similar.
With a limited population designated for each step of the neural pathway, the problem of forming neural connections was now tractable. The connections were created using Linear Programming to optimize the creation of links between the afferent population, individual tracks in the track sets, and the efferent population. The cell body centers are now all linked together in a large mathematical causal network.
Each edge of the network has a length in microseconds based distance between the two somas, use of a track, and the action potential velocity profile. The whole system of edge lengths could be inserted into a least-squares solver to find the optimal firing time for each soma. This yielded a spatial-temporal network of cell body centers.
At this stage, we are using color hue as a proxy for time. The spatial-temporal network was consulted and each cell body centers waw replaced with the appropriately colored vectorized cell bodies from the original art files. The keys were removed algorithmically. The axons were drawn in using a combination of smoothing filters and noise filters. These were colored along their length with smoothly varying hue to denote the propagation of the axon potential along its length.
Each neural pathway was saved as a separate vector artwork file.

One of the twenty-five plates fully rendered.  Hue denotes firing time. — One of the twenty-five plates fully rendered. Hue denotes firing time.

Finally, all of the vector artworkfiles were then opened together and composed into one single image. This image was then divided and saved as 25 different raster images, each representing a separate plate of the microetching.

Scanning Electron Micrograph of an extremely small section of the fabricated piece.

Hatching

Microetchings are comprised of collections of angled ridges that are designed to gather light from a light source and one location, and reflect it to an observer at another location. If either the light source or the observer moves, the regions that were just bright on the piece should suddenly go dark. However, a nearby region might be engineered to be bright for this observer light-source combination. Through a smooth and subtle changes in the angles of the ridges, microetchings can appear to animate either by moving the light source or moving the observer. In the case of this work, we choose to “move” the light source by creating a giant linear array of almost 200 LEDs lights illuminating the piece spanning close to 40 feet. Each one activated in turn to illuminate the piece from a slightly different location giving the appearance of motion of the light source.

The color of each pixel is used to denote the time of the neural activation. However, given the mechanism through which microetchings work, each color now corresponds to a different LED light within the array. Each pixel of the resultant image then is subdivided into an 8x8 grid and replaced with a binary stripe pattern that will be used to create the ridges. Details on the mathematics on calculating the stripe pattern can be found here. These 25 very high resolution black and white images were enormous. These were sent off to be printed on clear plastic sheets by a printer who specializes in the creation of masks for microfab lithography. These sheets were then used to fabricate the plates in a process which is detailed here.