How close are we to being able to read people’s brains today?
Minor spoilers ahead
First some background
Netflix’s Black Mirror explores possible futures for humanity and civilization in each episode. In the latest fourth season, episode 3 (titled “Crocodile”) is set in a future where a device called a “Recaller” has been developed. A recaller is a small device which is placed on someone’s head right by the temple and decode’s a person’s memories as they are being recalled. The device transmits the decoded memories to a small computer which can play the memories like a movie. In this episode we follow Shariza who is using a recaller to investigate an insurance claim by a man hit by a self-driving pizza truck.
What is a memory engram?
In order to talk about how this technology would be possible we first need to know what exactly a memory is so we can figure out ways to read them. As said in the episode, the recaller is reading memory engrams. An engram is indeed a real concept which neuroscientist use to describe memories. The idea of an engram is that is it the changes to the brain which an experience causes and these changes allow us to remember these experiences later. Another to describe an engram is all the neurons which are firing when we experience something get connected together. For example, if in the morning you’re trying to remember where you last put your keys, the neurons in your visual cortex which were responsive as you were putting your keys down the night before are reactivated. The neurons in the visual cortex tend to encode quite rudimentary information though, like lines, colors, and basic shapes. Neurons in brain regions which make up the ventral visual stream, a group of brain regions that take rudimentary shapes, identify what they are, and send them to the memory centers, are also reactivated. Further reactivated are the memory centers of the brain which organized the memory of putting your keys down last night, not to be mixed up with where you put your keys two nights ago. Thus, all of these neurons which took part in giving you the ability to see your hand put down your keys, understand that the things you had in your hand were keys, and make sure you didn’t confuse last night with other nights, come synchronously activate through their connections to help you remember. All of these neurons are the engram of you key placement memory.
How do we see these neurons in an engram like the recaller?
One way to see an engram is to literally just look at these neurons with a special microscope, called a miniscope. Miniscopes are very small microscopes that show neurons firing. The neurons are first given special proteins which light up when the neurons fire, just like glowing jellyfish (actually literally from jellyfish). This way you can see all the neurons which are activated while the mouse or rat is going about his day. Once you knew which neurons were active for an experience you could tell what the mouse was doing just by his brain! For example on day one of an experiment, you see neurons 1, 2, and 3 active while the mouse visited his friend and neurons 4, 5, and 6 active while he drank water. On day 2 you don’t see the mouse but see that neurons 1, 2, and 3 where active you’d have a pretty good idea the mouse was seeing his friend.
While this technique is very cool it does require a quite a bit of surgery to get the miniscope in place. Additionally it only lets scientists look at a small part of the brain, a few millimeters square. Although, in those few millimeters there are hundreds of neurons which give scientists a good idea about what the mouse is doing. Instead of implanting a miniscope, what about tiny electrodes? Scientists and engineers are making smaller and smaller electrodes to implant for research projects in animals and humans. These electrodes pick up the electrical activity of hundreds to thousands of neurons. Additionally they can be placed in various brain regions to eavesdrop on neurons all over the brain. Electrodes work well on monitoring all the neurons that make up an engram. For example, researchers can record neurons from the navigation brain center then use the activity to decode the paths an animal took. Even better than that, you can record these place neurons while a rat is asleep and see him dreaming about the maze and the paths he took earlier that day!
Both the miniscope and electrodes allow scientists to read the brains of rats and figure out what they are doing and even what they are dreaming of doing. But what if you don’t like undergoing brain surgery just to be a witness in an insurance case? Well we are beginning to be able to read minds using non-invasive fMRI brain scanners. And not only can we read people’s brains, but a paper in 2011 by Nishimoto et al. was able to reconstruct movie scenes based on the brain activity of people watching those scenes (video below). In other words, for every second of brain activity in the visual cortex their computer model was able to find movie clips (out of 18,000,000 possible clips) which were similar to the clip participants watched during that second. Putting all these estimated clips together produces the results below. That is they used their brain activity while watching a movie to try to recreate that movie. You can see how close this is to the Black Mirror recaller. Perhaps the recaller takes people’s brain activity, compares it to a database of brain activity taken from people watching movies, and tries to reconstruct what the person saw.
So, we already have a recaller?
Nooot quite. All of these brain reading techniques require researches to know what the mouse was doing or what the person was seeing at the brain is being recorded. We already knew neurons 1, 2, and 3 were social neurons from day 1 of the experiment. Researchers using the brain scanners had to first know the movies the participants saw. Without knowing what caused the brain activity in the first place there is no way to estimate what brain activity might mean. It is possible, as I speculate above, that the Black Mirror recaller uses a known database which links the average brain activity of people watching movies to the brain activity as a witness is remembering to recreate what they are remembering. Keep in mind from earlier, when you remember something, all the same neurons which were active at the time of the experience (the engram) reactive. Thus, remembering a murder scene is similar to actually seeing a murder scene, be it a real murder scene or one in a movie. However, unless the maker of the recaller does indeed have a massive database, we currently have no way of looking at an engram and knowing what caused it.
Another issue is size. The recaller was the size of a coin and looked to painlessly stick to a witness’ skin. Meanwhile, fMRI scanners are massive machines the size of a small car and cost as much as big house. An alternative to fMRI machines for much less expensive, non-invasive brain activity monitoring is EEG. EEG measures electrical activity like electrodes inside the brain do but EEG is only from the scalp. However, the skull and scalp dampen the resolution of EEG and only overall brain region activity can be monitored (as opposed to individual neurons with electrodes in the brain). EEG is used quite successfully in decoding movement activity in the brain. This has allowed paraplegics or those who can’t move due to stroke control an exoskeleton or move around in video games (below). Due to this loss of resolution, EEG currently isn’t very useful for decoding complex information, like visual scenes. However, many researchers are working on EEG. A recent paper tired to guess the type (dog, ball, plant, etc) of object people saw based on their EEG activity. Using 2,000 pictures divided into 40 types, they could predict the class with 40% accuracy (2.5% would be chance guessing levels, like flipping a coin). While this is pretty good, but no where near admissible in court level accuracy. And again, researchers already knew the types of images whereas the recaller seems to be able to reconstruct any type of scene from an engram.
Will we have a recaller?
Our current technology today seems quite close. If we assume the recaller uses a database to reconstruct activity, everything else about the recaller comes down to advances in engineering. Scaling down our current brain monitoring devices to the size of the recaller doesn’t seem too far fetched. We can already monitor rodent and bat brains with wireless electrode devices weighing a few grams and a laptop. EEG can be picked up by little wearable patches that link to your smartphone. Rather, the biggest issue somehow getting enough neural data from such a small device, enough data to decode an entire memory. The recaller is put over the temple (as a side note this is pretty odd; there is very little brain there and the visual cortex is in the back of the head and the memory center is by the ear) and is able to recreate scenes from people’s memories. Our current devices monitor a few hundred single neurons in one region (miniscope, invasive electrodes) or gross activity all over the brain (EEG, fMRI). Merging the fidelity of invasive techniques with the scope of non-invasive techniques would be quite the feat. The largest hurdle to overcome would be physics itself. As neural signals spreads across the brain and tissue it decays and loses energy. This is why EEG has poor resolution and can’t record single neurons. Picking up activity from the visual cortex from the temple would need uber sensitivity to detect any energy left originating from visual neuron activity. At present, this physics problem seems impossible to overcome. More likely, recaller’s in the future will look like modern EEG caps with electrodes all over the skull to detect activity all over the surface of the brain. That is of course if we don’t all get elect to get brain implants to boost our memory and telepathically talk to friends.
Leave a Reply