Peripheral Neural Interfaces

Motivation

Augmented and virtual reality are loci of increasing research and development funding. The smartphone improves only asymptotically; at present, models of industry leaders differ only superficially each year. AR, in particular, offers opportunity for a fundamentally new basis of human-computer interaction, making indistinguishable the real and virtual worlds and perfect substitutes of many real and digital goods, all in a space more natural than a planar display. Because vision is fundamental to the AR or VR experience, most resources designated to the space are isolated to innovating optical architectures for an improved visual experience, particularly in firms with AR ambitions: AR optics are difficult. However, as major players release VR products with visual quality sufficient to have widespread utility, technology for interface with the other senses has become a topic of increased interest. Methods of interaction in 3D space are necessarily more complex than those for a planar display, and design of AR systems for continuous waking use seriously constrains any additional systems used for control. Auditory and haptic augmentation are of particular interest. Auditory technology is mature and simple to unobtrusively integrate. Haptic technology presents more of a challenge: Few consumers are willing to carry joysticks or wear haptic gloves at all times. Methods of high-resolution localization of the body, particularly the hands, are also of particular interest, and are again constrained by social norms and the necessity of continuous use. Consumers are leery of constant video surveillance, if only for the purposes of hand tracking.

I focus on hand tracking technology: nascent but necessary for any simple AR or VR user interface. Specifically, any manipulation of virtual objects in 3D space requires sensing and feedback of the hands: these are the agents by which we manipulate real objects in 3D space. Even basic additional sensory augmentation, however, can reap substantial experiential benefits. Obvious but unwieldy methods of localization and realistic haptics exist: the aforementioned joysticks and gloves. These technologies represent current consumer-grade product deployments, mostly in VR, where the physical and social costs of such systems, as well as the period of continuous use, are reduced. I explore technologies read directly from and write directly to neurons. There already exists an information system in the human body. Terminally, it is the rational choice for both measuring movement (motor signals) and producing stimulation. Some sensations are inaccessible by other means. For example, muscle spindles, Golgi tendon organs, and joint receptors produce the sensations of physical resistance basic to physical interaction but are not available for any direct stimulation (as contrasted to touch/pressure). But for exoskeleton concepts, neural technologies are the only clear solution.

The Peripheral Nervous System

The nervous system is tremendously complex. Much of the nervous system, particularly the brain, is poorly understood. The peripheral nervous system, the nerves and ganglia exterior to the brain and spinal cord, is an attractive locus for an interface precisely because its structure and dynamics are comparatively simpler, but even the peripheral nervous system represents a massively interconnected nonlinear system employing a diversity of electrical and chemical communication techniques in a nearly intractable variety of nervous and transductory cells. Because the hardware produced in this thesis is noninvasive and therefore relatively low resolution, a neuron-scale understanding of the forearm might be thought unnecessary. Indeed, the nerves of the arm have not been mapped to such precision, and individual anatomy varies widely. Teams now at Meta producing forearm-based electromyographic (EMG) systems similar to that in this thesis, however, claim the ability to "reconstruct the activity of individual neurons", and have, accordingly, built models that predict the position of particular muscles important to the position of the hand. I therefore expound the peripheral nervous system to neural resolution.

The Neuron

The basic cellular unit of the nervous system is the neuron. Neurons are distinguished from other cells by their electrical excitability and function of information transfer and processing in organisms. Neuron anatomies are diverse, even among neurons of the same function, but typically include a soma (cell body), dendrites (receptive filaments), and an axon (transmissive filament). Information transmission within individual neurons occurs primarily via action potentials, binary, rapidly propagating inversions of the membrane potential; however, information can also be transmitted slowly over short distances via graded potentials, as in the retina. Action potentials are also known as spikes and sequences of action potentials as spike trains. Information travels between neurons via the emission and reception of chemical neurotransmitters. Dendrites, dense ligand-gated ion channels or metabotropic receptors, which indirectly modulate ion channels, produce changes in membrane potential as a function of local neurotransmitter concentration, and are present in profusion, largely electrically isolated from each other by the narrow volume adjoining each to the soma. The soma contains the nucleus and majority of organelles. Its surface is dense with voltage-activated ion channels, which transmit signals produced in the dendrites to the axon. The region where the axon joins the soma, the axon hillock, has the highest density of voltage-gated ion channels, used to trigger action potentials on the axon in response to sufficient collective dendritic stimulation. The axon, a protruding filament as long as several meters, propagates the spike, insulated intermittently by the myelin sheath, which prevents ion transfer across the membrane. Myelin, a type of glial cell, increases the velocity of signal transmission and reduces signal decay but is incompatible with ion channels. Accordingly, nodes of Ranvier, axonal regions absent myelin and dense in voltage-gated ion channels, exist intermittently to boost the signal. At the axon's extremity, presynaptic terminals contain action-potential-released neurotransmitters. The chemical diffusion of neurotransmitters between cells is slow as compared to the rate of electrical signal transmission, adding latencies of at least 1 ms. Some particularly time-sensitive neural circuits employ electrical synapses in which the transmissive and receptive cell membranes are joined by pores called connexons, eliminating such delays. Neurons are facilitated in their function by glial cells.

Glial cells, non-neuronal cells of the nervous system, provide assistive function to neurons, forming a matrix by which neurons are mechanically supported, transporting nutrients and oxygen to the neurons they surround, and protecting those neurons from pathogens. Glial cells also have important electrical function, forming myelin, a material that aids long-distance action potential propagation, and electrically isolating contiguous neurons. Glia remove dead or damaged neurons, and in the peripheral nervous system have been demonstrated to perform neuronal repair and regeneration. Present research suggests that the ratio of glial to neuronal cells correlates to neural performance; greater metabolic function is supported by larger volumes of glial cells, and even that increasing glial ratios in the evolving human brain might have contributed to increased performance relative to other primates, though those specific conclusions remain contested.

Information Transfer in Neurons

Action potentials are the basic unit of neural information transfer. In neurons they permit the swift transmission of information. Neurons maintain a static membrane potential of approximately -70 mV, an equilibrium of sodium, potassium, and calcium ions' concentration gradients and electrical potentials produced by a variety of (selective) ion channels and pumps in the membrane. Action potentials are enabled by voltage-gated ion channels in the membrane, which open in response to increases in local membrane potential past a threshold voltage (typically approximately -45 mV). Sodium voltage-gated ion channels, which sustain voltage inversions for less than 1 ms, facilitate the majority of action potentials, but calcium voltage-gated channels, which produce inversions of more than 100 ms, exist in some neurons and drive repeated sodium action potentials. Sodium-channel-based action potentials begin with the opening of sodium voltage-gated channels, which produces a transient increase in membrane potential toward the sodium equilibrium potential (approximately +70 mV). This increase in membrane potential activates potassium voltage-gated ion channels, reducing membrane potential toward the potassium equilibrium potential (approximately -70 mV), causing an ephemeral reduction in action potential below static potential (hyperpolarization). After the passage of an action potential, a refractory period is required to restore prepotential ion concentrations. Information can also be transmitted by less binary changes in membrane potential.

Less commonly, information is transmitted in neurons via graded potentials. Action potentials are thought to encode information basically digitally; the amplitude of a particular action potential is irrelevant, only whether membrane potential is substantially perturbed from stasis or not. Graded potentials, however, transmit information using amplitude. In neurons, the mechanisms of graded potential propagation limit their range. Graded potentials are very susceptible to noise, but can transmit more information than spike trains, and otherwise share many of the advantages and disadvantages of analog information transmission as compared to digital information transmission. Graded potentials are observed in systems like mammalian retinas, permitting discrimination of fine gradations in light pattern and intensity but sacrificing speed, temporal resolution, and threshold discrimination. The final major mechanism of information transfer in neuronal systems is neurotransmitters.

Neurotransmitters, synaptic signaling molecules, are used by a majority of neurons for information transmission between neurons. Excitatory and inhibitory neurotransmitters exist; approximately one hundred distinct neurotransmitters have been identified. Generally, neurotransmitters are stored in vesicles that merge with the cellular membrane upon sufficient electrical stimulus. Neurotransmitter molecules then diffuse across the synaptic cleft to receptors on the surface of the target cell, introducing a delay of about 0.5 ms to the transmission of the information. Neurotransmitter dynamics are complex and the study of much pharmacological study but have not been employed in neural interface systems. An understanding of the mechanisms of neural information transfer is prerequisite to an understanding of neuronal coding schemes.

Neural Coding

Much work remains before to produce a detailed understanding of neurons' information coding schemes, particularly among and by ensembles of neurons, but some basic behaviors have been identified. Neurons are thought to code near the channel limit. Because distinct action potentials within neural systems are nearly identical, the form of the action potential is not thought to be important to information transmission. And action potentials are typically distinct, well separated. Information is therefore encoded in the temporal sequence of action potentials (spike trains). Particular temporal coding schemes are a subject of considerable research and debate.

The first coding theories emerged from the 1926 experiments of Adrian and Zotterman, who tensioned muscles using masses of different weights and noted a correlation between the number of recorded spikes and weight used for tension. Rate coding, the model that emerged in response, holds that information in the spike train is encoded in the firing rate. A variety of neural systems have since demonstrated performance far in excess of rate coding, transmitting accurate information about stimuli with characteristic timescales of the order of the period of a single spike. Sound localization, for example, requires coding on the millisecond scale. Even in neurons for which rate coding remains a candidate model, recent experiments have demonstrated that the coding scheme fails to account for much of the signals' total information content. Furthermore, in many systems, rate coding, if partially explanatory, has been demonstrated to be highly nonlinear to stimulus.

As a consequence of the failures of the rate coding model, a number of more complex models of neural coding have emerged. Temporal coding holds that neurons transmit information via the precise timing of spikes (rather than the time-averaged rate of spikes), and has been demonstrated in coding of texture, for example, in macaques, in combination with rate coding, or in rats' coding of whisker stimulation. In phase-of-firing models, spike count and timing relative to a reference oscillation are thought to encode information. Phase-of-firing models again fail to explain the total information content of neural signals, but have been shown to contribute to total information transmission in systems like the visual cortex of macaques. In correlation coding models, correlations between spikes in a train transmit information additional to that communicated by spike timing. Correlation coding is also thought to implement redundancy essential to sparse temporal models. As the redundancy of communication decreases (corresponding to increases in possible information content), noise becomes increasingly destructive to information transmission. Neurons almost certainly communicate information via parameters of their collective response. Population coding also conceives that the collective responses of sets of neurons is used to encode information densely at timescales shorter than possible by individual neural response. Control of eye and arm movement is more accurate than theoretically possible as a consequence of single neurons. The population encoding of efferent motor signals, however, is not well understood. Empirical studies have revealed a variety of coding schemes in different contexts; often, several coding schemes each describe a portion of the total Shannon information transmitted by the nerve. Neuronal noise (and the correlation of noise between neurons), however, complicates coding studies: neuronal responses to identical stimuli are never identical, even in simple neural systems. The mechanisms of neural information transmission, coding, remain equivocal in practice.

Theoretical work, however, provides some basic results. Decoding sparse temporal codes is simpler than their encoding. Nonlinear encoding, for example, typical of modes of sensory transaction, does not require nonlinear decoders. Shannon information theory bounds the total information content temporally encoded (digital) spikes and rate encoded spikes, suggesting that several bits of information may be encoded by a single spike. Empirical information content approaches theoretic limits, suggesting that theoretic results may be approximately representative. Theoretic models suggest that temporal coding is the brain's dominant transmission scheme, that individual spikes, even in ensembles of neurons, are basic to total information content. The details of neuronal coding, particularly ensembles of neurons, are not well understood. Machine learning models are therefore useful for elucidating their content.

Sensory Transduction

Mechanisms of information input to the peripheral nervous system are similarly complex. Stimuli are encoded logarithmically into the amplitude of the graded change in receptor potential (short receptors) or modulation of the frequency of a spike chain (long receptors). Sensory receptors generally saturate at high stimulus levels and adapt to sustained stimuli. Receptors that adapt over long durations are useful for measuring stimulus duration; those that adapt quickly measure rapid changes to stimuli.

Afferent neurons in the skin, somatosensory (delocalized) nerves, terminate in a variety of structures, each activated by distinct stimuli and corresponding to a particular sensation. Such structures include Merkel's disc, small receptive fields stimulated by sustained pressure; Ruffini's corpuscle, large receptive fields stimulated by sustained tension; Meissner's corpuscle, small, extremely sensitive receptive fields stimulated by light pressure and low-frequency pressure; the Pacinian corpuscle, large receptive fields stimulated by vibration and low frequency pressure stimulus; and hair follicles, sensitive to minute movements of particular hairs.

Receptors are dense and diverse in the skin; decoding output of even a small area (particularly when the nerve signals are very spatially dense) is exceedingly complicated.

Motor Output

In humans, motor impulses originate in the primary motor cortex. Motor output density (and somatosensory density) varies widely; the hand (primary mode of physical interaction) as well as the lips, jaw, and tongue (regions associated with speech production) are particularly dense in motor output. Nerve output to the hand is therefore a promising candidate as a region to use for high data rate information transfer out of the brain. Impulses travel to and down the spinal cord via the upper motor neurons and to the α motor neurons, which transmit the signals to the extrafusal muscle fibers via the neuromuscular junction. α motor neurons are heavily myelinated and larger in diameter than other nerve cells as to increase transmission speed. An individual motor neuron may innervate many muscle fibers. Extrafusal muscle fibers compose the majority of muscle tissue.

Neuroanatomy of the Hand

Three primary nerves service the hand. Branching from the brachial plexus, itself formed by the C5 and T1 spinal nerves, are the median, ulnar, and radial nerves. Each contains both efferent and afferent fibers. The median nerve innervates specific actions of the elbow, forearm joints, wrist joints, motions of the outer joints of the second through fifth digits (the fingers), and motions of all joints of the thumb. The radial nerve controls the same joints as the median nerve, but in different directions or in distinct degrees of freedom. The ulnar nerve communicates signals to the inner and outer joints of the second through fifth digits (the fingers).

The nerves are remarkably dense; the ulnar nerve, for example, has cross sectional dimensions of approximately 6 mm by 2 mm. The specific number of innervating neurons dedicated to each muscle varies approximately as the number of independently acting muscle fibers. Cross sections of the spinal cord bisect tens of thousands of axons, several thousand of which enter the three primary nerves of each arm. Neural decoding is made difficult by such extreme spatial proximity.

If surface electrodes are to be employed to measure information in the arm, the signals collected are actually the potentials of the relevant muscles, delocalized, amplified electrical responses of the relevant nerves: electromyography. Knowledge of the locations and functions of the muscles driving movements of the hand is therefore critical to electrode placement. Such muscle delineation can be accomplished ad hoc by moving each joint of each digit in turn and identifying the locations of muscle activity. The hand is controlled by an extrinsic group, located on the forearm, and a smaller intrinsic group, located in the hand. Muscles of the forearm are stratified. Most superficial (closest to the skin) on the palmar side (side of the palm) are the pronator teres (elbow), flexor carpi radialis (wrist), palmaris longus (elbow and wrist), and flexor carpi ulnaris (wrist), under which is the flexor digitorum superficialis (second through fifth fingers). Under both are the flexor pollicis longus (wrist and thumb) and flexor digitorum profundus (wrist and second through fifth fingers). Though the muscles controlling the fingers and thumb, then, are not immediately available, some sensors should be placed over the flexor digitorum profundus, flexor digitorum superficialis, and flexor pollicis longus to aggregate maximal information content about finger location from the palmar side of the forearm.

On the dorsal side, the extensor carpi ulnaris (wrist), extensor digiti minimi (wrist, the fifth digit), extensor digitorum (wrist and second through fifth fingers), extensor carpi radialis longus (elbow and wrist), and extensor carpi radialis brevis (elbow and wrist) are layered atop the abductor pollicis (wrist and thumb), extensor pollicis brevis (wrist and thumb), extensor pollicis longus (wrist and thumb), and extensor indicis (wrist and second joint). The extensor digiti minimi, extensor digitorum, abductor pollicis, extensor pollicis longus, and extensor indicis, then, should receive sensor coverage to ensure maximal information about finger movements is collected.

The muscles surrounding the base of the thumb are relevant to its fine control but introducing sensors to the palm seems a sufficient impediment to normal activity as to be unrealistic.

Prior Neural Interfaces

Since the identification of the brain as the locale of information processing, methods of more directly interacting with that processing have been of interest to the research and medical communities. Humans have developed extremely efficient methods of information transfer: written language can communicate information at approximately 50 bits/s, and speech communicates information at approximately 39 bits/s, independent of the particular language. As platforms of experience have evolved, so have modes of communication in and out of the brain, many localized to the hands. The typewriter brought the keyboard, the desktop computer, the mouse, the laptop, the trackpad, the smartphone, and the touchscreen. In emerging platforms, especially augmented reality (AR) and virtual reality (VR), new methods of information transfer with the brain are required. Many, including major (economic) players like Meta, believe that a more direct link to nerves via a noninvasive neural interface may be the solution.

A History of Neural Interfaces

Artificial interface with the nervous system has been practiced since antiquity. Pedianos Dioscurides (76 CE) and Avicenna (150 CE) described the use of "electric rays" for treatment of seizures and chronic headaches. The relationship between electrical stimulus and neural activity was not made concrete until the experiments of Galvani and Volta (eighteenth century). Galvani, whose contrived experiments with frog legs were as entertaining to his contemporaries as they are today, established the correlation between electrical stimulation intensity and muscle contraction, saturation, and the refractory period. Volta, executing similar experiments, realized that electricity was a broader phenomenon, inventing the pile battery and establishing that electricity could be produced by non-biologic systems. A mapping between stimulation of the cortex and motor activity was established by Galvani's nephew, who produced contraction of facial muscles of the heads of recently beheaded criminals. Thereafter, cortical maps were produced for dogs. Efforts to understand brain damage began in earnest during the First World War. Stimulation of the occipital cortex during surgeries to remove debris from the brains of gunshot victims confirmed its relation to vision. During this period, stimulation began to be used as a method to localize ablation, further localizing functions in the brain, but the questionable ethical framework on which the practice was based led to its abandonment by the 1960s. However, the success of the implanted pacemaker renewed interest in neurostimulatory devices, eventually leading to the commercialization of "deep brain stimulation" for the treatment of Parkinson's. By the 1970s, the cochlear implant, the most successful neural interface, had been developed.

Approaches

Hardware to record and stimulate the nerves has been produced since the discovery of electricity and its near simultaneous discovery in biological systems. Neuroscientists for centuries drove research. Researchers interested in brain-machine interfaces have also made substantial contributions of late.

Recording Neural Signals

Invasive Approaches

Neuroscientists' interest in reading neural signals solely drove instrumentation research until the advent of neuroprosthetics. The earliest measurements were conducted at muscle-scale using galvanometers (1827). Measurement timescales were decreased with Bernstein's differential rheotome (1880) to tens of microseconds. The isolation of giant squid axons for neuroscience experimentation in 1939 (enabling Hodgkin and Huxley's seminal theoretic work on the neuron) accelerated the field without the requirement of an instrument for measuring smaller neurons.

Not all invasive technologies have been designed for the cortex, and most of the aforementioned techniques have been adopted for measurement of peripheral nerves. However, an assortment of invasive electrode designs specifically for interfacing with the peripheral nerves have also been introduced and implemented as treatment of medical conditions including migraines, fibromyalgia, tinnitus, and spinal pain, among others, though these techniques generally stimulate nerves rather than record neural data. Cuff electrodes, which surround the nerve of interest, are safer and have a longer useful life than intraneural electrodes, but are less specific, accessing only the nerve fibers on the surface of the bundle. Intraneural electrodes are more specific but produce greater physiological disruption.

Noninvasive Approaches

Noninvasive data collection can be performed by electrodes contacting the skin. Electromyography (EMG) and electroencephalography (EEG) record impulses of the muscles and the scalp, respectively. EMGs and EEGs provide low spatial resolution, collecting data from whole muscles or macroscopic patterns of activity in the brain. Because of challenges associated with performing invasive work and because invasive systems are unsuitable for widespread consumer adoption, I use a noninvasive system. EMGs are the singular noninvasive technology for measuring peripheral nerves, so EMGs I use. Fortunately, the burgeoning commercial interest in methods of improving AR and VR experiences has led to substantial activity in the space, CTRL Labs the most notable example.

CTRL Labs

In 2018 the American firm CTRL Labs developed a device on which my design is based. After purchasing the rights to the Myo Armband hardware from another firm, Thalmic Labs (which pivoted to the AR space before being acquired by Google in 2020), CTRL Labs demonstrated "algorithms for determination of hand movements and forces and real-time control from neuromuscular signals," allowing users to "create their own control schemes at run-time". CTRL Labs was quickly bought by Facebook, where development continues, if less publicly. A recent release by Facebook indicates that the firm is integrating haptic response as it proceeds with CTRL Labs's original ambitions of realizing a generalized control system from the EMG signals of the hand (including high speed, motionless typing). Though the complexity of CTRL Labs surpasses that of the initial design of my Thesis (CTRL Labs had a dedicated team of engineers and neuroscientists at work on their product), tools and off-the-shelf hardware have since progressed such that a simplified model is possible to produce. I do so.