Fine-Tuning: Improving Music by Adding Pitch
On March 7th, the New York Times published a sunny review of cochlear implants. The article— Risks Fall, Hopes Rise for Hearing Implants— boasted decreased woes and rising effectiveness for cochlear implant users. Times’ science writer Nicholas Wade said that the story of cochlear implants in the past 20 years “has been an increasing, even surprising, success.” Reconfigured implant models have meant that far fewer patients are contracting meningitis, an infection previously prevalent in implant wearers. In fact, according to a study begun in September, 2002, only 1 out of 3,436 implanted children has had this infection. That’s because manufacters have redesigned implants with smarter seals that minimalize bacterial growth near the brain. Meanwhile, the devices are helping children as young as 1 to develop normal language by the time they are 3 or 4, something never before seen in profoundly deaf youth. Yet amid accolades for implants, those who use them, including 11,000 Americans under the age of 18, are not entirely content with their products. One complaint in particular still lingers: the wonders of Beethoven, Bach, and the Beatles elude implant users. This group can perceive many sounds, including speech, but they cannot clearly hear music.
Music is an especially rich form of sound. In general, sound is introduced into air by a vibrating object, like a guitar string, for example. The vibrating object disturbs the air, or any other medium through which it moves, causing the particles of that medium to move back and forth at a particular frequency. This frequency, when perceived by the brain, is called pitch. Like other sounds, music is composed of pitch, which might be compared to a note, and it is also composed of temporal information, the change in a pitch’s volume over time. Unlike speech, however, music appreciation relies more heavily on the former—on pitch. The range of pitches needed to convey musical sounds is far wider than the range needed to convey frequencies in the human voice. And it is this range that is so severely compromised in implant users, making their music listening experience less colorful.
To grasp what an implant patient might hear, try to imagine your favorite song played entirely with a single note, so that the song changes merely based on that note’s volume. Or, equally unpleasant, consider a song played with only eight notes, when it requires hundreds more. While people who hear normally can perceive pitches numbering into the thousands, people with implants are limited to little more than 20. Their “rainbow” of sound is reduced. It’s as if they were seeing portratis of sunsets painted in yellow and black when such natural wonders would normally require oranges and auburns, golds and yellows, and even hues of red, to be truly appreciated.
These scenarios are analagous to those faced by implant users, and they occur for one reason: people with these devices can only hear as many notes as they have electrodes. Electrodes are the means through which cochlear implants resolve frequencies from the air into information that the brain recognizes as sound. Each electrode—a wire capable of receiving electrical stimuli like that delivered by the speech processer— responds to a set of frequencies yet conveys only a single pitch to the brain. To date, the only way to increase the number of pitches that implants can convey is to increase the number of electrodes.
But unlike the microscopic hair cells that perform a simliar role in a functional ear, electrodes— which signal the hearing nerves and send sounds to the brain—cannot be present in infitnite numbers. Even the most advanced implant available today, Cochlear’s Nucleus Freedom 22, has but 22 of them. That’s because there isn’t much room within the the depths of the inner ear, where the electrode spray nestles, curled about the cochlea like wiry fingers, and any increase in electrode number would require reducing the space between these individual “fingers”. This reduction in space prevents electrodes from stimulating independent sets of nerves. If there are 25 nerves spread along a certain segment of the cochlea, for example, and only two electrodes, electrode 1 can stimulate nerves 1-10 while electrode 2 stimulates 11-20; the implant will register two clear notes. If 2 more electrodes are added however, so that there are 4 wires within the same small cochlear space, the nerve populations being stimulated by those electrodes will likely overlap: electrode 1 might stimulate nerves 1-10, 2 might stimulate 5-14, 3 would stimulate 11-20, and so on, in an overlapping sequence. Since the nerve sets are triggered by more than one signal, they are more frequently active, and so they must also recover from activation more often. This recovery period—known as refractory mode—is a time when nerves are less responsive to stimuli, muddling their ability to send sound to the brain.
A simple solution might seem to be to not use neighboring electrodes, but rather to use 1 and 3 instead of 1 and 2, for example, in hopes of hitting different nerve sets and generating two distinct pitches. However, this assumes that you know which nerves an electrode stimulates, and scientists do not yet know the electrode to nerve pattern.
Since increasing electrode number to improve hearing is not an option, then, implant recipients have had to rely on only 22 separate pitch perceptions to convey music. There are infinitely more than 22 notes in the spectrum of music. A piano, for example, has 88 keys, each representing a different note. And so the cochlear implant seem like a straitjacket; after hearing a sound, the device is forced to choose the pitch that comes closest to that sound’s actual pitch. With only 22 pitches to choose from, though, rarely do the chosen and actual pitches match.
Just when it would seem that electrodes would doom the lives of the deaf to a dim world of 22 notes, however, scientists are proving otherwise. Another way—not yet tried— exists to provide more pitch options, and Leslie Collins, an electrical engineer at Duke University, is being funded by both the NIH and implant manufacturer Cochlear Corporation to study this alternative. “It’s called fine-tuning,” she said, “and a lot of people don’t think it will work, but we do.”
Fine-tuning works not by increasing the number of electrodes, but rather, by increasing the number of available frequencies within each electrode. Right now, implants are designed to allow deaf people to hear speech, but fine tuning will manipulate the available electrodes in the existing versions so that the implants can respond to a broader range of frequencies—frequencies beyond those prominent in the human voice—allowing people to more clearly hear music.
That is not to say that cochlear implants perform a small feat by making 22 notes known to the brain. This process does permit language understanding, and it does so by stimulating electrodes in a fashion that mimicks the way the frequency of a pitch stimulates the cochlea. For each pitch that occurs, a different location within the cochlea resonates. Low pitches affect parts of the cochlea deep within the ear. High pitches stimulate parts father out. In an implant, this is analagous to specific pitches vibrating specific electrodes. Pitches vibrate certain electrodes based on work done by the speech processor, where all the software is located, which sifts through the frequencies that make up a sound and then stimulates electrodes accordingly.
But for each pitch that occurs, something else that effects hearing happens, too. In the hearing ear, not only is a certain location of the cochlea stimulated, but it is stimulated at a pulse, so many beats per second, corresponding to the frequencies in the pitch. This means that for normal listeners, stimulation is a function of both pulse and location. Implants do not take advantage of the method of pulse to affect sound, however. Instead, they use one common rate.
A few years ago, in 2002, biomedical engineer Fan-Gang Zeng at University of California at Irvine experimented with pulse. He used cochlear implants to define the relative contribution of location codes versus pulse codes in pitch perception. Because the two codes vary together as the frequency of sound varies, their relative contribution to pitch has been a topic of continuous debate for over 100 years. It still remains unsettled, however, and cochlear implants provide the perfect opportunity for understanding pitch components because, with them, location can be studied independently of pulse, and vice versa. That’s what Zeng did. He isolated pulse and tested its affect on pitch by changing the stimulus frequency—and using multiple frequencies— on a single electrode pair. He performed tests in people with implants and found that they could detect differences in pitch for frequencies up to roughly 300Hz. These results suggested that 300Hz is the upper boundary of the pulse code for pitch perception. Collins, inspired by Zeng’s initial look into fine-tuning, is investigating whether pulse rate can be added to gain even more benefit, so that ultimately, implant wearers will have more than 22 options.
“Electrodes offer people one pitch. Just one pitch per electrode.” Collins said, “But we want to offer them 2, 4, or even 8, or maybe more than that someday, all within one wire.”
Collins’ goal is to determine the optimal number of frequencies to add to a single electrode to improve music perception. “By designing each channel such that it provides multiple pitch sensations,” Collins said, “we will allow the implant to provide a more accurate pitch estimate. ” With a cochlear implant however, there’s never a guarantee that a frequency vibrating a certain electrode will register a certain note. “Even with rate there to help, you can never assume that kind of accuracy,” Collins said. Furthermore, even with pulse, today’s implant users can only discern frequencies of up to 300 Hz.
Applying pulse does increase pitch options though. “In one electrode, you get a different pitch out of the model for every rate used, up to about 500 pulses per second. You get fine-tuning.” At 50 pp, you may have a C, and then as you increase rate you will get some progression of notes. “You cannot claim you will get c# then d then d# etc,” said Collins “because you cannot necessarily tell 50 pps from 51 pps and so you cannot micromanage the process. There are some discrete rates that you can tell apart— maybe 50, 60, 70, 90, 110, 150 etc—and each of those rates on the same electrode will have a different pitch.” Changing pulses per second doesn’t change the location of the nerve that is stimulated by the electrode. Rather, the same set of nerves is stimulated but it fires at different rate. When it does so, the brain recognizes a new note, depending on pulse.
Collins and her colleagues are not physically working with implants, but with models, and so the frequencies they’re considering are not technically added to electrodes. Instead, they are processed in “channels,” electrical abstractions which exist within the acoustic models on computers in Duke’s labs. Collins’ models process speech in the same manner as a cochlear implant would. However, instead of stimulating the electrodes of an implant user, the speech sound bites processed by these models are presented acoustically to normal-hearing listeners, sitting in labs. “The research lives and dies by the people that are willing to sit here and listen and participate in these sometimes excruciatingly boring experiments,” Collins said. Those people assess quality of speech emering from the computers They assess sounds an implant might convey in a quiet environment, and those from a noisy environment.
In the typical model—one lacking pulse stimulus—one electrode might be selected to represent hundreds of frequencies. The lowest-frequency electrode, for example, might represent 250-350 Hz, and it would be stimulated in an identical manner, in terms of location and pulse, for any frequency within that range. It’s like a cell phone ringing the same ring every number dialing it beginning in “123”. Adding pulse is like adding a caller ID system, however; the phone rings differently for each incoming number, and the phone owner distinguishes these different sounds. When pulse is added, the “ring” changes. If a tone of 250 Hz occurrs, no longer is the pitch percept the same as it would be at 350Hz. Rather, 250 Hz would be associated with a lower pulse rate which would act to lower the pitch percept, “fine-tuning” the sound of the note. And if a tone of 350 Hz occurred, the pulse rate would be increased accordingly.
Collins used pulse models to test 3 levels of frequency tuning. She sought to determine the minimal amount of tuning necessary, or the minimum amount of frequencies to provide, to substantially improve hearing in implant patients.. Collins was surprised to discover that minimal tuning—just 2 frequencies per channel, compared to the standard of 1— was all that was necessary for normal-hearing subjects to notice significant improvement.
Tests are still on-going. Neither Collins nor Zeng have sold their ideas yet, nor have they been implemented in the marketplace. But Cochlear Corporation is funding both of their work, and its website boasts fine-tuning as one of its newest initiatives.
Collins admitted that the idea to focus her lab’s research specifically on improving music listening wasn’t her idea: “I have a couple of students who are extremely enthusiastic about applying this research into fine-tuning to music perception. They are essentially the driving force behind the music work – we had originally been continuing to work in speech but one student in particular is a musician and has an incredible drive to provide better music quality to implant subjects.”
She’s very excited about the implications—and some of the results gathered so far—of her research in fine-tuning, however. “I always tell a story about my Ph.D. research where I was testing out a hypothesis that if I changed the way people’s speech processors were tuned, that I could make them understand speech better. We had done a ton of experiments trying to get the basic data, then we had started to bring in the subjects to test their speech recognition based on what we had measured experimentally. We measured people’s speech recognition using their standard speech processing algorithm, then reprogrammed their speech processor based on the data and the hypothesis. I turned on the new processor for one of the subjects, and before I could even get one subject into the sound proof booth to measure his speech recognition, he turned to me and with this amazing light in his eyes said “Oh my gosh, this sounds so much better!” When we measured his ability to understand sentences, his recognition had gone up by something like 50%. I was basically hooked from that day forward.” Collins has been studying improving sound for implant patients ever since. “The most exciting thing for me,” she said, “is thinking that you might actually be able to help these folks understand speech and appreciate music just a little better.”
Meanwhile, Collins is aware that some patients could be receptive to this new technology, but others may not be. They’ve found variability between patients in their studies at Duke. “No one knows what it is based on,” she said. “My sense with implant subjects is that it is always variable – some do great already, and you can’t help them. Some do very poorly already, and you can’t help them. This is probably going to be the most effective for the middle of the road people, although it may improve the ‘quality’ of speech and music even for those that are already doing well. I guess we will just have to wait and see.”
.MGW.
