My psychophysics as a bilateral cochlear implant user

I am deeply grateful for your words. It has been just over a year since I received my cochlear implant. Thanks to you, I learned for the first time today that this methodology is not yet mature enough to be used in clinical practice. I started this because I wanted to understand the sounds of the world I now hear, but there is still so much I don't know. That is why I am actively participating in and collaborating on cochlear implant research in Korea as much as I can. I also sincerely thank you for your dedication to advancing science.

Is there any research into other ways to 'get sound into the brain', generally speaking? If so: some pointers would be much appreciated.

After being hard-hearing since early childhood (bad living conditions -> tuberculosis -> ototoxic antibiotic, streptomycine iirc), my mum had a CI for a # of years. But due to infections, it got removed at some point. Since then I've always wondered if there's other routes to get sound into the brain. Say, using vibrations on the skin. Or an electrode-stuffed 'helmet' of sorts.

Even if temporary, it can be a life-changing experience for someone who's never experienced sound.

Any research in that direction?

Thanks for posting. If you have time to write some more thoughts on those limits and individual strategies I'm sure it would be interesting to many of us here.

I have history of hearing losss and a more recent episode of bilateral SSHL post viral infection with relatively decent recovery. I am left me with deficient sloping loss in the high frequencies 50-60 db. I wonder if the day will come when I will need CI. So far I can manage but since medically no doc is able to point out the cause of the loss I’m starting to familiarize myself with them. I hope progress is being made on CI front and the regenerative one as well

I was born with normal hearing and acquired language while I still had it. But now, no matter how hard I try to think back, I can’t remember what things sounded like back then. I know that people’s voices sounded clear, but at this point, that’s merely a semantic memory. The sound itself, the actual sensory experience of it, is something I can’t fully recall.

    > actually able to play a sound in their head without humming

They remember the narrative of the order of events but not the subjective experience of normal hearing. Quite common for all kinds of experiences, because stories are culturally held.

They might be speaking from the memories of their family members rather than their own.

I guess it's childhood amnesia[1].

[1] https://en.wikipedia.org/wiki/Childhood_amnesia

They might have started losing it at an early stage of their childhood.

My wife lost hearing before she turned 2. There's plenty of space between being born and forming permanent memories.

It’s AI slop.

The essay is a little odd because the author mentioned that she was born with normal hearing.

Supposedly, the deaf-blind Helen Keller learned to speak clearly in the same way.

The vocal vibrate primarily because of the Bernoulli effect and the over pressure of the lungs. Just vibrations alone, called the voice sources, carries almost none of the information that we use in speech. That information arises as the pressure wave goes up through the larynx tube and into the mouth (or up the nose in the case of nasals). It is possible to measure just the voice source by means of a laryngograph.

> Voicemod

> Miku

Interestingly, the same researcher in charge of the VOCALOID software is now in charge of research for Voicemod.

This is an intriguing proposal. In my experience, however, Hatsune Miku's pronunciation is not clearly intelligible upon first hearing one of her songs. The words become distinct only when I read along with the romanized lyrics. With repeated exposure to the same song, I am eventually able to understand it without the lyrics; however, I would argue that this reflects top-down prediction grounded in memory rather than any increase in the clarity of the sound itself. This is analogous to the phenomenon of sine-wave speech, in which sounds initially perceived as meaningless whistles or electronic tones are suddenly heard as clear, intelligible speech once the listener has been shown what is being said.

[dead]

Yes, I have. For a whole year, I asked about it at every mapping session. But no matter how many times my cochlear implant was remapped, there wasn't much improvement. I still don't know whether the cause lies in my auditory nerve (peripheral) or in how my brain processes sound (central), or something else entirely. I told my audiologist that speech sounds blurred, and I also brought it up with my doctor, but nothing really improved.

Pretty much all of the difficulty arises from the transliteration of Greek ψ, υ, χ, and φ, and then inconsistent pronunciations of these letters themselves in loanwords (including pronouncing υ as both /aɪ/ and /ɪ/)!

That's not to say that it's not very tricky, just that the difficulty kind of arises from one core reason.

Edit: OK, one s as /s/ after the c and another s as /z/ between vowels is also tricky and doesn't have to do with Greek.

Could you try taking a musical track and doing a low-pass filter to remove anything above 100 hz? If it doesn't sound about the same to you, then that is proof that you are hearing and processing some sound above that 70 hz limit. The inability to distinguish speech sounds more like trouble with higher frequencies (which help discriminate between different syllables) than with frequencies in the 100-400 hz range typical of human speaking f0s - because the latter are also vital to what tends to make music sound full and rich.

A sort of anti HN comment to say but very interesting reading, I don’t know if I have answers to any of your questions but thank you for sharing, I will read more and consider it.

Pretty fascinating read up!

I always told myself I would try and do some fun things with my cochlear implant (Med-El Sonnet) but I never did so reading this was like dusting off some old thoughts of mine.

I was the same, I was born relatively hearing, but started to lose it rapidly - due to a mitochondrial disease (mutation A7445G) which is very rare, but is in a few families around the world. I wore hearing aids for most of my childhood and then got the implant when I was mid 20s.

I've heard a lot of your complaints from others. My sister for example, didn't have a great journey, and even today she reckons everything is still robotic sounding. She however went with a Cochlear Nucleus system, and I was the only one in 30+ people in extended family to go with Med-El (because I liked their tech better) and it's always sounded crisp and clear to me.

I'd have liked to have gotten two, but I was only funded for one - plays havoc with directional hearing. You could have 50 people turn towards a sound, and then there's me turning the opposite direction! :)

I'd be really keen to know if you have any other experiments planned.

Thanks for the writeup! It's fascinating that you can hook yourself up to a PC and get music played directly into your implant. I wonder what would happen if you shift all frequencies in a song (or PC microphone capture) into a much smaller range, proportinally? And then test some target ranges. Or you could slow down a song but keep all the frequencies?

AI can write such programs quickly.

Have you tried listening to chiptune, MIDI, or electronica music? I am congenitally hearing-impaired myself and also enjoyed Vocaloid, although I had never wondered about a connection before. I've sometimes thought that when Vocaloid producers don't try for realism but go for simpler easier 'purer' Vocaloid voices, it sounded like chiptune/MIDI/electronic. So those might be an interesting musical control group.

Thank you for sharing this. It is all very interesting, and I hope you are able to find out more and experience even more sounds with your implants you do. I would be very interested in being able to experience the tests myself. There are a lot of descriptions of what is being done with the audio, but I have no real intuition as to what any of it actually means. The github repo seems to only have the data in it, and not the tests themselves. Though I could certainly be missing them as well. Including the tracks, or code used the generate the tracks, would be nice.

This was beautiful to read. Thank you for writing it! Don’t worry about it reading like slop - your voice comes through very distinctly to me.

You mention a lot of ethics concerns; I am totally uneducated on medical ethics but I think those are pretty important questions. It doesn’t sound like you’ve had a lot of support from this startup.

I might suggest to you that you publish this on a website - GitHub pages comes to mind as something Claude or codex could do for you easily - it would let you segment this information in a way that someone who wanted to help with the medical questions could get right to that stuff, and someone who wanted to help with ethics could also jump to what they were interested in.

Either way, thank you for the write up! Your curious and open attitude is lovely.

I'm glad to hear you discuss listening to music. There's no doubt it will sound different to you, but still well worth enjoying. My best wishes to you for an interesting, and I hope exciting time in your life.

Wow what a thorough analysis, thanks! Admittedly, I'm not sure how to translate the numbers to audio. AI helped me find these websites:

https://hzgenerator.com/sweep-generator

https://onlinetonegenerator.com/frequency-sweep-generator.ht...

A looping 10 to 70 Hz sweep got the best result with my 20+ year old Apple sound sticks because they have a subwoofer.

With typical hearing, it sounds like almost a dozen distinct frequencies to me. Higher frequencies are just a matter of degree, they aren't qualitatively different IMHO. But I'm not musically inclined, and only have synesthesia when I sort of see color in various gray patterns on the computer screen, which look more blue or whatever sometimes, so I'm sure someone will disagree with me because they sense feelings in audio that I don't.

I'd bet that improvements in technology over the next few years (especially with the help of AI) might get the upper end to perhaps 100 Hz or more, where it begins to sound more like a tone than a hum. Loosely, a beat (like a drum) sounds like perhaps 10 to 20 Hz. A hum sounds like 60 Hz (like electricity). Also I wonder if a cochlear implant can hear below 10 Hz because it's electrical stimulation instead of physical, to maybe hear frequencies that typical hearing can't? But I don't know.

I'm fascinated by this stuff, because sometime in the next 5-10 years we'll get direct digital signal to nerve implants. That could allow people to hear above 20 kHz (like a dog) or even restore smell in people who have had it diminished due to illness or injury.

Many people can see ultraviolet light after getting cataract surgery, which supposedly looks like a blacklight but more blue because there is a little rise in the red visual receptor the overlaps blue to make violet. So technically the blue receptor ends up being a violet receptor because red isn't removed normally. Which means that violet and purple are different colors, because the visual cortex makes sense of them differently at opposite ends of the spectrum based on how much green is present. Of course all of those distinctions may go away when someday the visual nerves can be stimulated directly with an implant and make color more of a continuous spectrum like sound, rather than having distinct color bands like in a rainbow.

Subjectively, I've noticed when I wake up in the morning that the ceiling has grey and yellow splotches, because we have fewer blue receptors in the fovea at the center of the eye, and they haven't adjusted yet. Also at dusk, the rods begin to sense more, so the world fades from color to cyan and then gray. I feel like similar things might happen with audio, because I feel like I can hear better when I'm camping, especially in the morning, but I don't know why.

There are interesting experiments where they put goggles on chickens to flip the world upside down. After 2 weeks, the chickens saw normally again due to neuroplasticity. Then when they took the goggles off, the chickens walked goofy until their brains readjusted.

Also I've read that lion's mane mushrooms regrow neural connections and may even increase neuron synthesis. I've had neurological injuries due living a rough and tumble life that seemed to have healed somewhat after drinking mushroom coffee for a few weeks. Also magic mushrooms scramble nerve connections and let them reestablish themselves, sort of like starting over with a blank slate in neural networks. Mushroom medicine research has been blocked by a century of ignorant legislation, so people have missed out on spiritual healing that strays outside of what science can explain. I think of a brain or neural net as an antenna that resonates with the consciousness field, which is some sort of 5D-like field that transcends our normal 4D spacetime (otherwise it would be a frozen crystal with no observer or passage of time). When our brains get rewired after learning/trauma/injury/illness/healing, our attention shifts to different things, which alters our subjective experience. Meaning that in a very real sense we perceive the world differently and/or feel like we've shifted to a different (often better) timeline. Plant medicines and spiritual practices may allow changes to subjective experience that science can't yet talk about.

Anyway, not sure what point I'm trying to make here with these breadcrumbs, other than that the future isn't written yet, and in fact there are infinite futures, which I believe free will allows us to choose. So there is always hope and a reason to keep going.

This is interesting and I'm going to write out my thoughts for you here. Just some impressions.

Right off the bat after skimming this I'm caught up with a detail that you confirm multiple times - that you can tell literally nothing apart given any input over 70 hz. You have had this measured a few times and you definitely say that nothing is coming across above 70 hz.

Yet you are trying to hear all sorts of things that have most all of their information content above 70hz.

The Hatsune Miku music? You're hearing the bass and the kick drums and maybe some subharmonics generated in the reverberation of the higher stuff. But that is a tiny fraction of the music, and, I hate to tell you this, but that computer girl singing is all way, waaay up higher than 70hz. The beautiful sounds you are getting across are the cool synth basses of all that music. It's very modern, punchy music and the bass of that type of electronic music is very different from other classical music so whatever modulation your implants are doing to it from 70 hz down likely sounds very cool compared to a lot of other music, for sure.

But, stop- you're getting nothing by noise and wind sounds with no information differentiation over 70hz because your brain isn't demodulating that information, maybe due to something about how the implant was... implanted... or just your specific condition, who knows, but what you can do is work in the band you have.

You mention not getting human voices because the fundamentals of human voices are more than 2x where your cutoff is.

Have you tried using a program like Audacity - or - Get claude to write a script for you that takes in MP3's of modern music and slows them down 300-400 %. The music will be very slow, but it will shift instruments from higher up (pianos, horns, people's signing) down into the 20-70 hz range, and you will maybe hear patterns that are new. Did no one in the study post-op sit with you and go thru this kind of thing? You have all these external trials listed but most all of it is moot if everything above 70hz is missing and you aren't able to modify the modulator hardware.

Note that the type of rhythms in the bass of the bass synthesizers of that Hatsune Miku tracks are different that the more complex rhythms and melodies played by instruments higher up (like pianos and lead synths). (This is true for most all music genres...) Shifting those higher up things down in frequency, even if that means they play back slower, may give you an different musical experience.

Continuously pegging your implant modulation hardware with frequency content above 70hz attempting to discover something about how it interacts with your brain when you continuously get nothing but broadband noise over 70hz seems like a dead end exercise.

I mean hell, have you tried putting a low pass filter to block out on everything in the music/microphone input above 70hz just to make sure that information isn't somehow overloading the way the implant works with your brain, and maybe just causing confusion.

Lastly it sounds like you could benefit from talking with someone with a real audio / digital signals processing / electrical engineering background that can inspect your hardware directly and make suggestions because educating yourself with just GPT runs might be spinning you in circles.

For instance... you mention things like " The logic of my test apps, which generate the test sounds and move the staircases, was checked with the Lean 4 formal verifier,"

Man... this has nothing to do with needing a 'Lean 4 formal verifier' to check your app or whatever. You need someone to sit with you and your damn hardware and work with some audio input. You need the someone that setup the original implant modular to explain the hardware mapping and filtering options to you and see if you can go back and change how that works, to get more options. You're getting some wildly mixed signals (bad pun) about how all this works, I think.

You clearly have the hacker mindset (compliment). I hope your investigations are fruitful, for yourself and others who may learn from your experiences!

This did not read like slop and was fascinating to look through. I hope you and the ENT doctor in the thread can have a nice conversation after this.

>Those of you who hear the world in three dimensions would know the answer, but am I hearing a flat plane?

You typically should feel sound as a 3d point, it maps roughly to the same sensory system that maps light or touch direction. Loud noise would feel as directional and material as bright sun or strong poke to a head. (Brain does this mapping transparently based on the small difference of time it takes the sound to reach your ears.) Sometimes the sound is not clear, you would either hear from a vague direction or with no direction channel provided from your brain at all.

The culture here is pretty split. Honestly, there's a sizeable contingent that will likely flame/flag you for mere use of AI. Try not to take it too hard; they don't represent everyone and I at least thought your post was a valuable addition here.

Please don't take HN threads on generic tangents [1] and especially not snarky ones. This isn't a response to the article - it's simply repeating a pre-existing association to a recognized name.

We're trying for more interesting conversation here and that requires specifics, not generics.

Another way of putting this is that we want reflective comments rather than reflexive ones [2].

[1] https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

[2] https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor....

Can't say I've had better experiences with GE appliances but their jet engines seem fine.

I also can't imagine what you thought this comment was contributing to the conversation but that's not my problem I guess.