I went through this, it’s not uncommon. Even in a quiet room there’s some small amount of background noise, fridge hum, traffic or wind outside. HA are setup to amplify (gain) sound levels by a certain amount, and the gain levels are in tiers… quiet, medium, loud.
What your describing is a textbook case of the gain for the quiet sounds is brining up those background sounds to a level that’s noticable and intrusive. Simply adjust the gain on the quiet sounds, a little reduction can take the hiss down to a point your brain isn’t noticing it, but there’s still enough gain to help overall with comprehension.