As a teenager I spent many days swimming in the ocean surf with goggles and flippers. My ears were wet most of the day. I dried out when I came out of the water or went home at night. The water wasn’t chlorinated and to the best of my recollection, I never got an ear infection from this activity. I did get an ear infection as an even younger kid who stuffed cotton in his ears to avoid hearing his parents yakking downstairs at night, then went swimming, forgetting to remove the cotton, and having in the wet mass in my ears for many days (and nights) straight. So the degree of moisture exposure vs. drying out is very important as well as the bacteria/mold/microbes, etc., you happen to be exposed to.
Edit_Update: to go along with links in my previous post, here’s a quote out of the Wikipedia article on “Swimmer’s Ear:”
" The two factors that are required for external otitis to develop are (1) the presence of germs that can infect the skin and (2) impairments in the integrity of the skin of the ear canal that allow an infection to occur. If the skin is healthy and uninjured, only exposure to a high concentration of pathogens, such as submersion in a pond contaminated by sewage, is likely to set off an episode."’