A benefit-risk assessment of ofloxacin otic solution in ear infection. - PubMed - NCBI says that there is no ototoxicity associated with that drug. Ototoxicity means toxic to the ear.
I think you’re much more likely to experience permanent hearing loss by not taking it. Hearing loss from a ruptured ear drum is no surprise. It’s like when you poke a hole in a drum skin. It doesn’t vibrate properly. If your ear drum doesn’t vibrate properly it doesn’t transmit sounds properly to the rest of your ear. The ENT said it would heal, so unless you have a sudden change in symptoms, I’d wait for it to heal.
I’m not a doctor but hopefully that takes the edge off your panic until your next appointment. If it doesn’t, go back and see your doctor so he/she can address your worries.