You can irritate or inflame your ear canal, too. I did that with a mold that had fit pretty well until then. If your ear canal swells up a bit, you can get into a vicious cycle of irritating it more, more swelling, and worse fit.
I reduced my wearing time to help the irritation go away. Since I am a psoriatic and have a range of different strength steroid creams or ointments, I used a little desonide ointment, about the mildest topical steroid there is, safe to apply to facial tissues, to lubricate the mold before insertion. The idea was that the steroid would gradually suppress the inflammation. I normally use a bit of olive oil as lubricant out of a dropper bottle, as suggested by my audi.
You may just have a bad-fitting mold, but in my case, I had irritated my ear canal, which caused it to swell, so the mold no longer fit as it should.