The production of silicone domes is done by injection moulding, this leaves a surface that is molecularly complete and largely defect free.
Other moulds are produced by 3D printing, forms of casting/hand finishing and using evaporate products, which leaves micro-voids on the product surface and also potentially remnants of the catalyst reaction. Finishing is usually with dipped/UV cured acrylic lacquer, which can be (rarely) hand polished.
Something in one of these processes might be causing a reaction, or simply the micro abrasions/voids in the otherwise perfect surface allow bacterial/fungal production and byproducts to build and irritate the skin. Some people are allergic to the plastics, some to their own bacterial/fungal byproducts, some to both.