Speaker phone only picking up my voice

Hello,

My iPhone is set so when I receive a call the audio comes through my hearing aids. Sometimes I need to switch it to speakerphone so other people can join in the conversation. When I do this the audio comes through the phone speaker so everyone can hear it, by the microphone (presumably in my hearing aids?) are only picking up my voice so the person on the call can only hear me.
Which settings do I need to change to sort this? I’ve spent a long time looking but nothing seems obvious.

Thank you.