The ‘speech banana’ is just the average of the likely energy for speech sounds.
If you look on a real ear measurement, speech usually contains multiple complex peaks and resonances for the main sounds. However some (especially vowel sounds) have a predominantly single peak output. So, the ‘banana’ is just the general area where speech hits.
Lower (Male) voices skew the average energy towards of the left, while higher voices move the whole average right. Those individual phonemes are a bit misleading as they can’t represent either the actual pitch or the complexity of given sound. This may also vary for different accents and languages too.