I experienced the exact same problem and it totally turned me off listening to music. After a few months I decided to try and solve it myself.
Using the Target software and experimenting with extreme settings I finally found it was the speech setting of the streaming option. Apparently the streaming of music starts of with the setting for music but changes to speech with the kind of music I was listening to. It would switch between those 2 settings depending parts within an song.
What solved it for me was completely wiping the fine tuning settings for streaming and setting it up again from scratch. Getting the speech and music settings as close together as possible (very difficult indeed) I am now almost unable to detect the changes between the 2 settings when listening to music. But most importantly I got rid of the extreme shift to one side.