And I believe @Rand will find this answer useful: you can ask your fitter to change TV connector program to not connect automatically when it detects the gadget but on manual (there are two manual options, I chose the one without any sounds).
Then when you want to listen, you either click the button on HA to switch to that program, or for lazy of us, you grab the phone and select it from there.
I usually know when I want to watch TV and have phone near me (or HA button), however I’m also lazy to stand up and turn on/off TV connector, as I saw many people do as a solution. So I share mine
So basically, remote is there, just not for TV connector but your HAs.