I also have the Phonak TV connector. It’s is optically connected straight out of the Tv. It works perfectly with no lag at all (as a musician I am also very sensitive to audio latency)
I also have a sound bar connected via HDMI which works completely independently so my wife has the volume she prefers.
So no complaints at all now, it all just works automagically … but the previous methods to connect HAs to the same system by other methods, I.e. different makes and models of hearing aids, ConnectClip, BT transmitters, headphones etc were miserable - so much so that I stopped watching tv almost completely for the last few years. Now I can hear every word clearly, at the volume I prefer without annoying the wife, and with lip-sync. It has been a game changer for sure!
Unfortunately, with the wide variety of TVs and audio equipment out there your mileage may vary. I have a great Audiologist who asked their Phonak rep for a loaner Tv adapter to try out. Maybe you can too?