The problem people have is exhibition of a cock and balls in a female only privacy space, or someone with a cock and balls committing voyeurism in a female only privacy place,
or the exhibition of a vagina in a male only privacy space, or someone with a vagina committing voyeurism in a male only privacy place.
These concerns are independent about whether or not the person is transgender, although obviously the issue is that people with these concerns do not wish to make exceptions for the transgender population unless they have had sex type reassignment surgeries.
Most crimes of exhibitionism occur not in bathrooms, but in public transportation areas, or parks, etc.
However, most crimes of voyeurism do occur in bathrooms, and changing rooms, although these days those crimes are just often committed with the assistance of electronic devices (cameras hidden or used in those areas).
None of these concerns are equivalent to malice regarding the transgender population, or the desire that the transgender population not be here, in most cases.
However, it is true, that mainstream Islam is an exception, and some extreme Christian groups are an exception, but those groups would take more exception to the fact that I am a public Atheist than they care about people being transgender.
If you and I went to Saudi, we would both likely be executed if I was public about being Atheist, and you were public as being transgender.
If we both went to Iran, I would be executed for being a public Atheist, and Iran would require that you has a gender reassignment surgery, and then Iran would be fine with you after that.
As far as an Extremist Christian group, they would consider my Atheism the worse sin, but if they were violent in nature, yeah, they might throw rocks at both of us, which is ironically exactly what Christ allegedly said not to do, but those groups and individuals do exist.
So, the central issue is, everybody wants to feel comfortable in a changing space.
Ironically, I have traveled to plenty of countries where the changing rooms and spas are nude, and at least on certain days, coed.
So, cultural acceptance of coed nudity or the reverse, is also a factor.