This is no reason to use the term transphobic to hush others peoples freedom of speech, make them conform to your standards or otherwise try to control them
I completely agree with you, nobody should use words or names to censor other people and their beliefs/observations.
Though, it is also okay for them to assume things about you based on their own beliefs, as long as they aren't trying to censor you and/or convert/control/disown you based on misassumptions and/or disagreeing with your views/beliefs.
Also, I'm glad that you take a chance to get to know these people individually even if you don't completely agree with their lifestyle choices and/or group affiliation (sorry if those are incorrect assumptions), that's very good of you to do.
Nobody should use words to disown a person just because they don't agree with them, this is something that happens a lot between multiple groups.