The swastika is actually a Buddhist, Hindu, and Jain symbol of luck. It has a deeper meaning than most people realize, but the western world only knows the swastika as the evil symbol
So in the west, yeah Nazi imagery is offensive as crap because it typically denotes one meaning: Nazis.
But if I was at a Buddhist temple in Kyoto, I wouldn't mind seeing a swastika chiseled into the thousand year old temple wall, unless of course it wasn't a part of the original structure and some dummy was being a vandal.