I'd have to say religious fanatics have the ability to be the most harmful.
Racists are terrible people, but they're usually working off their own experiences in life. Maybe they were robbed by a black person, or maybe a white person treated them like crap. That formed their perspectives of who they feel they can't trust. Those types of thoughts can be reversed if given the opportunity.
Religious fanatics aren't really thinking for themselves. Their own life experiences don't come into play on how they're supposed to think of others. You listen to the religious leader, and that's all. If a pastor says to hate ________, you will hate them because you were told to. It doesn't matter if you have friends who are __________, and they've been nice to you all your life. You are commanded to hate them. If you don't, hell awaits you.
Racism can also have gray areas. If I were to talk to a racist white guy, and asked "What do you think of this black guy who's a doctor and in MENSA?". They may break from their usual attacks and say "Well, they're one of the rare good ones". It's not much, but it's admittance that they can think highly of that type of person. Whereas religious fanatics can be asked about an atheist who's done charity work all their life, and they'll still come to the conclusion that it doesn't matter. They're still going to hell in their minds.
That being said, if I have the choice of just talking to a single racist or a single religious fanatic, nine times out of ten, I'd prefer to talk to the fanatic. But the question is what is worse for the world as a whole. Anger is bad, but not allowing people to think critically is much more dangerous overall in a society.