High ping when others are hosting servers

[Eon-Rider]

This is directly copied from a thread I posted in the Steam forums.

When I play games such as Team Fortress 2 I generally get pings that average around 30. The same goes if I play L4D2. In addition to this, I regularly host SRCDS for both games with other players having low ping. Ports for both games are forwarded.

It seems that when someone hosts a game via listen server in L4D2, for example, my ping averages at around 400 while others (non-hosts) will still remain with low ping. I find this consistent in other games such as Trine 2 and Magicka. I always seem to be the only one with high ping.

I had this problem for a long time now and I'd really get this issue resolved as Castle Crashers, the latest game I have problems with slows down the connection of everyone to the slowest person.

UPDATE: It works fine with Hamachi.