In an 8-bit (1 byte) unsigned value the largest number you can represent is 1111 1111 or 255 (2^8 - 1) and the lowest number is obviously 0.
It's possible that once you get to the 4th badge, the number overflows to somewhere around 255 which would explain why it seemingly works without...