There's more to drifting JoyCons than first meets the eye...Nintendo changed the motherboard on the left Joy-Con at some point and so swapping out the analogue stick doesn't always cure the issue....
You can see on the RED joy-con below the motherboard extends up to the right of the analogue...
The ribbon cable spends too much time bent and warm, it's a common faulty. Replace the SR/SL ribbon, this part is less than £4.00 and an easy fix if you are confident with ribbon cables etc. Google SR/SL repair guide.
@Xdqwerty, Osu! Tatakae! Ouendan! is the Japanese version of the game, different settings/characters/songs but otherwise identical mechanics. I played that before I knew about Elite Beat Agents lol. Both fantastic games https://en.wikipedia.org/wiki/Osu!_Tatakae!_Ouendan