Oofff, I was worried this might be the case. This is a combo, 3.5mm. It's repurposed from an audio plug. The outer sleeve is *usually* the shield, followed by a number of rings, and the pinouts of those are quite varied, especially on a custom system where they may as well be random.
You have at least one part of it working - the video shows up at all, so one part of the cable is soldered correctly, you would have to probe the rest. It's hard to advise you at this stage, you may have to splice the cable yourself and test different combinations until you find a working one. If you can't solder yourself, you can twist the cables and use a termination strip - picture quality will suffer, but it will work. If you don't want to splice for the sake of correct colour coding, you can try playing around with the colours - they're meaningless if you don't know the pinout, you can label them later.
RCA to 3.5mm jack have different standards depending on the implementation and the modder may or may not follow any of them. If I made the cable, I would've put the video on the tip, the audio channels on each of the two rings and the ground on the sleeve, but that's not a rule. If the system had individual RCA receptacles for each signal it wouldn't be a problem, but combo jacks like this are a different story - the cables either work or need to be reworked, it's always a gamble.
It's possible that the reason why your signal suffers is because the original plug had less rings than the current one and the audio signal is crossing over onto the sleeve when in the receptacle. 3.5mm jacks are either stereo or stereo with inline (usually a mic), so either 3 poles or 4 poles. Without seeing the pins of the receptacle you have to guess. If the system is still mono, it could be using a 3 pole jack, which would cause this behaviour (video, audio, ground). Try disconnecting just one of the audio channels and see if the issue persists.
If you look at the jacks, the first ring on a four-pole jack intersects with the sleeve of a 3-pole one - that would cause the audio signal to cross over to the shield, which in turn would pollute the video signal.