This looks like the capture card outputs RGB while the monitor expects YUV or something like that. The monitor might only be able to accept RGB signals at certain resolutions or the card itself is missinterpreting the signal of the console (like console outputs YUV but the card thinks it's RGB (again, might be resolution dependant), so it tells the monitor it's a RGB signal).
Anyhow, this clearly is a issue of the capture card, so try another brand.
Anyhow, this clearly is a issue of the capture card, so try another brand.