Just did another test involving windowing, and it seems the problematic areas I mentioned in prior posts revolve around non-single-window cases. when I forced the PPU_ComputeSingleWindow function to ignore the "if" case, the window effect shown by @dronesplitter in the MMX screenshot was not there, but the problematic windowing I mentioned was still there.
edit:
My mistake. It wasn't so much the non-single-window cases, but that it defaulted to the non-single-window case when it shouldn't have. I made a few changes in ppu.c that corrected this (which fixes both hardware and software rendering of the effect). Forgive me,
StapleButter, if you were planning to use the WinLogic array for something else, as I ended up using it for this purpose (because I didn't see anything else using it).
(in PPU_Write8)
case 0x23:
PPU.BG[0].WindowMask = val & 0x0F;
PPU.BG[1].WindowMask = val >> 4;
PPU.WinLogic[0] = (val?1:0);
break;
case 0x24:
PPU.BG[2].WindowMask = val & 0x0F;
PPU.BG[3].WindowMask = val >> 4;
PPU.WinLogic[1] = (val?1:0);
break;
(in PPU_ComputeWindows)
PPU_WindowSegment* first_s = s;
u8 Win12 = PPU.WinLogic[0] & (PPU.WinX[0]<PPU.WinX[1]);
u8 Win34 = PPU.WinLogic[1] & (PPU.WinX[2]<PPU.WinX[3]);
// check for cases that would disable windows fully
if
(((!((PPU.MainScreen|PPU.SubScreen) & 0x1F00)) &&
(((PPU.ColorMath1 & 0x30) == 0x00) || ((PPU.ColorMath1 & 0x30) == 0x30)))
||
(!Win12 && !Win34))
{
s->EndOffset = 256;
s->WindowMask = 0x0F;
s->ColorMath = 0x10;
return;
}
// first, check single-window cases
if (!Win34)
{
PPU_ComputeSingleWindow(s, PPU.WinX[0], PPU.WinX[1], WINMASK_1);
}
else if (!Win12)
{
PPU_ComputeSingleWindow(s, PPU.WinX[2], PPU.WinX[3], WINMASK_2);
}
else
{
// okay, we have two windows
if (PPU.WinX[0] < PPU.WinX[2])