ColecoDS - Improved

[wavemotion]

Not ready for a new build yet, but did just check in 5.4a with fixes for Resetting MSX megaROM games (128K or larger).

And a really nice improvement for 256K and 512K games. Right now I'm using the spare 128K of VRAM as a fast-copy buffer... and if the ROM is larger than 128K I don't use it. Which is silly... for example, on a 256K ROM, I can still put HALF of the ROM in the fast VRAM buffer. Half-fast (say that 10 times fast) is better than all-slow. So now 256K and 512K games get a speed boost especially if there is a lot of bank switching happening in the lower 128K.

 

[wavemotion]

https://github.com/wavemotion-dave/ColecoDS

V5.5: 09-Jan-2022 by wavemotion-dave

• Major improvement in speed for MSX megaROM games.
• Fixed RESET of MSX megaROM games.
• Slight optimizations to all emulation cores.

I feel the need... the need... for speed!

A full day of optimizations to move VRAM memory via DMA copy which provided a nice 5-8% speed boost for any ROM more than 64K. Games like MSX Gradius are now full speed, full frame rate without any need for changing the configuration settings. Even Gradius 2/3 are just about playable on the DSi if you switch down to the faster DrZ80 core (actually Gradius 2 seems to be sustaining 60FPS with DrZ80 and Vertical Sync disabled). Gradius 1 is also now playable on the DS-LITE at full speed with the DrZ80 core enabled.

Even smaller ROMs got a bit of a boost - things like SordM5 Apploon (one of my favs!) is now able to keep 60FPS on the older DS-LITE/PHAT without having to turn on frameskip.

Overall it's a polished speed release - hopefully the switch to DMA copy didn't break anything but my tests all looked good.

 

[wavemotion]

So I went down a rabbit-hole of mystery, intrigue and internet drama over on the MSX forums. For the MSX, a 32K ROM can be mapped in a number of ways - starting at a different page offset and sometimes it has mirrors (copies of pages that show up in other ranges of memory) and sometimes it doesn't.

Up until yesterday, I was only supporting the first layout in this diagram:

Namely, ROM is loaded between 0x4000 and 0x8000 in memory with mirrors above and below. This seemed to work for 90% of the 32K games I tried. But it's not really accurate. It turns out the header of the ROM (which I'm already using to auto-detect MSX vs Colecovision games) has some information that helps an emulator know where the ROM should be loaded. So now I support both layouts in that diagram. This improves compatibility to at least 95% and maybe as high as 97%.

Having said that, these are not the only two possibilities. And there are ROMs that want to be loaded in different ways (swapped mirrors, no mirrors, load at 0x8000 with no lower mirror, etc). I will do my best to support these on a case-by-case basis but at least we're auto-detecting a wider variety of games with the most recent (5.5b) check-in.

 

[wavemotion]

https://github.com/wavemotion-dave/ColecoDS

V5.6: 11-Jan-2022 by wavemotion-dave

• Improved loading of MSX 8K, 16K and 32K ROMs for higher compatibility.
• Fixed so we only return joystick input for Port 1.
• Fixed bug in memory write to Slot 1 (rare).
• Minor cleanups across the board.

Big update in terms of the algorithm for loading/launching MSX 8K, 16K and 32K ROMs. This should improve the compatibility - I tried 200 games and 193 of them ran properly for a better than 96% compatibility. You may have to fiddle with the mapper and BIOS settings a bit - but generally you've got the tools necessary to run most MSX games now. Not bad for a Colecovision Emulator

 

[CrashMidnick]

Really nice Dave !

Thanks for the rom mapping explanation. I like when devs are doing such things

 

[wavemotion]

You're welcome, Crash. I like talking it out - it helps me organize what I'm doing and even helps me debug to try and explain it.

I did just fix one of the 7 problem games (out of the first 200). Turns out the TMS9918 video chip can address 4K or 16K of video memory. I guess this was done to help designers save money when it was introduced in 1979 as RAM was pretty expensive. There is a register that software can use to tell it if 4K or 16K is available.

ColecoDS emulates this register and will switch from 4K to 16K mode... however, the MSX only supports 16K mode - and the bit to configure 4K vs 16K VRAM is "ignored" according to the MSX technical documentation. Well, I wasn't ignoring it - and the game HYPE (Bytebusters 1987... great game!) was inadvertently setting this bit as part of the VDP configuration. As soon as ColecoDS switched into 4K video RAM mode, the game showed garbage.

This fix is checked in. We continue to match towards improved compatibility

 

[ber71]

Wow. Finally, you're drilling into the inner details in the msx world, down to the atomic level, a single bit in a IC. You're a lost cause

As always, every milestone is accompanied by a very instructive explanation.

 

[wavemotion]

I've added a custom keypad template for MSX games and am working on making the '5' key user-defined so it can represent any key on the MSX keyboard. This should allow the few games that require odd keyboard commands to get running (i.e. some games require you to press 'J' for Joystick to start the game). I'm trying hard not to add a full MSX keyboard to keep this primarily a lean-mean-retro-gaming-machine!

I also found the root cause of 2 more of the remaining non-running games (32K BASIC-enabled games) and put in a fix for those. Compatibility on the first 200 test games is now 98%

 

[wavemotion]

https://github.com/wavemotion-dave/ColecoDS

V5.7: 12-Jan-2022 by wavemotion-dave

• Fixed RESET of Colecovision games.
• Added MSX keypad template and configurable MSX key '5'
• Improved detection of 32K MSX basic games for better compatibility.
• Allow 64K ROMs to be loaded in linear memory.
• Allow 48K ROMs to be memory mapped.
• Allow 32K ROMs to be loaded at 0K, 4K or 8K.

The MSX keypad is in place. You can change the map of the '5' key - this allows virtually any game to be started. No, you won't be able to play some hacked-cart version of Zork as you don't have a full keypad - this is a gaming console emulator

I've also come to learn that not all MSX 64K ROMs use a memory mapper - some (very few) are linear mapped from 0x0000 to 0xFFFF (the entire Z80 memory space). The amazing new homebrew Mutants from the Deep is one such game. So now you can specify that. I've also learned that some 48K ROMs can use a memory mapper - and so that's now supported. You can also force a 32K ROM to be loaded at 0K, 4K or 8K though the algorithm I'm using now for 32K or smaller ROMs is very accurate but I wanted to make sure I had overrides for ROMs that are unknown to my testing.

You set these memory map overrides in the Game Options. Most games auto detect the memory map but some you’ll have to override to launch such as the Linear 64K games.

Yes, this is all very techy... and you've read through it: so go enjoy some Mutants from the Deep as a reward!!

 

[wavemotion]

With coffee this morning, I added an MSX Compatibility section to the ColecoDS readme documentation.

MSX Compatibility :​

Considering this is a Colecovision emulator, the MSX1 support and compatibility is reasonably high. In Game Options you will notice that the default MSX Mapper is set to "GUESS" which does a fairly good job loading the ROM - especially for 32K or smaller games. However, if a game doesn't run, you can try these suggestions:

• A small number of games don't work with the open-source C-BIOS. In this case you would need a real msx.rom BIOS. You can set this up in Game Options.
• Most 64K games use the ASC16 memory mapper - so you can try that one... but a few (e.g. Mutants from the Deep) are linear mapped from 0-64K and you will need to pick LINEAR64 in Game Options.
• The auto-detection on KONAMI8 and KONAMI-SCC mappers is pretty good... but many other games using ASCII8 or ASCII16 don't detect well - you should try those mappers if the "larger than 64K" game won't run.
• Some of the really big games (128K or larger) run slow - if you're not getting full frame rate, you can try switching to the DrZ80 core (not all games will run with this) and/or you can disable Vertical Sync and/or enable Frame Skip. Recommend you turn off Vertical Sync at a minimum to get the most speed from large MSX games.
• Occasionally one ROM won't run but an alternate dump might. For example, the 384K version of R-Type is a bit of a mess for the emulator to handle, but someone made a clean 512K version that loads and runs great.
• With a little diligence in trying different mapping/BIOS combinations, you should be able to achieve a 97% run rate on MSX1 games.
• MSX2 games are not supported and will not run.

 

[CrashMidnick]

Well done !! Thanks.

 

[wavemotion]

Not ready for a new release but I did just check in 5.7a with one significant improvement. It turns out that I've been using dmaCopy() to move chunks of memory around from VRAM to main memory (as part of the MSX memory mappers). That moves memory in half-words (or 16-bits of memory). There is another function dmaCopyWords() that moves memory in full-words (or 32-bits of memory). Now the VRAM bus is only 16-bits wide and so this doesn't have a 2x improvement... but it does have a 7% instant speed gain on larger MSX games! And that's huge - Gradius II aka Nemesis II is now playable at full-speed on a DSi without frameskip. Even the notoriously CPU intense Salamander is playable with "Show 3/4" frameskip enabled and VSync off. For Gradius II and Salamander be sure to seek out the FRS Dynamic versions where some clever programmer (a decade ago) removed the bottlenecks on those games so they play smoothly on the MSX (these games were notorious for slowdown even on real MSX hardware in their original versions). At this point, even the largest MSX games are playing fairly well on the emulator.

Who knew that dmaCopy() came in more than one word-size flavor?! Woot!

 

[wavemotion]

https://github.com/wavemotion-dave/ColecoDS

V5.8: 16-Jan-2022 by wavemotion-dave

• Improved DMA memory handling of MSX to bump mega ROM games speed by up to 10%
• Improved loading database so more games detect memory mapper correctly.
• Faster RAM swapping for improved loading speed on games.
• Other cleanups and improvements under the hood.

Another spit-and-polish release. A few more games added to the compatibility list - Vaxol now runs properly (it's a game known to do 'garbage writes' to some of the memory mapper hotspot areas which needed to be guarded against). Just about everything on the MSX side is running great. I also made a sweeping pass back through testing the Colecovision library (after all, this is a CV emulator!) and those games all still run great with no significant issues of any kind.

Enjoy!

 

[wavemotion]

https://github.com/wavemotion-dave/ColecoDS

V5.9: 19-Jan-2022 by wavemotion-dave

• Refactor of memory to gain another 128K of fast VRAM to improve Coleco MegaCart and MSX games.

Big change under the hood - refactored memory to gain back an additional 128K of faster Video RAM for emulation use. This 128K was utilized by the "walking plumber" animation on the top screen along with swapping in various micro-screenshots. That stuff isn't done when the actual game is running to that 128K could be freed up and re-purposed to Emulation use.

So that means I now have 270K of fast Video RAM available for general emulation use (up from 144K originally).

16K of this is tied up in color look-up tables and the other 256K is available for improved memory manipulation.

For the Colecovision, this allows us to put MegaCarts up to 256K into the faster VRAM to improve Bank Swapping. This includes all but the very largest game (the only game I have that is larger than 256K is Wizard of Wor). This should ensure that even the really big games run fast enough even on the DS-LITE.

For the MSX, I've decided to utilize the memory a little differently. After some experimentation, I decided on this:

64K for the shadow-copy of MSX RAM
32K for the shadow-copy of the MSX BIOS
160K for the shadow-copy of the MSX Game ROM

This gives a nice performance boost to all games up to 128K and even beyond (32K of the next 128K for really big games is also in fast memory). In practice this allows games like Gradius III to be playable at full speed and many of the other huge games that were struggling to keep up at 60FPS now have plenty of CPU power to make the game play smoothly.

 

[wavemotion]

Not ready for a new build - but did check in 5.9a with timing improvements to the MSX interrupt handling.

This fixes the following games which now work correctly:

• Blow Up!
• Super Cross Force
• Haunted House
• Pharaoh's Revenge
• SOLO
• Star Trek - SoS
• Probably a few others ...

Out of 423 MSX games tried, these are the only ones I know of that still have problems:

• Astro Blaster - runs but only at ~10FPS due to excessive memory slot swapping.
• Avenger - no in-game sounds.
• Bomber King - won't load.
• Ghostbusters - won't run.
• King's Knight - runs slowly at ~50FPS due to heavy memory bank swapping.
• Konami SCC sound chip is not emulated - so background music on Gradius II, III, Salamander, Parodius and a few others isn't audible. The games are still very playable with in-game sounds working.

I'd say the compatibility on MSX1 games is above 98% and that seems reasonable enough until another developer wants to push it closer to 100% in the future!

 

[CrashMidnick]

I'd say the compatibility on MSX1 games is above 98%

And you told me not to be excited Congratulations, and thanks for all hours spent on it.

 

[wavemotion]

https://github.com/wavemotion-dave/ColecoDS

V6.0: 21-Jan-2022 by wavemotion-dave

• Improved MSX compatibility - more playable games.
• Added D-PAD to emulate MSX arrow keys.
• Added D-PAD diagonals emulation.

Version 5 saw the addition of MSX1 support.
Version 6 gets it as close to perfect as I can probably manage.

I've tweaked a few more games into playable shape. You would be hard pressed to find too many games that you really want to play that you couldn't make work on ColecoDS. When you add it up, I've probably tested close to 1000 Colecovision, Sord M5, Sega-1000 and MSX1 games - the vast majority play great.

There were a few games that required the use of the MSX arrow keys and I've added a new configuration option to emulate that with the DS D-Pad. You can also switch into 'diagonals' mode - it's really not easy on a D-Pad to hit the diagonals for games like Q-Bert so this will auto-map the diagonals to UP/DOWN/LEFT/RIGHT - if you've ever played Q-Bert on the Colecovision vs MSX you will know what I'm after here.

Enjoy! All of the features I wanted to add are in place - this emulator goes into maintenance mode soon.

 

[wavemotion]

Coming in the next day or so... a semi-full MSX keyboard. There are a few games that just need the traditional keyboard to be playable on the MSX. This will be one of the custom overlays you can pick in Game Options for any MSX game.

I'm also adding SRAM support for Hydlide II and Dungeon Slayer 2 - both RPGs which really want the full keyboard

Edit: 6.0b is checked in with the new MSX keyboard support. I've also enabled SRAM on a few of the most common games that use it. I also added 'beeper music' support as a few of the ZX spectrum ports that use the built-in MSX beeper to produce sounds (much like the old ZX Spectrum did)... so games like Avenger, Batman (The Movie) and Masters of the Universe now have in-game sound (it's not perfect but good enough).

Edit2: 6.0c is checked in with a fix for the ASC8 mapper that was preventing some games from loading. Bomber King is now playable (yay!) as is Batman - The Ronin and a few others.

 

[wavemotion]

https://github.com/wavemotion-dave/ColecoDS

V6.1: 25-Jan-2022 by wavemotion-dave

• Added full MSX keyboard overlay - Choose "MSX Full" in Game Options.
• Improved ASC8 mapper so Bomber King, Batman Rovin and others now playable.
• Added SRAM support to make Hydlide II, Dragon Slayer II (Xanadu) and Deep Dungeon 2 are now playable.
• Added 'beeper music' sounds so games like Avenger, Batman - The Movie and Masters of the Universe have sound.

 

[wavemotion]

Not ready for a new build but I did check in a build which expands the AY noise from 3 frequency levels to 7 levels.

ColecoDS uses Flubba's awesome SN sound chip emulation core - which is perfect for Colecovision, SordM5 and SG-1000. However, the Super Game Module and MSX use the AY chip which is similar - but different. One big difference is that the AY supports 32 frequencies of noise. This has always been difficult to map onto the SN sound chip. So I've modified the driver to support an extra bit bringing the noise levels from 3 to 7 - it's still a far cry from 32 but it's a big step towards having the MSX games sound more authentic.