Windows Ce is not "entirely" on the console, it's fragmented and its pieces are scattered on game discs and the main console. It allowed using numerous functions that were PocketPC specific, for example internet connectivity.
Most Dreamcasts are entirely capable of playing backups out of the box. Sega expected that their new format, the GD-Rom will be enough to stop hackers. Then CD-Rom overburning started being popular, matching the GD-Rom's capacity and thus "dooming" the console since it was compatible with CD-Roms due to the built-in CD-R/CD player.
You can connect to the internet using the DreamKey browser which is freely available with some games of hardware, you can also burn it.
To connect you will either need the extremely rare LAN connector which replaces the built-in modem OR you will need to build a LAN-to-Dial-Up adapter (yes, it's possible and it's cheaper then the LAN connector if you're technologically inclined to build one).
The Dreamcast also had a keyboard perhaptial which greatly improved internet browsing. It's still available as far as I know.
VMU's could be used as a sort of "portable console" - some games allowed you to install minigame data and play those minigames on your memory card. Unfortunatelly those bad boys ate battery life quite quickly. Most notable title would be Sonic Adventure, allowing you to change your VMU into a tamagotchi.
There's alot to discover in the world of Dreamcast. I honestly believe it died way ahead of its time, but despite that, it has a decent ammount of games in its library and running homebrew on it is as easy as it gets.
Did I miss something?