(1) As far as the common 3DS/WiiU definition goes, it is a homebrew program that (re)loads the console's OS with some tweaks applied (said tweaks being at the discretion of the individual CFW's developer and possibly with some, often limited, user-selectability)
On the WiiU, examples of CFWs are: Haxchi and CBHC (only in part - their homebrew-forwarder features are not CFW), Mocha, the pair of "sigpatch2*" homebrews...
(2) The older but still used definition of CFW, popularized by the PSP scene, consists of a pre-modified operating system that is somehow loaded : the first WiiU CFWs, using fw.img files, were of this type; and a (v)wii CIOS could be pedantically called a CFW according to this same definition
(3) A third possible definition is "any replacement for the official operating system (even if not directly compatible)": while there aren't any WiiU examples off my mind, any ARM9 homebrew for the 3DS is one of these - including but not limited to those which meet definition (1); BootMii is a Wii CFW by this third definition, ...
Confusing? That's because there is are a lot of possible interpretations for the words "firmware" and "custom", with the preferred ones being both subjective and linked to context
Installing "Android" on a PC and calling it a CFW will likely get you laughed at, since hardly anyone calls "Android" a firmware; but doing the same thing on a phone is likely to be more acceptable despite the fact "Android" does mostly the same thing, due to a combination of openness and historical reasons
(the curious meanings of "custom rom" and "hard reset" in the phone world are imputable to the same factors)