I am not even sure how to define a guilty hardware pleasure. Easy enough for films/TV, music, books, games, plays and even whole styles of art (this being things that people you normally respect the opinions of/would seek approval from, or possibly have to exist within the opinions of in the case of some kind of mass media, consider bad or trash but you enjoy anyway). I could possibly even do it for tools too (while I have had plenty of old timers make their lives, and mine, a thousand times harder than they need to be I have also met those that can still achieve spectacular results with things I would only have picked up because I like old techniques). This computing lark is hard though. Software too could be a thing, even though you immediately slam into a use case and skill wall -- (la)tex is an absolute pig but I will go toe to toe with any microsoft word user (any -- if there is such a thing as best at microsoft word I will take them on, and I am by no means anything more than competent when it comes to latex) for things I need to achieve there and expect to win.
At first I wondered if it was some kind of input-output peripheral -- using a track ball mouse or drawing tablet despite neither being ideal for a lot of things. Some people look at me oddly for not using any kind of wireless if I can at all help it (and I can -- I would not have a monitor using radiation to transmit data to me if I could inject it into my optic nerve or brainstem instead... should probably look into a cochlear implant too lest I be a hypocrite) but I mainly look at them as slaves to batteries and fads.
I guess you could find someone that stuck with CRT for reasons other than latency. I had ones until they all died for good mainly as they were nice high resolution (especially when I overcooked it). I have a ps/2 keyboard I guess but the ports are still there and it is a lovely mechanical thing that sounds like an 80s cheesy film person racking a shotgun before the final confrontation so there is that. Anybody still using serial or parallel ports probably has a good reason to be doing so, and their simplicity works well too. I think we hunted down and killed all the computer wild men of the woods that were still using zip drives so there is not that any more. I don't tend to use digital microphones (though that is mostly because I can't get skype to play with the playstation eye microphones) but also don't need quality for what I mostly do. I still use VGA but that is mostly because it is what comes out of the back of my docking station, not to mention it still works.
You say trying to do what you can with older hardware which is something of a time tested trick (Quake providing some early examples but the Half Life and Half Life 2 peeps elevated it to an art form) but I don't know if I would call that a guilty pleasure. That said if "2005-2009" in PCs is now considered old then you have just managed to one up the "bloody hell I am old" stakes, which itself is an impressive feat. This mainly as things kind of stagnated around then for most purposes -- 3d graphics and high end video editing might require the fancy stuff but actually even those are often still good enough for most purposes on older gear, never mind something as banal as internet browsing and office work.
On older hardware still then most times I see that it is because software methods still leave something wanting, usually not for any technical reason other as much as "that'll do" (for an example of the opposite
https://www.sfchronicle.com/bayarea...Valley-quest-to-preserve-Stephen-12759775.php ). Efforts to recreate things (if I am sticking with sound then the soundblaster recreations being a good one here) also serve their purposes.
I am at a loss here really. Not only for myself but even in those others I have observed.
