With the existence of other comments like "Great", by avoiding those and simply saying "Not bad", you're either saying it's average or not good. It isn't a decisive remark, and can mean many different things. If you want to take "not bad" literally, then that would make the remark a very poor and vague one at that.
No, you've misunderstood me. The reason I claimed it to be the worst remark to make is not because it contains the word "bad", I'm not as delusional as you think... I claimed it so because it's not at all a decisive comment to make about something. If you're going to judge and comment about something, it's stupid to say 'it's not bad', at least tell me what it is, not what it is not.
By saying something is "not bad", you're leaving the object with so many other possible judgements (it now could be great, good, average, dull, etc, just other than bad...).
Just saying "not bad" and leaving it at is the same as just saying "good" and leaving it at that, those many other possible judgments (whether they'd be positive or negative is irrelevant right now) would be added in further comments.