I would not hang out with him, that's for sure!
You say you'd still rather wear the outfit. He's got nothing to say about that.