Your perfect HP man is Severus Snape.
You like a guy of loyalty and intelligence and don't really mind if he comes across as a bit harsh. Or a bit bastardy. Or if he happens to terrify every child who crosses his path. The point is that under that rough exterior lies...well, a rough interior. But under that is a soft, squishy center and you don't mind at all that you're the only person in the world who gets to see it.
(Fanart by Grant Gould GrantGould.com Used with permission.)