Why are men always portrayed as less attractive than women?

I think this is more of an indication of inherent misogyny in society. What does it say about our society that we have a double-standard where women are judged largely by their appearance and are expected to meet a particular standard, but men aren't expected to cater to their appearance as much? It seems unfair to women.