Cilla used the word "ugly", so although it's such an ugly word, I'll continue it's use.
What is ugly? Sounds like a good subject for contemplation.
I find myself just looking at what is. That's what it is. There is this that I'm seeing now, there is that, that I've seen at other times. What's the difference?
Beauty is in the eye (or should we say mind?) of the beholder. As is ugly. You can't have one with out the other, but both are based on nothing. A 6 inch rope is shorter than a 8, an 8 is shorter than a 10. The only way to know the difference is by comparison and then developing desire for or against whatever the differences we perceive there to be.
And then there is this word vanity. Vanity means "something that's worthless" - vain, no value. In our beauty culture we've changed the word to mean attention to one's self-beauty, but that is only one narrow application of this word. It is vain to obsess over beauty, it is also vain to obsess over "ugly". Neither has any value.
When I look in the mirror I see what is. When I see my wife, I see what is. Her looks cause me no grief nor joy, but our relationship can cause both - and yet even this must be overcome.
Maybe I'm rambling now...
I think your own self-aversion, painful as it is, is common with us all. We all share attributes that we don't like about ourselves, equally as much as you don't like this. It's like a job, all jobs have good and bad. Leaving this job to go to another one isn't going to release me from problems, mainly because wherever I go, there I am. And so it goes with relationships. I take myself into each one of them, thus the problem starts.
One thing I've found about every face I've ever observed (and yes, I'm an "observer of faces", mostly reading personality, mood, etc), is that the very first impression that hits my mind when someone is smiling is how attractive they are. Only after closer "observation" do I find things about them that I might not find attractive.
I believe all of us are attracted more to those who are content with our selves (sigh, so hard to do this - the self is the problem, to be content with our self is in itself not possible), people who exude happiness, and a relaxed acceptance of others.
And even this initial attraction must be done away with as it's based on emptiness. (Sorry, just had to throw that in.) Fortunately most people aren't busy throwing this initial attraction away. (Fortunately for the temporary ego who lives on being attractive, but ultimately it's not fortunate because the bearer of this subject-to-attractiveness nature suffers for having it).
But even in doing away with these attractions and aversions, I've found they have to come in steps. While I know it's vain for someone to wear make-up, and a health-hazard for someone to go to a tanning bed (just for two examples), and hope that one wouldn't do these things, I see that as the final destination (the not doing of these things). And the path to this final destination usually has to start with better self-esteem; and self-acceptance may only be achieved in the beginning by practicing these "vain" things. If someone needs to do "vain" things in order to first accept themselves, fine. As their self-acceptance grows, and ignorance is replaced, these things will be replaced by what is real (which I'm quite certain is nothing, in several different senses).
Now I feel like I'm sounding like a shrink-ologist.
Eventually we come to the place where there's no self to have esteem over, no self to accept, no ugly, no beauty, and we see our body, house, car, etc, only as tools that are subject to our will (yet another problem) to perform good in this world. I've seen that in glimpses, but it's still out of reach because I still obsess over "me".