But this phrase doesn't always mean to say that "You" are really ugly. Between friend, it can be an expression of a little annoyed or even the speaker prentends to be annoyed.
In a real situation (spoken), "you" can laugh without feeling of irritation at all.
Only in a small number of cases, this expression is used to express anger to others.