You're putting words in Qrow's mouth. He never said it doesn't affect him, if that actually is the case. But again, I come back to the main point that this is all in Qrow's head because he has low self-esteem and a low sense of self-worth. He believes that he causes misfortune and thinks that everything bad that happens around him is his fault despite the fact that this probably isn't true at all. But thoughts like these are the products of depression and not rational thinking. (Trust me, I know). The truth of the matter is that this line of thinking is really only causing misfortune for himself because he feels so worthless. Remember, semblances are more than just abilities that exist in a vacuum. They are extensions of the person's soul and their personality, and just as there is an actual physical component, there is a metaphorical one as well correlating to the person's personality and character. Qrow's semblance is that he transforms into a crow, which is an extension of his low self-worth because he believes he isn't reliable and just causes problems for those he cares about.