Something that you do not value much in your self-concept would not effect your self-esteem if is a negative trait. But if you do value it in your self-concept than it could affect oyur self-esteem easier.
Example from text:"For example, I am not very good at drawing. While I appreciate drawing as an art form, I don’t consider drawing ability to be a very big part of my self-concept. If someone critiqued my drawing ability, my self-esteem wouldn’t take a big hit. I do consider myself a good teacher, however, and I have spent and continue to spend considerable time and effort on improving my knowledge of teaching and my teaching skills. If someone critiqued my teaching knowledge and/or abilities, my self-esteem would definitely be hurt."