http://www.birdchannel.com/bird-behavior-and-training/afraid-of-hands.aspx
There was one specific part of this article which took my attention:
Is your parrot really afraid of hands?
One easy way to differentiate is to see if there are times when the bird does not act fearful of hands. For instance, will your pet bird readily take a special treat from your fingers? If so, it’s illogical to think it is afraid of your hands. Some birds avoid hands except when they wish to leave a situation they are not comfortable.
After reading this, I thought again to what happens with my bird. He will take a treat from my hands without being scared, even when I move my fingers, but he will avoid them at all other times. I can't think of a time when I have done something negative to him with my hand, except for when my mom grabbed him to put him back in his cage. That's the only reason I can think of of why he would be scared of the outcome of going on hands.