Don't use his hunger to make him your friend. Parrots are highly intelligent animals and they know very well you are the bringer of food so, if you take it away and force him to take it from your hand, you are not endearing yourself to him - quite the contrary! This is what we call a 'flooding' technique. We used to use them years and years ago but that was because we did not know any better. Now we know that, although they might work in the short term, they backfire in the long run because you are forcing the bird to accept something he doesn't want to accept.
If this bird is a baby, you need to give him soft food (warm and served fresh twice a day), if he is not, soft food is still recommended (it's the best way to make sure he is eating a good diet with veggies and fruits) but only once a day, early in the morning. But, baby or adult, a lovebird is a parent-raised bird and they never really bond to people. They can learn (with a lot of patience) to trust you and even like you but they will never love you the way they would love another bird. This is especially true of species like lovebirds which are extremely pair oriented so the best thing for them is to have a mate (it's the only thing that makes them really happy).
I have a pair of lovebirds. The female is tamed and will step up to my finger, kiss my cheek, come out to fly and perch on my shoulder, etc but the male is an ex-breeder which is not tame at all. He used to act like yours (freak out every time I put my hand inside the cage for cleaning or feeding) but he no longer does -he even stays perching right next to the bars when I am standing next to the cage and I can now touch him through the bars without him scrambling away but it took a long time to achieve this (all I did was talk to him and keep the same exact routine, ignoring him when he would fly away from me, and he slowly calmed down).
Put the cage high up so he is at the same level as your face when he is perching. Don't stare at him, look at him from the corner of your eye and talk, sing, whistle to him as much as you can. Get him a LARGE cage and put branches inside for him to climb up and down. Put him on a fresh food diet (lovebirds are very good eaters, they love all greens and fruits and quite a number of vegetables) for the morning and just give him a good quality seed mix at night for dinner. Keep him at a strict solar schedule with full exposure to dawn and dusk and a good light source for during the day.
Once you see that he is no longer scared of you (he will not scramble to get as far as he can from you when you get close to his cage), start offering him treats through the bars of the cage but, if he doesn't take the seed you are offering, just put it into his food bowl (it's supposed to be a token of friendship, he is not supposed to do anything for it). Mind you, for the treat offering to work, he can't be free-fed seeds or it will never work. Once he starts taking the seed from your hand (it's going to take a looooong time so don't get impatient and give up before you get any results), you can start target training him from inside his cage.
But, above all, I suggest you get him a mate. There is absolutely nothing better for him than having a mate or, at the very least, a companion... it's the only way they are happy. But you need to make sure you end up with either two males or a male and a female because you can't house two females together.