Advice on a nervous pony please?

It sounds like your doing a really good job, I think it's just a case of the horse getting used to his new owners and surroundings. When I got my horse I brought her home and was so worried at how spooky and restless she was, like your horse she hadn't had much human contact with previous owners, but it took her about 3 weeks to settle down and once she was in a routine she soon found her place and relaxed. I think doing the same things in the same order around about the same time each day will help because then he will learn what to expect and realise that your not going to confuse him. Hope it helps, don't be disheartened, I was at first but it does just take time by the sounds of things youl make a lovely little pony of him :) good luck