The meeting should take place right after you got the amulet. As soon as you take it, he comes and talks to you.