It was really poor design to allow the player to get all the way there and do half the temple before finding out something was missing.
Worse still, once getting him I fast traveled to the shrine closest to the temple only for him to then appear, say he'll wait up ahead, and disappear. Turns...