Villagers shift locations based on the time of day. Make a habit of doing rounds in the morning and evening to maximize your daily relationship points.