On the internet, you can find several comments telling you that you must complete this quest on the first day, which is not true.
I completed it wandering in Ryo's house on the first day, then slept, and on the second day I could do it.
Just simply, don't go to the village if you're not ready to do this.