Try replacing your last villager that you invited then. As I said, I have not tried changing anyone aside from the last villager I invited. So that was just my theory.
I was thinking because I have some "wrong ones" the game might look for the wrong one cause it may think they don't have a house since it doesnt find the right house, instead of a good one I put in now... I might try removing the wrong ones and... It may take a while