Ninja other than Joe the farmer

How do you get ninja other than joe the farmer in your manor?

I had the quest to banish ninja before and had the girl and other ninja, but after the quest they don't appear anymore..
I'm asking because during the ninja banishing quest they drop collection items for femme fatale collection and ninja war trophies.
Since Joe the farmer do not have the items such as claw the collection is incomplete. I keep trying to banish joe the farmer in hope that the other ninja will show up but they do not. I also tried to visit my friend's manor in hope they have it. But even when my friend is having the quest (they confirmed it) i still can't see the other ninja when i visit my friends.

Any advise?

I have to visit friend rooms to get chan, agronomist, and kun. I mostly see joe, but some of my friends do have the other ninjas. I too did the quest and only have joe floating around.