Filter users by chat ID when checking ineligible chat members during chat source refresh

[danoctua]

I don't believe it will kick more than it should, but indeed, it could be overkill to check all chats where these users are participating.

Will do it as a separate improvement.

Originally posted by @danoctua in #232 (comment)