Besides a private instance with federation disabled (and even then), I don't think Lemmy is really designed for private communities. Even in NSFW communities, guest users can still see the posts if they have direct links to them, or use a non-lemmy fediverse viewer ( https://mastodon.social/@[email protected] ).
Even if there was a way to do it (perhaps with a Lemmy fork), you also have to trust that any instances users are on also comply with your requests. Depending on your requirements for privacy, this may not be something you want to rely on.
Is this even a desirable feature Yiffit turned on, or is it just a quirk of how Lemmy works? It certainly hurts discoverability of content here, IMO.