this post was submitted on 21 Aug 2024
322 points (100.0% liked)
196
17376 readers
870 users here now
Be sure to follow the rule before you head out.
Rule: You must post before you leave.
Other rules
Behavior rules:
- No bigotry (transphobia, racism, etc…)
- No genocide denial
- No support for authoritarian behaviour (incl. Tankies)
- No namecalling
- Accounts from lemmygrad.ml, threads.net, or hexbear.net are held to higher standards
- Other things seen as cleary bad
Posting rules:
- No AI generated content (DALL-E etc…)
- No advertisements
- No gore / violence
- Mutual aid posts require verification from the mods first
NSFW: NSFW content is permitted but it must be tagged and have content warnings. Anything that doesn't adhere to this will be removed. Content warnings should be added like: [penis], [explicit description of sex]. Non-sexualized breasts of any gender are not considered inappropriate and therefore do not need to be blurred/tagged.
If you have any questions, feel free to contact us on our matrix channel or email.
Other 196's:
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I am confused, does this mean Reddit is not going to be searchable on search engines anymore?
Unfortunately yes. It was reported on last month.
oh no, Reddit is like, the only way to have google still be useful.
Funnily enough, google is also the only way to have Reddit be useful.
Their own search function has been nothing but garbage.
That's the catch, Google made a deal with Reddit and remains the only search engine allowed to access its data for indexing. It cuts off every other search engine
Tell me that there is an anti trust suit over this.
There's a suit over google in general so this may well be part of it
really? ddg will show me reddit links, did they have to make a webscraper or something
There's a cutoff date, anything indexed before the robots.txt was changed stays in the index
We fucked the internet. It’s proprietary now.
kinky
cat5 sounding you say?
cat5-o-nine-tails
Good news! Google paid up and still has access I'm pretty sure.
That's bad news, that means the internet is dying
Sorry, the /s was sort of implied.
Ah, sorry. I have trouble with that sometimes :P
Perhaps, likely depends on the crawler though
Yeah i dont think ignoring robots.txt is even illegal. They can ofcourse just block your crawlers IP but that would be a cat and mouse game that they would lose in the end.