this post was submitted on 01 Mar 2025
32 points (100.0% liked)

Privacy

1144 readers
740 users here now

Protect your privacy in the digital world

Welcome! This is a community for all those who are interested in protecting their privacy.

Rules

PS: Don't be a smartass and try to game the system, we'll know if you're breaking the rules when we see it!

  1. Be nice, civil and no bigotry/prejudice.
  2. No tankies/alt-right fascists. The former can be tolerated but the latter are banned.
  3. Stay on topic.
  4. Don't promote proprietary software.
  5. No crypto, blockchain, etc.
  6. No Xitter links. (only allowed when can't fact check any other way, use xcancel)
  7. If in doubt, read rule 1

Related communities:

founded 3 months ago
MODERATORS
[email protected]
otter
[email protected]
[email protected]
32
Scraping for Me, Not for Thee: Large Language Models, Web Data, and Privacy-Problematic Paradigms (epic.org)
submitted 1 day ago by [email protected] to c/[email protected]
2 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] [email protected] 12 points 1 day ago

arXiv has bulk access methods -- you shouldn't need to scrape their website to get the data: https://info.arxiv.org/help/bulk_data.html

If you really want everything (5TB+), that's available from their S3 bucket if you're willing to cover the transfer costs: https://info.arxiv.org/help/bulk_data_s3.html