this post was submitted on 01 Mar 2025
32 points (100.0% liked)
Privacy
1144 readers
732 users here now
Protect your privacy in the digital world
Welcome! This is a community for all those who are interested in protecting their privacy.
Rules
PS: Don't be a smartass and try to game the system, we'll know if you're breaking the rules when we see it!
- Be nice, civil and no bigotry/prejudice.
- No tankies/alt-right fascists. The former can be tolerated but the latter are banned.
- Stay on topic.
- Don't promote proprietary software.
- No crypto, blockchain, etc.
- No Xitter links. (only allowed when can't fact check any other way, use xcancel)
- If in doubt, read rule 1
Related communities:
founded 3 months ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I've tried scraping arXiv before and they blocked my access to the website stating suspicious activity and that I should contact arXiv owners if I want to scrape it, despite the reason for arXiv existing being for Open Access to scientific articles.
Scraping for scientific articles is still on my plans, but not arXiv anymore.
The gap between the common user and the big technology players is more than just a gap of knowledge, these agreements to keep everything on the hands of big companies is problematic, especially when dealing with important philosophical concepts that should guide some websites, like Open Science.
But it is what it is, no use sending an email to arXiv telling they're wrong and they should change their minds. I'll have to look for options.
arXiv has bulk access methods -- you shouldn't need to scrape their website to get the data: https://info.arxiv.org/help/bulk_data.html
If you really want everything (5TB+), that's available from their S3 bucket if you're willing to cover the transfer costs: https://info.arxiv.org/help/bulk_data_s3.html