Whatever we think of archive.org's position, modern AI companies have clearly taken the same basic position.
Whatever we think of archive.org's position, modern AI companies have clearly taken the same basic position. And are willing to devote a lot more resources to vacuuming up the internet than crawlers did back in 2011. See https://news.ycombinator.com/item?id=43476337 for a random example of a discussion about this. My personal position is that robots.txt is useless when faced with companies who have no sense of shame about abusing the resources of others. And if it is useless, there isn't much of a point in having it. Just make sure that nothing public facing is going to be too expensive for your server. But that's like saying that the solution to thieves is to not carry money around. Yes, it is a reasonable precaution. But it doesn't leave me feeling any better about the thieves.
Posted by btilly on Aug 18, 2025, 6:49 PM
Tags
Related issues
- there are tons of TLDs that are cheap I'm not even sure what you are offering after reading this, but cloud is an even harder sell becau...
- > One can't expect that every driver package submission from every vendor is thoroughly vetted every time, this is a matter of trust and ...
- Too expensive for a worse vehicle than the Model Y (with incentives, it was a >$20K difference).
- Starlink only has 10 Million customers, too expensive for most countries already.
- The U7 Lite only does 2x2 MIMO.
- Iranians use VPNs simply because they want to access the internet.
Raise real-world problems, find people facing the same issue, and discuss solutions.2,000 issues0 comments
pricing (Network)
there are tons of TLDs that are cheap I'm not even sure what you are offering after reading this, but cloud is an even harder sell becau...
> One can't expect that every driver package submission from every vendor is thoroughly vetted every time, this is a matter of trust and ...
Too expensive for a worse vehicle than the Model Y (with incentives, it was a >$20K difference).
Starlink only has 10 Million customers, too expensive for most countries already.
The U7 Lite only does 2x2 MIMO.
Iranians use VPNs simply because they want to access the internet.
> When you don't want the ads and privacy invasion, you don't visit the website.
A vantage point from a very long time ago: The big social media services are pining for the days of CompuServe and Prodigy.
Whatever we think of archive.org's position, modern AI companies have clearly taken the same basic position.
Whatever we think of archive.org's position, modern AI companies have clearly taken the same basic position. And are willing to devote a lot more resources to vacuuming up the internet than crawlers did back in 2011. See https://news.ycombinator.com/item?id=43476337 for a random example of a discussion about this. My personal position is that robots.txt is useless when faced with companies who have no sense of shame about abusing the resources of others. And if it is useless, there isn't much of a point in having it. Just make sure that nothing public facing is going to be too expensive for your server. But that's like saying that the solution to thieves is to not carry money around. Yes, it is a reasonable precaution. But it doesn't leave me feeling any better about the thieves.
Comments (0)
No comments yet. Start the discussion.