US publishers tell Common Crawl to stop scraping and delete archive

toomuchtodo

Crawling will go underground à la Anna’s Archive.

khelavastr

This is shady. Copyrighters absolutely not get to control use of their copyrighted material when people mentally, sonically, or physically reproduce it for personal use. It's absurd to say "you can't record this book to a friend or robot". Nobody seems to actually reproduce the copyrighted materials. High-dimensional eigendecompositions which underpin AI similarity are some of the most literally derivative materials of texts that you can imagine.

Stagnant

Hard to see any practical benefit to go after common crawl. The situation of freely accessible crawled data is bad enough as it is with archive.org and CC being pretty much the only available sources. We need more initiatives like them, not less. The scary thing is how the anti-AI sentiment is being used to lock things down further.

Grimblewald

Its like they don't understand the problem common crawl solved rather neatly. You think the skid scrapers are bad? Wait till the competent players lose access to CC.

mindcrime

These guys (the publishers) are fighting last year's war. Nobody (to a first approximation) gives a shit about going to the NY Times website, or The Guardian website, or the BBC website, etc. to find information. They expect to use search engines and AI services to find stuff, and then maybe click through to the source site(s) for more details or whatever. The publishers need to rethink their entire take on how the Internet works or any "victory" they earn is going to be extremely Pyrrhic.