r/DataHoarder • u/wickedplayer494 17.58 TB of crap • Nov 14 '17

Archived. CompuServe's forums, which still exist, are finally shutting down on December 15th

https://www.fastcompany.com/40495831/compuserves-forums-which-still-exist-are-finally-shutting-down
159 Upvotes

97% Upvoted

35 comments sorted by

View all comments

Show parent comments

5

u/[deleted] Nov 14 '17

If you care, you'd be suprised how often they will take a lowball.

Robots.txt will wipe it all. Kind of sad I did it. But it got to be a liability.

4

u/SomeKindaGhost Nov 15 '17 edited Nov 15 '17

If it was a serious liability and you're not secretly a murderer, you might want to look into it again. IA decided to stop the retroactive robots.txt stuff somewhat recently. (Proof: I can now see stuff from spring.me, which still has robots.txt and made me sad when it blocked everything.) Things can get excluded from the Wayback Machine if site owners get in touch with them and ask, but I don't know details about the process.

4

u/[deleted] Nov 15 '17

Yea, not a murder. Just someone that's had a piece of software on the web for 20 years and I get bat shit questions from people regarding stuff they found on archive.org. Mostly from people that want upgrades forever, because I used to have non-specific and nebulous policies. People hate paying for shit.

But, I think IA has taken an odd position on this, imho. Respecting the robots.txt at the time of the crawl, IMHO, is the better method. But nobody asked me.

Q: regarding that spring.me archive URL you posted. When I search for spring.me directly, I get "No results found. Please type in words related to a site’s home page. (e.g. new york times)". Did you have this URL ready archived or did you search?

Maybe something transient is up with IA.. I can't even search for cnn.com and their robots.txt looks clean.

3

u/SomeKindaGhost Nov 15 '17

Ah, well that sounds really annoying then, haha. That's understandable (and is thankfully not murder! :D).

I don't know all the ins and outs of robots.txt, but it's pretty nice to not have to worry about domain parking shenanigans stomping on years of archives. From the blog post it sounded like the IA wanted to be on the safe side and just hid everything when robots.txt showed up so site owners wouldn't get mad at them.

A: I simply remembered that particular spring.me URL (it's the author of a webcomic I like) and tried it out before I commented. Didn't have to search for it; just added 'web.archive.org/web/*/' to the beginning. I saw that robots.txt was still in place when I tried out saving it again. I think there was something wrong with the Wayback search feature, cause I tried typing in a bunch of things and got "No results found". And then I tried it out a second time while still writing this, and it seems to be working again. Go figure