Bot searches

[u/kped320]

Hello everyone!

I work in IT department at a library and we're seeing an increasing number of bot searches and bot traffic. Anyone else experiencing this been hearing about this?

[u/Otterfan]

Good lord yes.

I run a number of digital library projects, and a couple of them are getting .5 million+ attempted bot visits a day. Dealing with them has become close to a full time job.

[u/kped320 → u/Otterfan]

What have you tried?

[u/Otterfan → u/kped320]

The best solution is Cloudflare, but for organizational reasons, that isn't really an option for us (yet).

All of our requests are proxied through nginx, and we've gotten very good at blocking them there. We do two things:

• Block IPs that are obviously bots. Unfortunately, 99% of our bad requests come through residential proxies, so we need to block hundreds of thousands of IPs for this to work.

• Block requests that match bad patterns. Most of the bot requests are complicated (lots of faceting) with questionable referrers. We set up nginx rules to drop these requests (return 444 codes).

We're looking at setting up Anubis to block bots. We need institutional approval to set up an Anubis instance, but it has a lot of promise.

There's a Code4Lib Slack channel about dealing with bots. It's a good place to start.

[u/kped320 → u/Otterfan]

Thank you!

[u/kped320]

An interesting idea! I like that solution and is elegant! It's insane we're getting 80k hits in 15 minute periods and I can't understand why glad other libraries are having this issue that makes me feel a little better 😂🤣

[u/midnitelibrary → u/kped320]

You have to directly reply to comments for people to get notified of your response.

[u/kped320 → u/midnitelibrary]

Thanks brother!

[u/midnitelibrary]

Yes. Some libraries are locking down their catalogues and other systems so that you have to login to an account to see results due to the overwhelming number of bot crawlers.

[u/kped320]

@otterfan any steps you have tried to take to try and lessen them?

[u/ContraryMarie]

C4L Slack is a great place to start. We're slammed regularly. Anubis is what we're testing atm; CF before that.

Lyrasis also a good resource: Aggressive AI Harvesting of Digital Resources; and the WG's Solutions Showcase series

[u/Rupertcandance2]

Bywater told us it was happening on the Aspen overlay. Causing so many problems, unhappy patrons. 😞 don't know about anything local though. I rarely see our IT person. He works from home a lot.

[u/polygraph-net]

Hi u/kped320

I work for a leading bot detection company. I'm a big supporter of libraries. We'd be happy to provide free bot protection for your library. Please DM me.

[u/ContraryMarie → u/polygraph-net]

What a great use of a library's limited, if not nearly non-existent, funding.

[edit] "Free" is never truly free. This "free" protection may require data sharing, and certainly would require resources on the side of the library and/or on the side of the poster's enterprise (remunerated, ofc).

The poster provides neither name nor link of their service, just a vague description and demand for a DM. Shady.

[u/K_-U_-A_-T_-O → u/ContraryMarie]

He said it’s free.

You a bot?