Search Engine crawlers

[bahua]

Our new VPS host bills its customers based on usage, in four categories: storage, memory, CPU, and network. Our old host charged a flat $28 a month for the tier of VPS we rented, and was a very capable host in terms of site performance. I would argue that performance is much better now. (though you may disagree from where you sit. If you do, I definitely want to hear from you via PM or replies to this post). I have been watching some tools on the server, and I've seen that the database is basically always running hot. I am starting to see why:

Code: Select all

$ wc -l access_kcrag.log
3579405 access_kcrag.log

That's 3.5 million hits, since Monday night. When I watch the access log live, I see lots and lots of search bots, so I decided to see how many we're dealing with:

Code: Select all

$ for i in bing google semrush baidu duckduckgo amazon; do echo -n "$i: "; grep -iw $i access_kcrag.log | wc -l; done
bing: 36077
google: 4306
semrush: 4417
baidu: 17
duckduckgo: 121
amazon: 9

36k hits for bing, and under 10k for all other identified engines put together, since Monday night. There's another scraper called ClaudeBot that alone accounts for almost half a million hits. Over 1.4 million hits come from requests for the site without https.

868k requests for search from guests. I just disabled that as I've been composing this. That should hopefully provide at least some relief to the database, but we'll see.

This is going to take some deeper investigation and some planning to address, but it appears the forum is a major target for web crawlers, which account for the vast majority of the site's traffic. I originally was writing this post to get your input on whether we want to actively block these bots, but the situation appears to be much more involved than that.

Any input anyone has would be appreciated.

[bahua]

On some further investigation, I've found that remote IP addresses starting with 47.75 account for 2.5M of these hits. The addresses belong to Alibaba, and originate in Hong Kong. I think I will block them.

[bahua]

This is fascinating. The requests from these Alibaba IPs are using the search function to list all of the posts of a given user, and then dive into those posts. I don't think these alibaba crawlers are populating a search engine. Based on the depth and frequency of their hits, I think they're feeding data to an AI. I dropped a deny directive for this IP block into kcrag's vhost config, and restarted apache.

I'm tailing the access log now, and the requests from these IPs are still pouring in, at about 1000 requests per minute, but now they're all getting 403 "forbidden" errors. I'm watching htop in another screen, and the database has dramatically calmed down. I will continue to monitor this, but I think with the rise of AIs, this kind of event will get only more common.

Please let me know if you see anything like or trouble like this.

[grovester]

So there are AI versions of all of us based on our posts? That's terrifying!

[bahua]

So there are AI versions of all of us based on our posts? That's terrifying!

Yeah, but don't worry. They won't emerge until we die.

[bahua]

Look at that! CPU and network use dropped off sharply!

[KCKev]

Man you got alot going on. I've noticed when I'm about to log off I check who's on line Z(out of habit) and it's usually 4 registered online
bing [bot],google[bot],KCKev,semrush[bot].

Bots are registered? Not guests?

[bahua]

No, they're not registered. They're detected by the board software.

[brooksidebadgers]

fascinating and thanks for sharing. I eat this kind of stuff up.

[bspecht]

Cloudflare offers solid bot protection with their free plan, likely adequate for a site like this. https://developers.cloudflare.com/bots/ ... rted/free/

[im2kull]

This is fascinating. The requests from these Alibaba IPs are using the search function to list all of the posts of a given user, and then dive into those posts. I don't think these alibaba crawlers are populating a search engine. Based on the depth and frequency of their hits, I think they're feeding data to an AI. I dropped a deny directive for this IP block into kcrag's vhost config, and restarted apache.

I'm tailing the access log now, and the requests from these IPs are still pouring in, at about 1000 requests per minute, but now they're all getting 403 "forbidden" errors. I'm watching htop in another screen, and the database has dramatically calmed down. I will continue to monitor this, but I think with the rise of AIs, this kind of event will get only more common.

Please let me know if you see anything like or trouble like this.

This is super interesting. Good interpretation of what you found.

[bspecht]

AI, as it currently exists, wouldn't be constantly crawling a site. An entity could/would do a one-time crawl of the information (though likely with a certain frequency) and parse it into a model for a dataset, which would then be trained on by an LLM (not doing any operations that would ping a website). That said, phpBB software is a rather basic data structure to grab & process + in demand given it's historical information uses.

[bahua]

Cloudflare offers solid bot protection with their free plan, likely adequate for a site like this. https://developers.cloudflare.com/bots/ ... rted/free/

Suspicious that the bandwidth alliance, with whom cloudflare coordinates to identify "good" bots and "bad" bots is cosponsored by Alibaba, who is responsible for 2.5 million of the bot hits we've gotten this week. I think I'll stick with my own methods for now.

[Anthony_Hugo98]

Cloudflare offers solid bot protection with their free plan, likely adequate for a site like this. https://developers.cloudflare.com/bots/ ... rted/free/

Suspicious that the bandwidth alliance, with whom cloudflare coordinates to identify "good" bots and "bad" bots is cosponsored by Alibaba, who is responsible for 2.5 million of the bot hits we've gotten this week. I think I'll stick with my own methods for now.

Create the problem and sell the “cure” setup I’d bet

[bspecht]

Comical. Cloudflare created and has dominated it's sector because it works incredibly well at reasonable cost.

I'll refrain from further suggestions. Conspiracy theories are more fun!

[bahua]

I use cloudflare extensively. I'm aware of its value, but I'm also aware it isn't infallible. I'm not using their services in this case, as the host we're using has geographic edge POPs already, and the performance boost from this is obvious, at least from where I'm sitting.

But companies often have to make sacrifices to get access to the Chinese market, and the association I've observed between cloudflare and alibaba might be an example. I don't know if it is, and I really have no way of knowing if it is, but I do know that bots from one of cloudflare's business partners hit us hard, and continue to do so. Blocking their class-B with an authz module directive is a simple way to reduce the load that has proven extremely effective.

And on a administrative note, I think we could all do without your snark. Please keep it civil here.

[mean]

Looking for Chinese dissidents or some shit, if I had to guess. If they're scraping for an LLM, it's to let AI figure out who is talking bad about the CCP or something, not build a new model.

Obviously that's wild speculation, but if you don't think that's at least plausible, then I'm afraid the joke's on you.