Attack of the AI Bots...

[Forceflow]

Hey everybody.

just an update on the situation regarding bots crawling (and bringing down) the site and forum.
Things are still ongoing and it's a constant battle to keep the crawlers out. I am having to use more and more drastic measures to protect the forum. This might now lead to collateral damage due to the way I am blocking the bots. As such, should get an error messages when trying to access the forum (it should state an error 403 code when loading the page) please contact me ( [email protected]) and I'll try to sort it out.

This is very unfortunate, but unfortunately required to protect the forum.

I have also activated a CDN for the forum that is apparently now included with my contract. This also includes some level of bot protection, so hopefully with this I might be able to lessen some of the restrictions that are currently in place.

[the_stag]

All kinds of crawlers are just ignoring the commands in robots.txt now. Well, that's been going on since the beginning. But now large companies with resources are ignoring it. Tarpitting can be a good tactic, but requires constant vigilance. Good luck on your quest. I'd be interested to see if any of the protections we get from the contract will help.

[KCDodger]

Good luck with this stuff guys. I understand this has been an issue for a long time and I hope there's a solution. It's rough to see it keep happening but y'all do a tremendous job keeping it all alive.

[4Aces]

If they're AI scraping you then I'd seriously consider legal action if you can identify the culprit. In my eyes it's no different than someone breaking into your home and just start using your stuff however and whenever they wish.

[Marcos_Edson]

Wow... I missed these events...
I have little knowledge of this, I understand a DDoS attack goal is to bring down a site on porpose, is this somehow different?

[Forceflow]

Wow... I missed these events...
I have little knowledge of this, I understand a DDoS attack goal is to bring down a site on porpose, is this somehow different?

Sort of, they don't really want to bring down the site, but they keep crawling the forum.not understanding how it works. So they keep going in circles and open a lot of concurrent sessions. This will put a lot of load on the database in the background, which will then finally stop.working