gitea: dis-allow a few AI crawlers that cause too much load

This commit is contained in:
Markus 2025-01-13 20:18:57 +01:00
parent 0d30074d8b
commit ce850f7210

View File

@ -2,3 +2,18 @@ User-agent: *
Disallow: /*/*/archive/*.bundle$
Disallow: /*/*/archive/*.tar.gz$
Disallow: /*/*/archive/*.zip$
User-agent: Amazonbot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: DataForSeoBot
Disallow: /
User-agent: FacebookBot
Disallow: /