]> gitweb.ps.run Git - ps-cgit/commitdiff
robots.txt: disallow access to snapshots
authorJason A. Donenfeld <Jason@zx2c4.com>
Tue, 28 May 2013 12:17:00 +0000 (14:17 +0200)
committerJason A. Donenfeld <Jason@zx2c4.com>
Mon, 12 Aug 2013 19:14:10 +0000 (13:14 -0600)
My dmesg is filled with the oom killer bringing down processes while the
Bingbot downloads every snapshot for every commit of the Linux kernel in
tar.xz format. Sure, I should be running with memory limits, and now I'm
using cgroups, but a more general solution is to prevent crawlers from
wasting resources like that in the first place.

Suggested-by: Natanael Copa <ncopa@alpinelinux.org>
Suggested-by: Julius Plenz <plenz@cis.fu-berlin.de>
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>

No differences found