Re: [CollEc] helos overload

30 Jul 2023

      Made a line in robots.txt

root@helos /etc/nginx # grep -A3 robots sites-available/collec.repec.org | grep -v \#\#
  location = /robots.txt {
    return 200 "# makes requests faster than we can handle\nUser-agent: Bytespider\nDisallow: /\n# Crawl-delay: 10\n";
  }

  Now I will reboot helos.

-- 
  Written by Thomas Krichel http://openlib.org/home/krichel on his 21240th day.