I added a cron job removing crashed app containers */30 * * * * docker ps --filter "ancestor=collec_app" --filter "status=exited" | xargs docker rm I also stopped and removed the running app containers. Obviously, I do not add that as a cron job as that would also abort legitimate user sessions. Robots continue to be a problem. The current system is not well suited to distinguish robots from human users. This should be managed at the middleware level which I do not develop myself. Robots will be much less of an issue with CollEc 3.0. Though, as you know, I will only work on that towards the end of the year. Christian Düben Doctoral Candidate Chair of Macroeconomics Hamburg University Germany christian.dueben@uni-hamburg.de https://www.christian-dueben.com -----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of Thomas Krichel Sent: Sonntag, 30. Juli 2023 14:09 To: CollEc Run <collec-run@lists.openlib.org> Cc: spider-feedback@bytedance.com Subject: Re: [CollEc] helos overload Thomas Krichel writes
Now I will reboot helos.
Reboot is a succes but bytespider is still there. krichel@trabbi~$ GET https://collec.repec.org/robots.txt # makes requests faster than we can handle User-agent: Bytespider Disallow: / # Crawl-delay: 10 -- Written by Thomas Krichel http://openlib.org/home/krichel on his 21240th day. _______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run