it's back. I have stopped nginx and stopped docker.socket. That seems to work but leaves CollEc not running. -- Written by Thomas Krichel http://openlib.org/home/krichel on his 21240th day.
Thomas Krichel writes
it's back.
I have stopped nginx and stopped docker.socket. That seems to work but leaves CollEc not running.
Now I'm trying to kill the container daemon.
-- Written by Thomas Krichel http://openlib.org/home/krichel on his 21240th day.
_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run
-- Written by Thomas Krichel http://openlib.org/home/krichel on his 21240th day.
Made a line in robots.txt root@helos /etc/nginx # grep -A3 robots sites-available/collec.repec.org | grep -v \#\# location = /robots.txt { return 200 "# makes requests faster than we can handle\nUser-agent: Bytespider\nDisallow: /\n# Crawl-delay: 10\n"; } Now I will reboot helos. -- Written by Thomas Krichel http://openlib.org/home/krichel on his 21240th day.
Thomas Krichel writes
Now I will reboot helos.
Reboot is a succes but bytespider is still there. krichel@trabbi~$ GET https://collec.repec.org/robots.txt # makes requests faster than we can handle User-agent: Bytespider Disallow: / # Crawl-delay: 10 -- Written by Thomas Krichel http://openlib.org/home/krichel on his 21240th day.
I added a cron job removing crashed app containers */30 * * * * docker ps --filter "ancestor=collec_app" --filter "status=exited" | xargs docker rm I also stopped and removed the running app containers. Obviously, I do not add that as a cron job as that would also abort legitimate user sessions. Robots continue to be a problem. The current system is not well suited to distinguish robots from human users. This should be managed at the middleware level which I do not develop myself. Robots will be much less of an issue with CollEc 3.0. Though, as you know, I will only work on that towards the end of the year. Christian Düben Doctoral Candidate Chair of Macroeconomics Hamburg University Germany christian.dueben@uni-hamburg.de https://www.christian-dueben.com -----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of Thomas Krichel Sent: Sonntag, 30. Juli 2023 14:09 To: CollEc Run <collec-run@lists.openlib.org> Cc: spider-feedback@bytedance.com Subject: Re: [CollEc] helos overload Thomas Krichel writes
Now I will reboot helos.
Reboot is a succes but bytespider is still there. krichel@trabbi~$ GET https://collec.repec.org/robots.txt # makes requests faster than we can handle User-agent: Bytespider Disallow: / # Crawl-delay: 10 -- Written by Thomas Krichel http://openlib.org/home/krichel on his 21240th day. _______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run
Düben, Christian writes
I added a cron job removing crashed app containers
*/30 * * * * docker ps --filter "ancestor=collec_app" --filter "status=exited" | xargs docker rm
Thank you.
I also stopped and removed the running app containers. Obviously, I do not add that as a cron job as that would also abort legitimate user sessions.
Well the running container should have stopped on the reboot.
Robots will be much less of an issue with CollEc 3.0. Though, as you know, I will only work on that towards the end of the year.
Sure. In the meantime we just continue. This time it was better than last because I know what was behind the problem. -- Written by Thomas Krichel http://openlib.org/home/krichel on his 21240th day.
Thomas Krichel writes
Düben, Christian writes
I added a cron job removing crashed app containers
*/30 * * * * docker ps --filter "ancestor=collec_app" --filter "status=exited" | xargs docker rm
After you removed the cron job, the overload is back, I had to stop nginx and docker again. Kindly edit etc/crontab end then run "crontab etc/crontab" to make changes. I put that enty back it's at the end of the file. -- Written by Thomas Krichel http://openlib.org/home/krichel on his 21240th day.
participants (2)
-
Düben, Christian -
Thomas Krichel