Yes, if their platform becomes even moderately successful, the load will likely exceed how much their shiny can handle. I warned them in the call about it, but they did not seem particularly interested. It could be that Posit’s managed shiny server costing thousands of dollars a year performs better than the free ShinyProxy middleware that we use for CollEc. Yet, even with the paid service the app is unlikely to meet the requirements and come anywhere close to a more professional setup. I understand why people use shiny apps. I use them myself. If you have a data science background, it is way easier to build an app with shiny than it is to build it with NodeJS. Shiny apps are convenient way of letting users explore a data set or a method. And in an environment with frequent staff turnover where people usually do not have a web development background, like at an economics department, maintenance is easier to ensure with a shiny app than with a more complex structure. I built an entire teaching platform in shiny and am fed up with that tool. Its poor performance, hidden reactivity layer, and limited capabilities make it annoying to work with in apps beyond simple data or method illustrations. So, my new colleague and I are currently transitioning to a React/ Next.js/ Deno/ Redis/ PostgreSQL stack. Doing that besides research, teaching, R package development, software development for the institute and university, other institute duties, and personal affairs means that it takes at least a few months until I will have time to rewrite CollEc. I presume it is fine to wait until then, as CollEc appears not to be very popular anyway. Christian Düben Doctoral Candidate Chair of Macroeconomics Hamburg University Germany christian.dueben@uni-hamburg.de<mailto:christian.dueben@uni-hamburg.de> https://www.christian-dueben.com From: Christian Zimmermann <chuichuiche@gmail.com> Sent: Freitag, 12. Mai 2023 13:53 To: Düben, Christian <christian.dueben@uni-hamburg.de> Subject: Re: [CollEc] Helos down This is the kind of problem I foresee for the Banque de France site... Christian Zimmermann On Fri, May 12, 2023 at 6:52 AM Düben, Christian <christian.dueben@uni-hamburg.de<mailto:christian.dueben@uni-hamburg.de>> wrote: There had been a bunch of exited Docker containers. I cleared them. CollEc should, in the medium term, move away from shiny apps. It is well possible that the current system does not support request bursts from bots. I currently do not have time for that, but I can schedule it for the end of this year. Christian Düben Doctoral Candidate Chair of Macroeconomics Hamburg University Germany christian.dueben@uni-hamburg.de<mailto:christian.dueben@uni-hamburg.de> https://www.christian-dueben.com -----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org<mailto:collec-run-bounces@lists.openlib.org>> On Behalf Of Thomas Krichel Sent: Freitag, 12. Mai 2023 13:22 To: CollEc Run <collec-run@lists.openlib.org<mailto:collec-run@lists.openlib.org>> Subject: Re: [CollEc] Helos down Thomas Krichel writes
I can ping it, but not more than that. I can't read email while this goes one, but you can to me at editors@nep.repec.org<mailto:editors@nep.repec.org>.
It has been up since about 7:50 UTC. Cezar rebooted. It was out of memory. I think if I had a root window open There are log entries for the oom killer root@helos /var/log # grep oom-killer kern.log May 12 03:18:06 helos kernel: [4449730.455246] systemd invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 May 12 04:20:22 helos kernel: [4459782.217108] systemd invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 May 12 06:53:10 helos kernel: [4469843.887742] mutt invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 The web access log, entries of today, contains 4716 entries of petalbot doing stuff like 114.119.145.116 - - [12/May/2023:11:04:22 +0000] "GET /app_direct/collec_app?_inputs_&navbars=%22tab_Coauthors%22&_values_&g_author=%22ppa246%22 HTTP/1.1" 301 162 "https://ideas.repec.org/f/ppa246.html" "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)" I supect that petalbot made too many requests. -- Written by Thomas Krichel http://openlib.org/home/krichel on his 21161st day. _______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org<mailto:CollEc-run@lists.openlib.org> http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run _______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org<mailto:CollEc-run@lists.openlib.org> http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run
Düben, Christian writes
I presume it is fine to wait until then,
We will try to make it fine.
as CollEc appears not to be very popular anyway.
Well ... I have kept a root window open and at this time I felt that the machine went slow. root@helos ~ # ps axf | grep -c shim 517 Should tell us why. Looks like petalbot is back. When I run a tail -f on the web log I get tail: inotify cannot be used, reverting to polling: Too many open files ooh dear. I am not sure how long it can hold this load. I am scheduled to go out in 50 minutes and will not get online until about 8:00 GMT. If you can keep a root window open and reboot in case it gets too bad that would be good. -- Written by Thomas Krichel http://openlib.org/home/krichel on his 21161st day.
I am setting the sorry site. I am running as shutdown of nginx root@helos ~ # systemctl stop nginx Failed to allocate directory watch: Too many open files waiting. I have 15 minutes more here. -- Written by Thomas Krichel http://openlib.org/home/krichel on his 21161st day.
Just deny that IP and be done with it. Christian Zimmermann On Fri, May 12, 2023 at 4:47 PM Thomas Krichel <krichel@openlib.org> wrote:
Düben, Christian writes
I presume it is fine to wait until then,
We will try to make it fine.
as CollEc appears not to be very popular anyway.
Well ... I have kept a root window open and at this time I felt that the machine went slow.
root@helos ~ # ps axf | grep -c shim 517
Should tell us why. Looks like petalbot is back. When I run a tail -f on the web log I get
tail: inotify cannot be used, reverting to polling: Too many open files
ooh dear. I am not sure how long it can hold this load. I am scheduled to go out in 50 minutes and will not get online until about 8:00 GMT. If you can keep a root window open and reboot in case it gets too bad that would be good.
-- Written by Thomas Krichel http://openlib.org/home/krichel on his 21161st day.
_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run
Christian Zimmermann writes
Just deny that IP and be done with it.
I will try https://collec.repec.org/robots.txt instead. Service is up again. -- Written by Thomas Krichel http://openlib.org/home/krichel on his 21162nd day.
I meant at the level of the firewall or web server configuration. Abusers are unlikely to obey robots.txt. Christian Zimmermann On Sat, May 13, 2023 at 4:20 AM Thomas Krichel <krichel@openlib.org> wrote:
Christian Zimmermann writes
Just deny that IP and be done with it.
I will try
https://collec.repec.org/robots.txt
instead. Service is up again.
-- Written by Thomas Krichel http://openlib.org/home/krichel on his 21162nd day.
Christian Zimmermann writes
I meant at the level of the firewall or web server configuration. Abusers are unlikely to obey robots.txt.
Hang on. We have no evidence of abuse. Huawei is certainly subject to debate, but there is no evidence I have available that they would not obey robots.txt. -- Written by Thomas Krichel http://openlib.org/home/krichel on his 21162nd day.
Thomas Krichel writes
no evidence I have available that they would not obey robots.txt.
Emminently reasonable, Thomas! I have written a script for you my dear: | #!/usr/bin/python | | import datetime | | fufi = '/var/log/nginx/access.log' | | now = datetime.datetime.now() | now_month = int(datetime.datetime.now().strftime('%m')) | now_year = int(datetime.datetime.now().strftime('%Y')) | | the_file = open(fufi, 'r') | line = the_file.readline() | count = 0 | with open(fufi, 'r') as file: | for line in file: | if 'PetalBot' not in line: | continue | start = line.partition('[')[2] | time = start.partition(']')[0] | day = int(time[0:2]) | hour = int(time[12:14]) | minute = int(time[15:17]) | sec = int(time[18:20]) | time = datetime.datetime(now_year, now_month, day, hour, minute, sec) | diff = str(now - time) | ## just look at the last hour | if not diff.startswith('0:'): | continue | count += 1 | average = 3600 / count | print('one hit every ' + str(int(average)) + ' seconds') This shows a request every 15 seconds. robots.txt says limit is 5. -- Written by Thomas Krichel http://openlib.org/home/krichel on his 21162nd day.
It's was sick again, I managed to reboot having a root window open. If at possible, Christian should call me on Skype or WhatsUp or Telegram to discuss what we should do next. -- Written by Thomas Krichel http://openlib.org/home/krichel on his 21163rd day.
After reboot my script showed a call from Huawei every 3 seconds. I moved to disallow them. It may take time for them to notice the change. -- Written by Thomas Krichel http://openlib.org/home/krichel on his 21163rd day.
root@helos ~ # systemctl stop nginx root@helos ~ # killall -9 containerd-shim-runc-v2 after the kill some new shims came up still. Sorry folks. I can't afford to bother Cezar, nor to have Edvard waiting for me. Machine feels fine now. -- Written by Thomas Krichel http://openlib.org/home/krichel on his 21161st day.
participants (3)
-
Christian Zimmermann -
Düben, Christian -
Thomas Krichel