Hi,
the CollEc app is not starting. It apparently stopped working on January 21. What happened that day?
I have already rebuilt and deployed the middleware container. It does not fix the issue. Do you have any idea of what causes the problem?
Kind regards,
Christian
Christian Düben
Doctoral Candidate
Chair of Macroeconomics
Hamburg University
Germany
christian.dueben(a)uni-hamburg.de<mailto:christian.dueben@uni-hamburg.de>
https://www.christian-dueben.com
It looks like we found the recurrent issue of disappearing cron jobs on Helos.
I read that crontab -e is the safer option, less prone to unintended consequences than directly editing the file is.
Christian Düben
Doctoral Candidate
Chair of Macroeconomics
Hamburg University
Germany
christian.dueben(a)uni-hamburg.de
https://www.christian-dueben.com
-----Original Message-----
From: Thomas Krichel <krichel(a)openlib.org>
Sent: Sonntag, 30. Juli 2023 17:08
To: Düben, Christian <christian.dueben(a)uni-hamburg.de>
Subject: Re: [CollEc] helos overload
Düben, Christian writes
> All my cron jobs have been deleted from the cron tab, including the daily updating job.
dear.
>
> Is that due to the fact that we edit the crontab differently? I use "EDITOR=nano crontab -e".
Colud be. I use the file /root/etc/crontab ; crontab /root/etc/crontab
--
Written by Thomas Krichel http://openlib.org/home/krichel on his 21240th day.
it's back.
I have stopped nginx and stopped docker.socket. That seems
to work but leaves CollEc not running.
--
Written by Thomas Krichel http://openlib.org/home/krichel on his 21240th day.
I get mails like these as root. I am adding Christian to the
root email recipients, temporarily.
----- Forwarded message from Cron Daemon <root(a)openlib.org> -----
Return-path: <root(a)openlib.org>
Envelope-to: root(a)openlib.org
Delivery-date: Sun, 30 Jul 2023 13:30:06 +0000
Received: from root by helos.openlib.org with local (Exim 4.95)
(envelope-from <root(a)openlib.org>)
id 1qQ6Uo-0023YW-0c
for root(a)openlib.org;
Sun, 30 Jul 2023 13:30:06 +0000
From: Cron Daemon <root(a)openlib.org>
To: root(a)openlib.org
Subject: Cron <root@helos> docker ps --filter "ancestor=collec_app" --filter "status=created" |
xargs --no-run-if-empty docker rm
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <LOGNAME=root>
Message-Id: <E1qQ6Uo-0023YW-0c(a)helos.openlib.org>
Date: Sun, 30 Jul 2023 13:30:06 +0000
X-SA-Exim-Connect-IP: <locally generated>
X-SA-Exim-Mail-From: root(a)openlib.org
X-SA-Exim-Scanned: No (on helos.openlib.org) SAEximRunCond expanded to false
9753fa54c274
Error response from daemon: No such container: CONTAINER
Error response from daemon: No such container: ID
Error response from daemon: No such container: IMAGE
Error response from daemon: No such container: COMMAND
Error response from daemon: No such container: CREATED
Error response from daemon: No such container: STATUS
Error response from daemon: No such container: PORTS
Error response from daemon: No such container: NAMES
Error response from daemon: You cannot remove a running container 2c03291b3e0024540d110b7ece2086c511da6a48ba02cb55766506b18296f34a. Stop the container before attempting removal or force remove
Error response from daemon: No such container: collec_app
Error response from daemon: No such container: R -e shiny::runApp(…
Error response from daemon: multiple IDs found with provided prefix: 3
Error response from daemon: No such container: seconds
Error response from daemon: No such container: ago
Error response from daemon: No such container: Created
Error response from daemon: You cannot remove a running container 2c03291b3e0024540d110b7ece2086c511da6a48ba02cb55766506b18296f34a. Stop the container before attempting removal or force remove
Error response from daemon: You cannot remove a running container d4c6aff38b37f7b74a8423e096e2768c393ca30a1ecb14c48737b796b608924e. Stop the container before attempting removal or force remove
Error response from daemon: No such container: collec_app
Error response from daemon: No such container: R -e shiny::runApp(…
Error response from daemon: multiple IDs found with provided prefix: 3
Error response from daemon: No such container: seconds
Error response from daemon: No such container: ago
Error response from daemon: No such container: Created
Error response from daemon: You cannot remove a running container d4c6aff38b37f7b74a8423e096e2768c393ca30a1ecb14c48737b796b608924e. Stop the container before attempting removal or force remove
Error response from daemon: You cannot remove a running container 70b9c3e1b04adbc6e7f332a0061f95cad27dd81d5a6c2621c64bb3eb7ec532a8. Stop the container before attempting removal or force remove
Error response from daemon: No such container: collec_app
Error response from daemon: No such container: R -e shiny::runApp(…
Error response from daemon: multiple IDs found with provided prefix: 4
Error response from daemon: No such container: seconds
Error response from daemon: No such container: ago
Error response from daemon: No such container: Created
Error response from daemon: You cannot remove a running container 70b9c3e1b04adbc6e7f332a0061f95cad27dd81d5a6c2621c64bb3eb7ec532a8. Stop the container before attempting removal or force remove
Error response from daemon: You cannot remove a running container c571fb87462380846189e6cf23ef7fe965ad217da46f26316190bdca0020f370. Stop the container before attempting removal or force remove
Error response from daemon: No such container: collec_app
Error response from daemon: No such container: R -e shiny::runApp(…
Error response from daemon: multiple IDs found with provided prefix: 5
Error response from daemon: No such container: seconds
Error response from daemon: No such container: ago
Error response from daemon: No such container: Created
Error response from daemon: You cannot remove a running container c571fb87462380846189e6cf23ef7fe965ad217da46f26316190bdca0020f370. Stop the container before attempting removal or force remove
Error response from daemon: You cannot remove a running container 5bfdfbe665a40b5d284d4777bcae45fd33fd8c18fc2ae54c8ff001dbf8e2a956. Stop the container before attempting removal or force remove
Error response from daemon: No such container: collec_app
Error response from daemon: No such container: R -e shiny::runApp(…
Error response from daemon: multiple IDs found with provided prefix: 5
Error response from daemon: No such container: seconds
Error response from daemon: No such container: ago
Error response from daemon: No such container: Created
Error response from daemon: You cannot remove a running container 5bfdfbe665a40b5d284d4777bcae45fd33fd8c18fc2ae54c8ff001dbf8e2a956. Stop the container before attempting removal or force remove
Error response from daemon: You cannot remove a running container 03dcafd59ee5be69acb2e860d1f381ea8b0e2c7497c07ae8b477925c33d8f8a0. Stop the container before attempting removal or force remove
Error response from daemon: No such container: collec_app
Error response from daemon: No such container: R -e shiny::runApp(…
Error response from daemon: multiple IDs found with provided prefix: 5
Error response from daemon: No such container: seconds
Error response from daemon: No such container: ago
Error response from daemon: No such container: Created
Error response from daemon: You cannot remove a running container 03dcafd59ee5be69acb2e860d1f381ea8b0e2c7497c07ae8b477925c33d8f8a0. Stop the container before attempting removal or force remove
Error response from daemon: No such container: collec_app
Error response from daemon: No such container: R -e shiny::runApp(…
Error response from daemon: multiple IDs found with provided prefix: 6
Error response from daemon: No such container: minutes
Error response from daemon: No such container: ago
Error response from daemon: No such container: Created
Error response from daemon: removal of container focused_turing is already in progress
----- End forwarded message -----
--
Written by Thomas Krichel http://openlib.org/home/krichel on his 21240th day.
helos is oveloaded again.
I killed nginx, now trying to kill docker. It needs
work forme CD.
--
Written by Thomas Krichel http://openlib.org/home/krichel on his 21240th day.
Should/can this be crontabed?
----- Forwarded message from Let's Encrypt Expiry Bot <expiry(a)letsencrypt.org> -----
From: Let's Encrypt Expiry Bot <expiry(a)letsencrypt.org>
To: krichel(a)openlib.org
Subject: Let's Encrypt certificate expiration notice for domain "collec.repec.org"
Hello,
Your certificate (or certificates) for the names listed below will expire in 19 days (on 2023-06-30). Please make sure to renew your certificate before then, or visitors to your web site will encounter errors.
We recommend renewing certificates automatically when they have a third of their total lifetime left. For Let's Encrypt's current 90-day certificates, that means renewing 30 days before expiration. See https://letsencrypt.org/docs/integration-guide/ for details.
collec.repec.org
For details about when we send these emails, please visit: https://letsencrypt.org/docs/expiration-emails/ In particular, note that this reminder email is still sent if you've obtained a slightly different certificate by adding or removing names. If you've replaced this certificate with a newer one that covers more or fewer names than the list above, you may be able to ignore this message.
For any questions or support, please visit: https://community.letsencrypt.org/ Unfortunately, we can't provide support by email.
To learn more about the latest technical and organizational updates from Let's Encrypt, sign up for our newsletter: https://mailchi.mp/letsencrypt.org/fjp6ha1gad
If you are receiving this email in error, unsubscribe at:
http://delivery.letsencrypt.org/track/unsub.php?u=30850198&id=a5cabdefc67e4…
Please note that this would also unsubscribe you from other Let's Encrypt service notices, including expiration reminders for any other certificates.
Regards,
The Let's Encrypt Team
----- End forwarded message -----
--
Written by Thomas Krichel http://openlib.org/home/krichel on his 21190th day.
I have just dogged another crisis.
Seemingly PetalBut make requests at accelerating speed. Then we hit
a problem where I may not be able to kill nginx
root@helos ~ # systemctl stop nginx
Failed to allocate directory watch: Too many open files
After that,
root@helos ~ # ps axf | grep nginx
804800 pts/2 S+ 0:00 \_ grep nginx
so it may not run any more, but repeating the nginx shoutdown shows
the same warning
And then, a few minutes after that, helos starts runing again.
I added
# case sensitive matching
if ($http_user_agent ~ (PetalBot)) {
return 403;
}
to
/etc/nginx/sites-available/collec.repec.org
That may keep them out. Then killing the containers (????)
root@helos ~ # killall -9 containerd-shim-runc-v2
and
root@helos ~ # systemctl start nginx
--
Written by Thomas Krichel http://openlib.org/home/krichel on his 21163rd day.
hi PetalBot gang,
https://collec.repec.org is a shiny app. Recently it has been
overloading the machine. I suspect is your reqests. From what I
understand at my last overload, about 90 minutes ago, you made a
request with less than 3 seconds delay but my robots.txt had
Crawl-delay: 5. I now set it to Disallow: / in an act of dispair.
Can we please respect indications in robot.txt?
--
Written by Thomas Krichel http://openlib.org/home/krichel on his 21163rd day.
Yes, if their platform becomes even moderately successful, the load will likely exceed how much their shiny can handle. I warned them in the call about it, but they did not seem particularly interested.
It could be that Posit’s managed shiny server costing thousands of dollars a year performs better than the free ShinyProxy middleware that we use for CollEc. Yet, even with the paid service the app is unlikely to meet the requirements and come anywhere close to a more professional setup.
I understand why people use shiny apps. I use them myself. If you have a data science background, it is way easier to build an app with shiny than it is to build it with NodeJS. Shiny apps are convenient way of letting users explore a data set or a method. And in an environment with frequent staff turnover where people usually do not have a web development background, like at an economics department, maintenance is easier to ensure with a shiny app than with a more complex structure.
I built an entire teaching platform in shiny and am fed up with that tool. Its poor performance, hidden reactivity layer, and limited capabilities make it annoying to work with in apps beyond simple data or method illustrations. So, my new colleague and I are currently transitioning to a React/ Next.js/ Deno/ Redis/ PostgreSQL stack. Doing that besides research, teaching, R package development, software development for the institute and university, other institute duties, and personal affairs means that it takes at least a few months until I will have time to rewrite CollEc.
I presume it is fine to wait until then, as CollEc appears not to be very popular anyway.
Christian Düben
Doctoral Candidate
Chair of Macroeconomics
Hamburg University
Germany
christian.dueben(a)uni-hamburg.de<mailto:christian.dueben@uni-hamburg.de>
https://www.christian-dueben.com
From: Christian Zimmermann <chuichuiche(a)gmail.com>
Sent: Freitag, 12. Mai 2023 13:53
To: Düben, Christian <christian.dueben(a)uni-hamburg.de>
Subject: Re: [CollEc] Helos down
This is the kind of problem I foresee for the Banque de France site...
Christian Zimmermann
On Fri, May 12, 2023 at 6:52 AM Düben, Christian <christian.dueben(a)uni-hamburg.de<mailto:christian.dueben@uni-hamburg.de>> wrote:
There had been a bunch of exited Docker containers. I cleared them. CollEc should, in the medium term, move away from shiny apps. It is well possible that the current system does not support request bursts from bots.
I currently do not have time for that, but I can schedule it for the end of this year.
Christian Düben
Doctoral Candidate
Chair of Macroeconomics
Hamburg University
Germany
christian.dueben(a)uni-hamburg.de<mailto:christian.dueben@uni-hamburg.de>
https://www.christian-dueben.com
-----Original Message-----
From: CollEc-run <collec-run-bounces(a)lists.openlib.org<mailto:collec-run-bounces@lists.openlib.org>> On Behalf Of Thomas Krichel
Sent: Freitag, 12. Mai 2023 13:22
To: CollEc Run <collec-run(a)lists.openlib.org<mailto:collec-run@lists.openlib.org>>
Subject: Re: [CollEc] Helos down
Thomas Krichel writes
> I can ping it, but not more than that. I can't read email while this
> goes one, but you can to me at editors(a)nep.repec.org<mailto:editors@nep.repec.org>.
It has been up since about 7:50 UTC. Cezar rebooted. It was out of
memory. I think if I had a root window open
There are log entries for the oom killer
root@helos /var/log # grep oom-killer kern.log May 12 03:18:06 helos kernel: [4449730.455246] systemd invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 May 12 04:20:22 helos kernel: [4459782.217108] systemd invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 May 12 06:53:10 helos kernel: [4469843.887742] mutt invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
The web access log, entries of today, contains 4716 entries of petalbot
doing stuff like
114.119.145.116 - - [12/May/2023:11:04:22 +0000] "GET /app_direct/collec_app?_inputs_&navbars=%22tab_Coauthors%22&_values_&g_author=%22ppa246%22 HTTP/1.1" 301 162 "https://ideas.repec.org/f/ppa246.html" "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)"
I supect that petalbot made too many requests.
--
Written by Thomas Krichel http://openlib.org/home/krichel on his 21161st day.
_______________________________________________
CollEc-run mailing list
CollEc-run(a)lists.openlib.org<mailto:CollEc-run@lists.openlib.org>
http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run
_______________________________________________
CollEc-run mailing list
CollEc-run(a)lists.openlib.org<mailto:CollEc-run@lists.openlib.org>
http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run
I can ping it, but not more than that. I can't read email while this goes
one, but you can to me at editors(a)nep.repec.org.
--
Written by Thomas Krichel http://openlib.org/home/krichel on his 21161st day.
