The CollEc app is offline. The docker images that the app's components use are static. No software, packages, or modules are updated. Therefore, the problem is unlikely to originate from within the app. The issue appears to originate at the web server level. Any ideas what might be going on? Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de<mailto:christian.dueben@uni-hamburg.de> http://www.christian-dueben.com
Düben, Christian writes
The CollEc app is offline.
It says it's a proxy error.
The docker images that the app's components use are static. No software, packages, or modules are updated. Therefore, the problem is unlikely to originate from within the app.
Sure but the app will still use some o/s level application, I suspect. I updated the o/s.
The issue appears to originate at the web server level. Any ideas what might be going on?
I restarted it many times to get the gitweb to run. I installed mod_perl. I did not change your configuration. Indeed root@darni /etc/apache2/sites-available # GET http://localhost:8080 Server closed connection without sending any data back at /usr/share/perl5/Net/HTTP/Methods.pm line 391. root@darni /etc/apache2/sites-available # telnet localhost 8080 Trying 127.0.0.1... Connected to localhost.localdomain. Escape character is '^]'. Connection closed by foreign host. Is the application running? -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Thomas Krichel writes
Is the application running?
Well it is now. I guess Christian started it. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
I indeed managed to restart it. Simply restarting the Docker containers did not solve the problem. I had to entirely get rid of the containers and restart the Docker deamon itself. It looks like it started malfunctioning following some o/s change on December 23. I am going to monitor the app over the next few days to see if any of the processes deviates from its expected behavior. How are we going to handle the migration to Helos, by the way? As long as the server runs at 100 percent CPU load, it is going to be difficult to run and test anything on it. Regarding GraphEc, I now plan on setting it up with a NodeJS backend rather than as a Shiny application with an R backend. Unlike CollEc, it does not require the functionalities of R or Python combined with C++. It exclusively reads pre-computed results from a SQL database and displays them. With front- and backend written in JavaScript, I can make use of pre-initialization. I.e. users do not have to wait for the app to initialize, as they do in the case of CollEc where ShinyProxy might take a few seconds to spin up a Docker container. However, I need to focus on my research during the week and only have time to work on my JavaScript skills and apps during the weekend. It might, therefore, take a few months to produce a satisfactory JavaScript GraphEc. Thanks for looking into the CollEc app crash on a Sunday. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Sonntag, 27. Dezember 2020 16:57 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: [CollEc] CollEc App Offline Thomas Krichel writes
Is the application running?
Well it is now. I guess Christian started it. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
While we are discussing CollEc, I am still looking for the new data. And no, rsync is not a solution for me, but ftp, http and https are fine from wherever suits you. Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist On Sun, 27 Dec 2020, D�ben, Christian wrote:
I indeed managed to restart it.
Simply restarting the Docker containers did not solve the problem. I had to entirely get rid of the containers and restart the Docker deamon itself. It looks like it started malfunctioning following some o/s change on December 23.
I am going to monitor the app over the next few days to see if any of the processes deviates from its expected behavior.
How are we going to handle the migration to Helos, by the way? As long as the server runs at 100 percent CPU load, it is going to be difficult to run and test anything on it.
Regarding GraphEc, I now plan on setting it up with a NodeJS backend rather than as a Shiny application with an R backend. Unlike CollEc, it does not require the functionalities of R or Python combined with C++. It exclusively reads pre-computed results from a SQL database and displays them. With front- and backend written in JavaScript, I can make use of pre-initialization. I.e. users do not have to wait for the app to initialize, as they do in the case of CollEc where ShinyProxy might take a few seconds to spin up a Docker container. However, I need to focus on my research during the week and only have time to work on my JavaScript skills and apps during the weekend. It might, therefore, take a few months to produce a satisfactory JavaScript GraphEc.
Thanks for looking into the CollEc app crash on a Sunday.
Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com
-----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Sonntag, 27. Dezember 2020 16:57 To: D�ben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: [CollEc] CollEc App Offline
Thomas Krichel writes
Is the application running?
Well it is now. I guess Christian started it.
--
Cheers,
Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run
Thomas, can you make it available through the web server? Since I crashed the web server a while ago, I am not eager to play around with it anymore. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Christian Zimmermann <zimmermann@stlouisfed.org> Sent: Sonntag, 27. Dezember 2020 18:38 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Thomas Krichel <krichel@openlib.org>; CollEc Run <collec-run@lists.openlib.org> Subject: Re: [CollEc] CollEc App Offline While we are discussing CollEc, I am still looking for the new data. And no, rsync is not a solution for me, but ftp, http and https are fine from wherever suits you. Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist On Sun, 27 Dec 2020, D�ben, Christian wrote:
I indeed managed to restart it.
Simply restarting the Docker containers did not solve the problem. I had to entirely get rid of the containers and restart the Docker deamon itself. It looks like it started malfunctioning following some o/s change on December 23.
I am going to monitor the app over the next few days to see if any of the processes deviates from its expected behavior.
How are we going to handle the migration to Helos, by the way? As long as the server runs at 100 percent CPU load, it is going to be difficult to run and test anything on it.
Regarding GraphEc, I now plan on setting it up with a NodeJS backend rather than as a Shiny application with an R backend. Unlike CollEc, it does not require the functionalities of R or Python combined with C++. It exclusively reads pre-computed results from a SQL database and displays them. With front- and backend written in JavaScript, I can make use of pre-initialization. I.e. users do not have to wait for the app to initialize, as they do in the case of CollEc where ShinyProxy might take a few seconds to spin up a Docker container. However, I need to focus on my research during the week and only have time to work on my JavaScript skills and apps during the weekend. It might, therefore, take a few months to produce a satisfactory JavaScript GraphEc.
Thanks for looking into the CollEc app crash on a Sunday.
Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com
-----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Sonntag, 27. Dezember 2020 16:57 To: D�ben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: [CollEc] CollEc App Offline
Thomas Krichel writes
Is the application running?
Well it is now. I guess Christian started it.
--
Cheers,
Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run
Düben, Christian writes
Thomas, can you make it available through the web server? Since I crashed the web server a while ago, I am not eager to play around with it anymore.
I will take care of that. Myself and the webserver ... that's what I have left to play with. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Great. Thanks. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Sonntag, 27. Dezember 2020 19:42 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org>; Christian Zimmermann <zimmermann@stlouisfed.org> Subject: Re: [CollEc] CollEc App Offline Düben, Christian writes
Thomas, can you make it available through the web server? Since I crashed the web server a while ago, I am not eager to play around with it anymore.
I will take care of that. Myself and the webserver ... that's what I have left to play with. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Christian Zimmermann writes
While we are discussing CollEc, I am still looking for the new data. And no, rsync is not a solution for me, but ftp, http and https are fine from wherever suits you.
http://data.collec.repec.org -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Thanks. I notice that the closeness number is now a millionth of what it was before. Just checking that this is correct. Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist On Tue, 29 Dec 2020, Thomas Krichel wrote:
Christian Zimmermann writes
While we are discussing CollEc, I am still looking for the new data. And no, rsync is not a solution for me, but ftp, http and https are fine from wherever suits you.
--
Cheers,
Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
That looks about right. Thomas, how do you get closeness values larger than 1? Do you scale the results by some factor? Or is the distance between co-authors not 1 in your case? With the closeness equation of C(v) = 1 / (\sum_{i \neq v} d(v, i)) where d(v, i) is the length of the shortest cost path between author v and author i and d(v, i) \geq 1, any closeness value should be between 0 and 1. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of Christian Zimmermann Sent: Dienstag, 29. Dezember 2020 19:29 To: Thomas Krichel <krichel@openlib.org> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: [CollEc] CollEc App Offline Thanks. I notice that the closeness number is now a millionth of what it was before. Just checking that this is correct. Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist On Tue, 29 Dec 2020, Thomas Krichel wrote:
Christian Zimmermann writes
While we are discussing CollEc, I am still looking for the new data. And no, rsync is not a solution for me, but ftp, http and https are fine from wherever suits you.
--
Cheers,
Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run
Düben, Christian writes
Thomas, how do you get closeness values larger than 1?
I use common sense. The closeness of a person is the average distance from one to any other, for all others. Since the distance between any pair is at least one, the average must be larger than one.
Do you scale the results by some factor?
No.
Or is the distance between co-authors not 1 in your case?
It is.
With the closeness equation of C(v) = 1 / (\sum_{i \neq v} d(v, i)) where d(v, i) is the length of the shortest cost path between author v and author i and d(v, i) \geq 1, any closeness value should be between 0 and 1.
There is something counter-intuitive in this approach. I said many times, if we don't use a binary model, we will leave our users confused. Alternative weighing schemes should be used to filter out from binary short paths that have the same length. However, the way you do that will not have any impact on the closeness, as expressed in my common sense understanding. It will only impact the betweenness. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
I do use binary paths here. Weighted paths are not exported. As far as I understand it, there is one fixed closeness formula: 1 / (sum d). Just like the definitions of the mean, the variance, and other statistical measures are fixed. Your alternative measure appears to be (sum d) / N. Thus, multiplying my results by d^2 / N should produce your results. This is not about different path defintions. The former CollEc's closeness values are simply a scaled version of the new CollEc's closeness values. How does this make my results counter-intuitive? There is nothing different about the underlying paths. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Mittwoch, 30. Dezember 2020 04:27 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: [CollEc] CollEc App Offline Düben, Christian writes
Thomas, how do you get closeness values larger than 1?
I use common sense. The closeness of a person is the average distance from one to any other, for all others. Since the distance between any pair is at least one, the average must be larger than one.
Do you scale the results by some factor?
No.
Or is the distance between co-authors not 1 in your case?
It is.
With the closeness equation of C(v) = 1 / (\sum_{i \neq v} d(v, i)) where d(v, i) is the length of the shortest cost path between author v and author i and d(v, i) \geq 1, any closeness value should be between 0 and 1.
There is something counter-intuitive in this approach. I said many times, if we don't use a binary model, we will leave our users confused. Alternative weighing schemes should be used to filter out from binary short paths that have the same length. However, the way you do that will not have any impact on the closeness, as expressed in my common sense understanding. It will only impact the betweenness. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Sorry, I meant multiplying by (sum d)^2 / N, of course. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of Düben, Christian Sent: Mittwoch, 30. Dezember 2020 10:46 To: Thomas Krichel <krichel@openlib.org> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: [CollEc] CollEc App Offline I do use binary paths here. Weighted paths are not exported. As far as I understand it, there is one fixed closeness formula: 1 / (sum d). Just like the definitions of the mean, the variance, and other statistical measures are fixed. Your alternative measure appears to be (sum d) / N. Thus, multiplying my results by d^2 / N should produce your results. This is not about different path defintions. The former CollEc's closeness values are simply a scaled version of the new CollEc's closeness values. How does this make my results counter-intuitive? There is nothing different about the underlying paths. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Mittwoch, 30. Dezember 2020 04:27 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: [CollEc] CollEc App Offline Düben, Christian writes
Thomas, how do you get closeness values larger than 1?
I use common sense. The closeness of a person is the average distance from one to any other, for all others. Since the distance between any pair is at least one, the average must be larger than one.
Do you scale the results by some factor?
No.
Or is the distance between co-authors not 1 in your case?
It is.
With the closeness equation of C(v) = 1 / (\sum_{i \neq v} d(v, i)) where d(v, i) is the length of the shortest cost path between author v and author i and d(v, i) \geq 1, any closeness value should be between 0 and 1.
There is something counter-intuitive in this approach. I said many times, if we don't use a binary model, we will leave our users confused. Alternative weighing schemes should be used to filter out from binary short paths that have the same length. However, the way you do that will not have any impact on the closeness, as expressed in my common sense understanding. It will only impact the betweenness. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel _______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run
And N would happen to be close to a million? Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist On Wed, 30 Dec 2020, D�ben, Christian wrote:
I do use binary paths here. Weighted paths are not exported.
As far as I understand it, there is one fixed closeness formula: 1 / (sum d). Just like the definitions of the mean, the variance, and other statistical measures are fixed. Your alternative measure appears to be (sum d) / N. Thus, multiplying my results by d^2 / N should produce your results. This is not about different path defintions. The former CollEc's closeness values are simply a scaled version of the new CollEc's closeness values. How does this make my results counter-intuitive? There is nothing different about the underlying paths.
Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com
-----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Mittwoch, 30. Dezember 2020 04:27 To: D�ben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: [CollEc] CollEc App Offline
D�ben, Christian writes
Thomas, how do you get closeness values larger than 1?
I use common sense. The closeness of a person is the average distance from one to any other, for all others. Since the distance between any pair is at least one, the average must be larger than one.
Do you scale the results by some factor?
No.
Or is the distance between co-authors not 1 in your case?
It is.
With the closeness equation of C(v) = 1 / (\sum_{i \neq v} d(v, i)) where d(v, i) is the length of the shortest cost path between author v and author i and d(v, i) \geq 1, any closeness value should be between 0 and 1.
There is something counter-intuitive in this approach.
I said many times, if we don't use a binary model, we will leave our users confused. Alternative weighing schemes should be used to filter out from binary short paths that have the same length. However, the way you do that will not have any impact on the closeness, as expressed in my common sense understanding. It will only impact the betweenness.
--
Cheers,
Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run
No, N is the number of authors in the respective graph. The main graph currently entails 45,594 people. Let me illustrate the results with an example. Assume some author is located at an average distance of 4 from other authors in the graph. Then the total distance is: sum d = 45594 * 4 = 182376. Plugging this into the closeness formula, C = 1 / sum d, produces a closeness value of 1 / 182376 = 5.483178e-06. Transforming it to the former CollEc's definition means multiplying it by (sum d)^2 / N: (182376^2 / 45594) * 5.483178e-06 = 4. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Christian Zimmermann <zimmermann@stlouisfed.org> Sent: Mittwoch, 30. Dezember 2020 16:22 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Thomas Krichel <krichel@openlib.org>; CollEc Run <collec-run@lists.openlib.org> Subject: Re: [CollEc] CollEc App Offline And N would happen to be close to a million? Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist On Wed, 30 Dec 2020, D�ben, Christian wrote:
I do use binary paths here. Weighted paths are not exported.
As far as I understand it, there is one fixed closeness formula: 1 / (sum d). Just like the definitions of the mean, the variance, and other statistical measures are fixed. Your alternative measure appears to be (sum d) / N. Thus, multiplying my results by d^2 / N should produce your results. This is not about different path defintions. The former CollEc's closeness values are simply a scaled version of the new CollEc's closeness values. How does this make my results counter-intuitive? There is nothing different about the underlying paths.
Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com
-----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Mittwoch, 30. Dezember 2020 04:27 To: D�ben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: [CollEc] CollEc App Offline
D�ben, Christian writes
Thomas, how do you get closeness values larger than 1?
I use common sense. The closeness of a person is the average distance from one to any other, for all others. Since the distance between any pair is at least one, the average must be larger than one.
Do you scale the results by some factor?
No.
Or is the distance between co-authors not 1 in your case?
It is.
With the closeness equation of C(v) = 1 / (\sum_{i \neq v} d(v, i)) where d(v, i) is the length of the shortest cost path between author v and author i and d(v, i) \geq 1, any closeness value should be between 0 and 1.
There is something counter-intuitive in this approach.
I said many times, if we don't use a binary model, we will leave our users confused. Alternative weighing schemes should be used to filter out from binary short paths that have the same length. However, the way you do that will not have any impact on the closeness, as expressed in my common sense understanding. It will only impact the betweenness.
--
Cheers,
Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run
Düben, Christian writes
How are we going to handle the migration to Helos, by the way? As long as the server runs at 100 percent CPU load, it is going to be difficult to run and test anything on it.
It runs at this beacues it calculate the values using my techology. I suggest we fix a date, say 2021-02-01. On that data I stop the calculations, move file to a legacy server space on helos, and make them avalibale at legacy.collec.repec.org. Then you can take over. helos is a faster machine than darni.
for the app to initialize, as they do in the case of CollEc where ShinyProxy might take a few seconds to spin up a Docker container.
I suspect that's the main problem with it,
However, I need to focus on my research during the week and only have time to work on my JavaScript skills and apps during the weekend. It might, therefore, take a few months to produce a satisfactory JavaScript GraphEc.
Aronud RePEc, rothing ever gets done fast. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Migrating the app on February 01, 2021 is fine with me. However, I have a few requests regarding the migration. 1. Can you give me root access on Helos, as you did on Darni? 2. Can you temporarily redirect collec.repec.org to app.collec.repec.org on the day of migration? 3. Can you set up a test URL again, e.g. test.collec.repec.org, with which I can test the app on Helos on the day of migration? 4. After migrating CollEc, I still need the ras_mans_texts xml file in /mnt/helos/home/icanis/icanis/opt/input to be generated once a day. It is the central input to the results available via the web application. On Darni, the web application does not consume a lot of computational resources. After migration, there is going to be plenty of space for further RePEc services on Helos. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Mittwoch, 6. Januar 2021 05:32 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: [CollEc] CollEc App Offline Düben, Christian writes
How are we going to handle the migration to Helos, by the way? As long as the server runs at 100 percent CPU load, it is going to be difficult to run and test anything on it.
It runs at this beacues it calculate the values using my techology. I suggest we fix a date, say 2021-02-01. On that data I stop the calculations, move file to a legacy server space on helos, and make them avalibale at legacy.collec.repec.org. Then you can take over. helos is a faster machine than darni.
for the app to initialize, as they do in the case of CollEc where ShinyProxy might take a few seconds to spin up a Docker container.
I suspect that's the main problem with it,
However, I need to focus on my research during the week and only have time to work on my JavaScript skills and apps during the weekend. It might, therefore, take a few months to produce a satisfactory JavaScript GraphEc.
Aronud RePEc, rothing ever gets done fast. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Düben, Christian writes
1. Can you give me root access on Helos, as you did on Darni?
root@helos ~ # grep Due .ssh/authorized_keys ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyncBxWKif1ki5K+8q0XbJYVqulW4cRfKuEGpVd2k0tPfZY6f17sg4p7ZI1+onW1IBN4kcFsawYg6fhridoDxrx7kM7ByEaLQ8L9TR0IrTnIs43A6qDXAb8WBnoCBkERw2k+NXyQV0zGVjQ/9mlWPXNz+T/J8mZ4OCc6bokMmm3Q9/Dvkvf+Vs8E/lceb9pNIl/ZtnXWMi8jA/VIilDa/oRxR3BGsGD+WO5vJz8US7n1XldicaT4lsyQgOLUNpOR8T46KO0KU2u772qvU7V8AOoN3VDnT8/BrEnQn0E3FqsDC07cDD9PDKrnEA162dqfrZwDlAzZT0FEVthXeBSA7Pw== Christian.Dueben@uni-hamburg.de BTW, I prefer ed25519 keys for brevity.
2. Can you temporarily redirect collec.repec.org to app.collec.repec.org on the day of migration?
We can make a DNS change. If you are familiar with bind, I'll be happy to give you access to the DNS acccount and show you how it works.
3. Can you set up a test URL again, e.g. test.collec.repec.org, with which I can test the app on Helos on the day of migration?
I created helos.collec.repec.org DNS entries.
4. After migrating CollEc, I still need the ras_mans_texts xml file in /mnt/helos/home/icanis/icanis/opt/input to be generated once a day. It is the central input to the results available via the web application.
As the file name implies, this already lives on helos.
On Darni, the web application does not consume a lot of computational resources. After migration, there is going to be plenty of space for further RePEc services on Helos.
Well, I will do some ArchEc work there that will consume disk space, but not be cpu intensive. Sure there should be resources for GraphEc. But we need to mention the sponsor. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
I generated an Ed25519 key and uploaded the public key file to Darni: /home/icanis/Helos_Public_Key. Feel free to move or to delete that file. Helos' IP address is 95.216.245.19, correct? I have no idea how to configure or modify DNS servers. I still need to learn how to do that. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Donnerstag, 7. Januar 2021 02:31 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: [CollEc] CollEc App Offline Düben, Christian writes
1. Can you give me root access on Helos, as you did on Darni?
root@helos ~ # grep Due .ssh/authorized_keys ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyncBxWKif1ki5K+8q0XbJYVqulW4cRfKuEGpVd2k0tPfZY6f17sg4p7ZI1+onW1IBN4kcFsawYg6fhridoDxrx7kM7ByEaLQ8L9TR0IrTnIs43A6qDXAb8WBnoCBkERw2k+NXyQV0zGVjQ/9mlWPXNz+T/J8mZ4OCc6bokMmm3Q9/Dvkvf+Vs8E/lceb9pNIl/ZtnXWMi8jA/VIilDa/oRxR3BGsGD+WO5vJz8US7n1XldicaT4lsyQgOLUNpOR8T46KO0KU2u772qvU7V8AOoN3VDnT8/BrEnQn0E3FqsDC07cDD9PDKrnEA162dqfrZwDlAzZT0FEVthXeBSA7Pw== Christian.Dueben@uni-hamburg.de BTW, I prefer ed25519 keys for brevity.
2. Can you temporarily redirect collec.repec.org to app.collec.repec.org on the day of migration?
We can make a DNS change. If you are familiar with bind, I'll be happy to give you access to the DNS acccount and show you how it works.
3. Can you set up a test URL again, e.g. test.collec.repec.org, with which I can test the app on Helos on the day of migration?
I created helos.collec.repec.org DNS entries.
4. After migrating CollEc, I still need the ras_mans_texts xml file in /mnt/helos/home/icanis/icanis/opt/input to be generated once a day. It is the central input to the results available via the web application.
As the file name implies, this already lives on helos.
On Darni, the web application does not consume a lot of computational resources. After migration, there is going to be plenty of space for further RePEc services on Helos.
Well, I will do some ArchEc work there that will consume disk space, but not be cpu intensive. Sure there should be resources for GraphEc. But we need to mention the sponsor. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Düben, Christian writes
I generated an Ed25519 key and uploaded the public key file to Darni: /home/icanis/Helos_Public_Key. Feel free to move or to delete that file.
I have authorized that key for root@darni icanis@darni and root@helos.
Helos' IP address is 95.216.245.19, correct?
Not really. helos, like all my other machine is in the openlib.org domain. There is a IP4 and at least one IP6 for each. krichel@amorp:~$ host helos.openlib.org helos.openlib.org has address 95.216.245.19 helos.openlib.org has IPv6 address 2a01:4f9:2b:276c::2 Hetzner give you a whole IP6 subnet for a server, so there could be more IP6 addresses.
I have no idea how to configure or modify DNS servers. I still need to learn how to do that.
It's not rock science but we can skip this for now and I will do it. There is some news. I broke a tooth. I'm cutting my NYC stay short. I'm now supposed to be back on the 27th in Nsk, but I'm afraid there may be more medical problems, so the 1st of February may not happen as planned. I will still try. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Thanks. Addressing health issues has, of course, a higher priority than migrating the websites. Take your time and let me know in case you prefer to postpone it. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Mittwoch, 13. Januar 2021 03:11 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: [CollEc] CollEc App Offline Düben, Christian writes
I generated an Ed25519 key and uploaded the public key file to Darni: /home/icanis/Helos_Public_Key. Feel free to move or to delete that file.
I have authorized that key for root@darni icanis@darni and root@helos.
Helos' IP address is 95.216.245.19, correct?
Not really. helos, like all my other machine is in the openlib.org domain. There is a IP4 and at least one IP6 for each. krichel@amorp:~$ host helos.openlib.org helos.openlib.org has address 95.216.245.19 helos.openlib.org has IPv6 address 2a01:4f9:2b:276c::2 Hetzner give you a whole IP6 subnet for a server, so there could be more IP6 addresses.
I have no idea how to configure or modify DNS servers. I still need to learn how to do that.
It's not rock science but we can skip this for now and I will do it. There is some news. I broke a tooth. I'm cutting my NYC stay short. I'm now supposed to be back on the 27th in Nsk, but I'm afraid there may be more medical problems, so the 1st of February may not happen as planned. I will still try. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Düben, Christian writes
Addressing health issues has, of course, a higher priority than migrating the websites.
Historically, every time I flew to Nsk in the winter, I had a cold in the week that followed it ... except last time, when I limited the flight to Moscow and took the train from there. With a departure on 23rd, the first of February would fall right into sick time. But I hope I will be lucky again. In that case I start to think that the train is a smart choice for me. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Good choice. I also prefer trains to planes. I am ok with any day in February, preferably not Valentine's Day though. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Mittwoch, 13. Januar 2021 18:16 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: [CollEc] CollEc App Offline Düben, Christian writes
Addressing health issues has, of course, a higher priority than migrating the websites.
Historically, every time I flew to Nsk in the winter, I had a cold in the week that followed it ... except last time, when I limited the flight to Moscow and took the train from there. With a departure on 23rd, the first of February would fall right into sick time. But I hope I will be lucky again. In that case I start to think that the train is a smart choice for me. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Düben, Christian writes
Take your time and let me know in case you prefer to postpone it.
I did arrive and I'm fine. I gave blood for the PSA test, but now I need to head to the village to give Segus a key. I'm working on a legacy site. This will have the RAS data from up-to-today. So we don't update the data at http://legacy.collec.repec.org. This will be written on all pages. I'm building a legacy site at http://legacy.collec.repec.org (lecol). This is supposed to use the old code. I will slowly update it so that it will show the RAS network as of today. At this time, http://collec.repec.org will be the same as the legacy. We will change that when you have migrated. lecol does not show now, I don't know why it does not seem to want to see the virtual host. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
The migration is going well so far. Stable Ubuntu is much less prone to compatibility errors than stable Debian is. Regarding the migration I would like to address two issues: 1. I see that Helos has been up for 601 days. Can we (upgrade) and reboot the machine, i.e. run the following common upgrading lines? Or are you currently running any scripts that cannot be interrupted? sudo apt-get update sudo apt-get dist-upgrade sudo apt-get autoremove reboot 2. What do you think about adding a SSL certificate? Let's Encrypt certificates are free. Google already punishes HTTP websites by assigning them a lower rank than HTTPS websites in search results. And the company intends to take further steps against HTTP. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Montag, 1. Februar 2021 06:58 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: the grand migration Düben, Christian writes
Take your time and let me know in case you prefer to postpone it.
I did arrive and I'm fine. I gave blood for the PSA test, but now I need to head to the village to give Segus a key. I'm working on a legacy site. This will have the RAS data from up-to-today. So we don't update the data at http://legacy.collec.repec.org. This will be written on all pages. I'm building a legacy site at http://legacy.collec.repec.org (lecol). This is supposed to use the old code. I will slowly update it so that it will show the RAS network as of today. At this time, http://collec.repec.org will be the same as the legacy. We will change that when you have migrated. lecol does not show now, I don't know why it does not seem to want to see the virtual host. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Düben, Christian writes
1. I see that Helos has been up for 601 days. Can we (upgrade) and reboot the machine, i.e. run the following common upgrading lines? Or are you currently running any scripts that cannot be interrupted? sudo apt-get update sudo apt-get dist-upgrade sudo apt-get autoremove reboot
sure you can do that and reboot.
2. What do you think about adding a SSL certificate? Let's Encrypt certificates are free.
Yeah but you have to reinstall it. I have been been bothering with this.
Google already punishes HTTP websites by assigning them a lower rank than HTTPS websites in search results. And the company intends to take further steps against HTTP.
Well that's a rumour. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
The migration is almost done. What is missing is the web server part. Could you point collec.repec.org and app.collec.repec.org to port 8080 and make sure that web sockets are enabled? Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Montag, 1. Februar 2021 14:55 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: the grand migration Düben, Christian writes
1. I see that Helos has been up for 601 days. Can we (upgrade) and reboot the machine, i.e. run the following common upgrading lines? Or are you currently running any scripts that cannot be interrupted? sudo apt-get update sudo apt-get dist-upgrade sudo apt-get autoremove reboot
sure you can do that and reboot.
2. What do you think about adding a SSL certificate? Let's Encrypt certificates are free.
Yeah but you have to reinstall it. I have been been bothering with this.
Google already punishes HTTP websites by assigning them a lower rank than HTTPS websites in search results. And the company intends to take further steps against HTTP.
Well that's a rumour. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Düben, Christian writes
The migration is almost done. What is missing is the web server part. Could you point collec.repec.org and app.collec.repec.org to port 8080 and make sure that web sockets are enabled?
<VirtualHost *:80> ServerName collec.repec.org ServerAlias app.collec.openlib.org ErrorLog /var/log/apache2/collec.proxy-error.log CustomLog /var/log/apache2/collec-access.log combined UseCanonicalName Off ProxyPass / http://localhost:8080/ RewriteEngine on RewriteCond %{HTTP:Upgrade} websocket [NC] RewriteCond %{HTTP:Connection} upgrade [NC] RewriteRule ^/?(.*) ws://localhost:8080/$1 [P,L] </VirtualHost> This is pretty much what we had on darni, bar the server name. The modules look enabled on helos. With this, browser will simply go to their own localhost, where the collection files. I have to revert. My lover is crying I have to look after her. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Thomas Krichel writes
<VirtualHost *:80> ServerName collec.repec.org ServerAlias app.collec.openlib.org ErrorLog /var/log/apache2/collec.proxy-error.log CustomLog /var/log/apache2/collec-access.log combined UseCanonicalName Off ProxyPass / http://localhost:8080/ RewriteEngine on RewriteCond %{HTTP:Upgrade} websocket [NC] RewriteCond %{HTTP:Connection} upgrade [NC] RewriteRule ^/?(.*) ws://localhost:8080/$1 [P,L] </VirtualHost>
Changed that to <VirtualHost *:80> ServerName collec.repec.org ServerAlias app.collec.openlib.org ErrorLog /var/log/apache2/collec.proxy-error.log CustomLog /var/log/apache2/collec-access.log combined UseCanonicalName Off ProxyPass / http://collec.repec.org:8080/ RewriteEngine on RewriteRule ^/bin/search.fcgi(.*) /var/www/icanis/cgi-bin/search.fcgi$1 [L] RewriteRule ^/nodes/(.*) http://legacy.collec.repec.org/nodes/$1 [R] RewriteRule ^/rank/(.*) http://legacy.collec.repec.org/rank/$1 [R] RewriteRule ^/symplectic_icon.png$ http://legacy.collec.repec.org/symplectic_icon.png [R] RewriteRule ^/doc.html$ http://legacy.collec.repec.org/doc.html [R] RewriteRule ^/css/icanis.css$ http://legacy.collec.repec.org/css/icanis.css [R] RewriteCond %{HTTP:Upgrade} websocket [NC] RewriteCond %{HTTP:Connection} upgrade [NC] RewriteRule ^/?(.*) ws://localhost:8080/$1 [P,L] </VirtualHost> The critical change is in the ProxyPass line. Now it lauches but with an ugly URL that has the port. The additional lines send all the robots to the legacy site. The legacy site needs a bit of wordsmithing but I suggest to keep it this way, change the RAS data on the first of January and then have the site update over time the path to that fixed network. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Thanks. Could we get rid of the 8080 in the URL? And could you point app.collec.repec.org, the URL that users are familiar with, to the app on Helos? I already deactivated the updating routine on Darni. What robots are you sending to the legacy site? Search engine robots? Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Dienstag, 2. Februar 2021 16:22 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: the grand migration Thomas Krichel writes
<VirtualHost *:80> ServerName collec.repec.org ServerAlias app.collec.openlib.org ErrorLog /var/log/apache2/collec.proxy-error.log CustomLog /var/log/apache2/collec-access.log combined UseCanonicalName Off ProxyPass / http://localhost:8080/ RewriteEngine on RewriteCond %{HTTP:Upgrade} websocket [NC] RewriteCond %{HTTP:Connection} upgrade [NC] RewriteRule ^/?(.*) ws://localhost:8080/$1 [P,L] </VirtualHost>
Changed that to <VirtualHost *:80> ServerName collec.repec.org ServerAlias app.collec.openlib.org ErrorLog /var/log/apache2/collec.proxy-error.log CustomLog /var/log/apache2/collec-access.log combined UseCanonicalName Off ProxyPass / http://collec.repec.org:8080/ RewriteEngine on RewriteRule ^/bin/search.fcgi(.*) /var/www/icanis/cgi-bin/search.fcgi$1 [L] RewriteRule ^/nodes/(.*) http://legacy.collec.repec.org/nodes/$1 [R] RewriteRule ^/rank/(.*) http://legacy.collec.repec.org/rank/$1 [R] RewriteRule ^/symplectic_icon.png$ http://legacy.collec.repec.org/symplectic_icon.png [R] RewriteRule ^/doc.html$ http://legacy.collec.repec.org/doc.html [R] RewriteRule ^/css/icanis.css$ http://legacy.collec.repec.org/css/icanis.css [R] RewriteCond %{HTTP:Upgrade} websocket [NC] RewriteCond %{HTTP:Connection} upgrade [NC] RewriteRule ^/?(.*) ws://localhost:8080/$1 [P,L] </VirtualHost> The critical change is in the ProxyPass line. Now it lauches but with an ugly URL that has the port. The additional lines send all the robots to the legacy site. The legacy site needs a bit of wordsmithing but I suggest to keep it this way, change the RAS data on the first of January and then have the site update over time the path to that fixed network. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Düben, Christian writes
Could we get rid of the 8080 in the URL?
Sure if somebody can come up with the right idea.
And could you point app.collec.repec.org, the URL that users are familiar with, to the app on Helos? I already deactivated the updating routine on Darni.
I think that just requires a DNS change. I have done that.
What robots are you sending to the legacy site? Search engine robots?
Anybody who is using the legacy paths, as suggested in the config. At this time, yes, there is the usual bulk of usage by robots. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Can you not just use essentially the same setting that you used on Darni? There it worked without the 8080 in the URL. app.collec.repec.org does not point to the app anymore. It just points to an Apache2 Debian default page. Why should robots not visit the app? Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Dienstag, 2. Februar 2021 17:20 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: the grand migration Düben, Christian writes
Could we get rid of the 8080 in the URL?
Sure if somebody can come up with the right idea.
And could you point app.collec.repec.org, the URL that users are familiar with, to the app on Helos? I already deactivated the updating routine on Darni.
I think that just requires a DNS change. I have done that.
What robots are you sending to the legacy site? Search engine robots?
Anybody who is using the legacy paths, as suggested in the config. At this time, yes, there is the usual bulk of usage by robots. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Düben, Christian writes
Can you not just use essentially the same setting that you used on Darni? There it worked without the 8080 in the URL.
Guess what I tried ... I don't know why that is not working. It maybe that the app is not installed in the same way? It maybe because the apache version is not the same? It maybe /etc/hosts, or something in the DNS?
app.collec.repec.org does not point to the app anymore. It just points to an Apache2 Debian default page.
ooops ... fixed.
Why should robots not visit the app?
By "app" you mean the official new CollEc? I'm not excluding any visits to it. I just redirect URLs that went to the legacy site to legacy as they will not be found on the app. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
What exactly does not work when you set "ProxyPass / http://localhost:8080/"? On Helos, I deploy the app via ShinyProxy 2.4.3 whereas I used ShinyProxy 2.4.1 on Darni. However, nothing relevant to our application should have changed. I went through the release notes and the ShinyProxy documentation a few times. Nonetheless, I suggest running a test. I just installed the app with ShinyProxy 2.4.1. Can you check, if the problem persists? Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Mittwoch, 3. Februar 2021 01:14 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: the grand migration Düben, Christian writes
Can you not just use essentially the same setting that you used on Darni? There it worked without the 8080 in the URL.
Guess what I tried ... I don't know why that is not working. It maybe that the app is not installed in the same way? It maybe because the apache version is not the same? It maybe /etc/hosts, or something in the DNS?
app.collec.repec.org does not point to the app anymore. It just points to an Apache2 Debian default page.
ooops ... fixed.
Why should robots not visit the app?
By "app" you mean the official new CollEc? I'm not excluding any visits to it. I just redirect URLs that went to the legacy site to legacy as they will not be found on the app. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Düben, Christian writes
What exactly does not work when you set "ProxyPass / http://localhost:8080/"?
I for one see that my client goes to its localhost, which is my local machine. There, nothing is listening on port 8080.
On Helos, I deploy the app via ShinyProxy 2.4.3 whereas I used ShinyProxy 2.4.1 on Darni. However, nothing relevant to our application should have changed.
I agree.
I went through the release notes and the ShinyProxy documentation a few times. Nonetheless, I suggest running a test. I just installed the app with ShinyProxy 2.4.1. Can you check, if the problem persists?
It does Не удается получить доступ к сайту Сайт localhost не позволяет установить соединение. Попробуйте сделать следующее: Проверьте подключение к Интернету. Проверьте настройки прокси-сервера и брандмауэра. ERR_CONNECTION_REFUSED -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
ShinyProxy is listening on port 8080. See "sudo netstat -ntlp | grep LISTEN" or open "http://95.216.245.19:8080" in your browser. As it neither works with ShinyProxy 2.4.1, this appears to be a web server problem. Do you set any firewall or modify iptables? According to the Google translation of your Russian error message, something appears to block access to the localhost. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Mittwoch, 3. Februar 2021 13:48 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: the grand migration Düben, Christian writes
What exactly does not work when you set "ProxyPass / http://localhost:8080/"?
I for one see that my client goes to its localhost, which is my local machine. There, nothing is listening on port 8080.
On Helos, I deploy the app via ShinyProxy 2.4.3 whereas I used ShinyProxy 2.4.1 on Darni. However, nothing relevant to our application should have changed.
I agree.
I went through the release notes and the ShinyProxy documentation a few times. Nonetheless, I suggest running a test. I just installed the app with ShinyProxy 2.4.1. Can you check, if the problem persists?
It does Не удается получить доступ к сайту Сайт localhost не позволяет установить соединение. Попробуйте сделать следующее: Проверьте подключение к Интернету. Проверьте настройки прокси-сервера и брандмауэра. ERR_CONNECTION_REFUSED -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Düben, Christian writes
ShinyProxy is listening on port 8080. See "sudo netstat -ntlp | grep LISTEN" or open "http://95.216.245.19:8080" in your browser.
Yes, I checked taht a long time ago.
As it neither works with ShinyProxy 2.4.1, this appears to be a web server problem.
yes
Do you set any firewall or modify iptables? According to the Google translation of your Russian error message, something appears to block access to the localhost.
No! It is accessing *MY* localhost, the computer on my bed here. There is nothing on 8080 running on that machine. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
So, the server redirects you to your own machine instead of using its own ports. Did you try it with " UseCanonicalName Off"? Or " 0.0.0.0:8080"? Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Mittwoch, 3. Februar 2021 14:26 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: the grand migration Düben, Christian writes
ShinyProxy is listening on port 8080. See "sudo netstat -ntlp | grep LISTEN" or open "http://95.216.245.19:8080" in your browser.
Yes, I checked taht a long time ago.
As it neither works with ShinyProxy 2.4.1, this appears to be a web server problem.
yes
Do you set any firewall or modify iptables? According to the Google translation of your Russian error message, something appears to block access to the localhost.
No! It is accessing *MY* localhost, the computer on my bed here. There is nothing on 8080 running on that machine. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Düben, Christian writes
So, the server redirects you to your own machine instead of using its own ports. Did you try it with " UseCanonicalName Off"?
Yes.
Or " 0.0.0.0:8080"?
Maybe later. Now I have to go out to get drunk. Sorry! I suggest this better sorted out by you calling me. I will head for errands tomorrow moring then be back in the afternoon. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Düben, Christian writes
So, the server redirects you to your own machine instead of using its own ports.
ProxyPassReverse / http://collec.repec.org:8080/ gets rid of the 8080, but we still have the long path. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
What long path? Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Donnerstag, 4. Februar 2021 15:23 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: the grand migration Düben, Christian writes
So, the server redirects you to your own machine instead of using its own ports.
ProxyPassReverse / http://collec.repec.org:8080/ gets rid of the 8080, but we still have the long path. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Düben, Christian writes
What long path?
app/collec_app -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
That is a ShinyProxy issue. Would you like to get rid of it? Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Donnerstag, 4. Februar 2021 15:25 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: the grand migration Düben, Christian writes
What long path?
app/collec_app -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Düben, Christian writes
That is a ShinyProxy issue. Would you like to get rid of it?
You are the boss here, but I for one would find it better not to have at there. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
The /app/collec_app that gets automatically attached to the URL has to do with the structure of ShinyProxy. The program can host multiple apps at once. So by default, users first visit a landing page (after login) that lists the available apps. I changed the settings so that users skip the default landing page and are automatically redirected to the app at /app/collec_app. We can try to cut the /app/collec_app via the web server configuration. I found a line of NGINX code that supposedly works: "rewrite ^ /app/myapp break". I do not know what the Apache counterpart would look like. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Donnerstag, 4. Februar 2021 15:46 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: the grand migration Düben, Christian writes
That is a ShinyProxy issue. Would you like to get rid of it?
You are the boss here, but I for one would find it better not to have at there. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Düben, Christian writes
The /app/collec_app that gets automatically attached to the URL has to do with the structure of ShinyProxy. The program can host multiple apps at once. So by default, users first visit a landing page (after login) that lists the available apps. I changed the settings so that users skip the default landing page and are automatically redirected to the app at /app/collec_app.
We can try to cut the /app/collec_app via the web server configuration. I found a line of NGINX code that supposedly works: "rewrite ^ /app/myapp break". I do not know what the Apache counterpart would look like.
http://collec.repec.org/app/collec_app now appears to be down. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
The ShinyProxy container is still up. However, every file connected to CollEc is now a GZ archive. What happened? Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Donnerstag, 4. Februar 2021 16:46 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: the grand migration Düben, Christian writes
The /app/collec_app that gets automatically attached to the URL has to do with the structure of ShinyProxy. The program can host multiple apps at once. So by default, users first visit a landing page (after login) that lists the available apps. I changed the settings so that users skip the default landing page and are automatically redirected to the app at /app/collec_app.
We can try to cut the /app/collec_app via the web server configuration. I found a line of NGINX code that supposedly works: "rewrite ^ /app/myapp break". I do not know what the Apache counterpart would look like.
http://collec.repec.org/app/collec_app now appears to be down. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Düben, Christian writes
The ShinyProxy container is still up. However, every file connected to CollEc is now a GZ archive. What happened?
I gziped the files in ~/collec. Now reverting... -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Thomas Krichel writes
I gziped the files in ~/collec. Now reverting...
In my defense, for what may appear to be a monumentally stupid act ... the files should not live in /root/collec and owned by root. I think it is best to open a collec account and have the files owned by that user, again in an opt. I hope CD will agree with me that it is now time to dismantle the installation on darni. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
I can change the directory. Something like /home/collec? I kept the setup on Darni (with the daily updating routine deactivated) in case the Helos setup takes longer than expected. But as the Helos setup appears to be stable now, I will clear the Darni installation. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Freitag, 5. Februar 2021 09:33 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: the grand migration Thomas Krichel writes
I gziped the files in ~/collec. Now reverting...
In my defense, for what may appear to be a monumentally stupid act ... the files should not live in /root/collec and owned by root. I think it is best to open a collec account and have the files owned by that user, again in an opt. I hope CD will agree with me that it is now time to dismantle the installation on darni. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Düben, Christian writes
I can change the directory. Something like /home/collec?
Looks fine to me.
I kept the setup on Darni (with the daily updating routine deactivated) in case the Helos setup takes longer than expected. But as the Helos setup appears to be stable now, I will clear the Darni installation.
Thank you. In the meantime, I have done some wordsmithing on the legacy site, and included a check for how many files are done in the get_paths script icanis@helos:~$ ~/perl/get_path_stats count: 46788 age: 0.000324074074074074 46.2732310353201 90.6303587962963 done: 1357 This sugests after roughly 4 days, say, of work we got 1357 done, say 300 a day, that suggest that we could get the full thing done in 155 days. So 3 processess appear to be enough. In fact we can prolly do with two. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
I cleared the Darni installation and files. Copying the Helos files to a different directory unfortunately caused a cascade of errors. I had to remove CollEc's MariaDB files and set the database up from scratch. There is still an issue with missing files. So, I am running the daily updating routine now which should generate the missing files. Hetzner recently sends me notifications regarding errors on Darni. Are you aware of that? Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Freitag, 5. Februar 2021 10:35 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: the grand migration Düben, Christian writes
I can change the directory. Something like /home/collec?
Looks fine to me.
I kept the setup on Darni (with the daily updating routine deactivated) in case the Helos setup takes longer than expected. But as the Helos setup appears to be stable now, I will clear the Darni installation.
Thank you. In the meantime, I have done some wordsmithing on the legacy site, and included a check for how many files are done in the get_paths script icanis@helos:~$ ~/perl/get_path_stats count: 46788 age: 0.000324074074074074 46.2732310353201 90.6303587962963 done: 1357 This sugests after roughly 4 days, say, of work we got 1357 done, say 300 a day, that suggest that we could get the full thing done in 155 days. So 3 processess appear to be enough. In fact we can prolly do with two. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
Düben, Christian writes
Hetzner recently sends me notifications regarding errors on Darni. Are you aware of that?
Yes, but I did not pay attention. They are gone now. We could ask Cezar to set up a monitor for Collec on helos. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
I set up the ftp. Sadly it seems to work only in passive mode krichel@trabbi~$ ftp collec.repec.org Connected to collec.repec.org. 220 (vsFTPd 3.0.3) Name (collec.repec.org:krichel): ftp 331 Please specify the password. Password: 230 Login successful. Remote system type is UNIX. Using binary mode to transfer files. ftp> ls 500 Illegal PORT command. ftp: bind: Address already in use ftp> pass Passive mode on. ftp> ls 227 Entering Passive Mode (95,216,245,19,153,89). 150 Here comes the directory listing. -rw-r--r-- 1 0 0 2094499 Feb 06 04:38 Betweenness.csv -rw-r--r-- 1 0 0 2452551 Feb 06 04:38 Closeness.csv 226 Directory send OK. ftp> exit 221 Goodbye. I did some mork wordsmithing on the legacy home page. I suggests that CD now replaces "Founder: Thomas Krichel" with a something like "This site replaced the <a href="http://legacy.collec.repec.org">Legacy Collec site</a>." I think this pretty much brings the migration to a close. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
I do not know what to do about the ftp. But I changed the footer. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Samstag, 6. Februar 2021 15:43 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: [CollEc] the grand migration I set up the ftp. Sadly it seems to work only in passive mode krichel@trabbi~$ ftp collec.repec.org Connected to collec.repec.org. 220 (vsFTPd 3.0.3) Name (collec.repec.org:krichel): ftp 331 Please specify the password. Password: 230 Login successful. Remote system type is UNIX. Using binary mode to transfer files. ftp> ls 500 Illegal PORT command. ftp: bind: Address already in use ftp> pass Passive mode on. ftp> ls 227 Entering Passive Mode (95,216,245,19,153,89). 150 Here comes the directory listing. -rw-r--r-- 1 0 0 2094499 Feb 06 04:38 Betweenness.csv -rw-r--r-- 1 0 0 2452551 Feb 06 04:38 Closeness.csv 226 Directory send OK. ftp> exit 221 Goodbye. I did some mork wordsmithing on the legacy home page. I suggests that CD now replaces "Founder: Thomas Krichel" with a something like "This site replaced the <a href="http://legacy.collec.repec.org">Legacy Collec site</a>." I think this pretty much brings the migration to a close. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
participants (3)
-
Christian Zimmermann -
Düben, Christian -
Thomas Krichel