Any ideas why it cannot be located and installed? sudo apt-get install libapache2-mod_proxy_wstunnel E: Unable to locate package libapache2-mod_proxy_wstunnel Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Montag, 10. August 2020 10:52 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: collec-run@lists.openlib.org Subject: Re: New CollEc Düben, Christian writes

It looks like the missing mod_proxy_wstunnel Apache module might be the issue. May I install it or would that clash with any webserver setting?

Sure try it. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

Thanks. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Montag, 10. August 2020 12:00 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: collec-run@lists.openlib.org Subject: Re: New CollEc Düben, Christian writes

Any ideas why it cannot be located and installed?

sudo apt-get install libapache2-mod_proxy_wstunnel E: Unable to locate package libapache2-mod_proxy_wstunnel

I will have a look. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

Great. Thanks. The app can now be accessed at test.collec.repec.org. In my opinion, the app is ready to be publicly released. Is there anything you would like to add or modify? Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Montag, 10. August 2020 12:26 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: collec-run@lists.openlib.org Subject: Re: New CollEc Düben, Christian writes

Thanks.

Looks like it's not in a separate package root@darni ~ # apt search ws_tunnel Sorting... Done Full Text Search... Done root@darni ~ # a2enmod proxy_wstunnel Considering dependency proxy for proxy_wstunnel: Module proxy already enabled Enabling module proxy_wstunnel. To activate the new configuration, you need to run: systemctl restart apache2 root@darni ~ # systemctl restart apache2 -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

It does not have rankings. The shortest paths can be accessed through the Shortest Paths tab. But I did not yet implement a functionality to export them. I recommend watching the tutorial on the functionalities. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Montag, 10. August 2020 14:02 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: New CollEc Düben, Christian writes

The app can now be accessed at test.collec.repec.org.

In my opinion, the app is ready to be publicly released. Is there anything you would like to add or modify?

Well is it a replacement for CollEc? Does it have rankings that we can link to? Where can we access the path data for export? -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

I can come up with rankings and with an exportable text file. But that has to wait. Over the past few months I spent much of my time on GraphEc, CollEc and teaching and neglected my research. With multiple upcoming conferences and potential research visits, that needs to change. I have to spend the next weeks on research. My suggestion is to keep both versions of CollEc running for now and to postpone the official transition between versions until October. Until then, we could run the web application at e.g. collec2.repec.org to highlight the upcoming transition. The text in the Shortest Paths tab mentions the potential existence of multiple shortest paths. And that is addressed by the choice between the binary case and three transition function cases. Which shortest path is displayed in the Shortest Paths tab does not affect the computed centrality measures. For closeness the number of shortest paths between two authors does generally not matter. Only the distance between the two people is considered. And the code deriving betweenness internally computes all shortest paths, but only writes the resulting betweenness result to disk. I know how it feels to spend multiple days on a bug. I faced such issues when fixing Shiny's internal reactivity. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Montag, 10. August 2020 14:31 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: New CollEc Düben, Christian writes

It does not have rankings.

Rankings are the flesh and blood of RePEc. We need to have something like the current ftp://ftp.repec.org/CollEc/CollEc.txt CZ uses this.

The shortest paths can be accessed through the Shortest Paths tab.

I understand this, and I can access a path, but I'm warned that there may be many.

But I did not yet implement a functionality to export them.

I think we need to get that done. If I was more advanced with ArchEc, I would love to help. I just spend one week just fixing one bug ... and there are more. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

I openly admit that I am not a fan of that two stage procedure with an unweighted (binary) first stage and a weighted second stage. However, I see that you really insist on having it in the app. Thus, I came up with a solution that produces the same results as your two stage procedure while being a lot more efficient: W_{i,j} = 1 - (N_{i,j} * 0.001). Unless any two authors published at least 500 papers together this transition function results in the same shortest path as the two stage approach. Just like you insist on the above mentioned unweighted-weighted combination, I am not dropping the weighted cases. To evaluate how intuitive the weighted edges are to new users I asked various people to test the app. And nobody struggled with the intuition behind edge weighting. It is the same as requesting the fastest route between two places on Google Maps. To account for your concerns I already set the binary option as the app's default case. Users have to actively tick a box to change from the unweighted (binary) to weighted cases. So, how about you get your optimized unweighted-weighted approach and I keep the weighted edges in the app? Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Dienstag, 11. August 2020 05:32 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: New CollEc Düben, Christian writes

I can come up with rankings and with an exportable text file.

Maybe somebody else can, if he knows how to access the raw data.

But that has to wait. Over the past few months I spent much of my time on GraphEc, CollEc and teaching and neglected my research. With multiple upcoming conferences and potential research visits, that needs to change. I have to spend the next weeks on research. My suggestion is to keep both versions of CollEc running for now and to postpone the official transition between versions until October. Until then, we could run the web application at e.g. collec2.repec.org to highlight the upcoming transition.

I fully agree. I will think about how to do this.

The text in the Shortest Paths tab mentions the potential existence of multiple shortest paths. And that is addressed by the choice between the binary case and three transition function cases.

I think there are two issues here. There are multiple shortest paths by one weighing schemes and there are Let me repeat what I said before. I don't think we should expose non-expert users to non-binary paths, because of the triangularity issue. Therefore I think it is best to examine a set of multiple binary paths, evaluate them against other weighing measures and discard those that don't have minimum length against them. That itself is presumably not difficult to do but it is difficult to do so that it runs fast. The exported data does not need to be completely up-to-date. We can first do new nodes, and then form a queue for the rest of the nodes.

Which shortest path is displayed in the Shortest Paths tab does not affect the computed centrality measures. For closeness the number of shortest paths between two authors does generally not matter. Only the distance between the two people is considered. And the code deriving betweenness internally computes all shortest paths, but only writes the resulting betweenness result to disk.

But all shortest paths of all nodes under all weighing schemes are available somewhere. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

In my research I usually apply graph theoretical methods in a geo-spatial context. And in that field I would not consider the two stage procedure a viable choice when compared with properly defined transition functions. I guess that is why I am not a fan. Let me explain how the proposed transition function replaces the two stage procedure without changing the results. Consider the graph attached to this e-mail. The numbers denote the quantity of joint papers. What is the shortest path between A and H? There are three potential connections: A-C-D-H, A-B-E-H and A-B-F-G-H. With the inverse transition function (W_{i,j} = N_{i,j}{-1}) the shortest path would be A-B-F-G-H because its sum of edge weights (1.38) is lower than in the other two cases (1.83 and 2). However, in your two stage binary process this path would have been ruled out in the first stage because there are five authors on it, compared two four authors in the other two options. Your procedure would choose A-C-D-H. It is one of the two shortest paths (A-C-D-H and A-B-E-H) in the binary case. And it is the shorter one among the two when weighted by the number of papers. The proposed transition function W_{i,j} = 1 - (N_{i,j} * 0.001) results in the same shortest path. A-C-D-H (length: 2.994) is shorter than A-B-E-H (length: 2.995) and A-B-F-G-H (length: 3.972). This transition function is the binary application, minimizing the quantity of authors, with a tiny adjustment for the number of papers selecting the shortest path among the binary case options. Is that clear? Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Dienstag, 11. August 2020 11:51 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: CollEc Run <collec-run@lists.openlib.org> Subject: Re: New CollEc Düben, Christian writes

I openly admit that I am not a fan of that two stage procedure with an unweighted (binary) first stage and a weighted second stage.

Why not?

However, I see that you really insist on having it in the app.

No, I don't but I want to have it exported. And I prefer to have it in the app, otherwise the data will be literally incredible. Users will look at weight data and seen triangles.

Thus, I came up with a solution that produces the same results as your two stage procedure while being a lot more efficient: W_{i,j} = 1 - (N_{i,j} * 0.001). Unless any two authors published at least 500 papers together this transition function results in the same shortest path as the two stage approach.

I don't understand this, sorry. Will the replace binary paths?

Just like you insist on the above mentioned unweighted-weighted combination, I am not dropping the weighted cases. To evaluate how intuitive the weighted edges are to new users I asked various people to test the app. And nobody struggled with the intuition behind edge weighting. It is the same as requesting the fastest route between two places on Google Maps. To account for your concerns I already set the binary option as the app's default case. Users have to actively tick a box to change from the unweighted (binary) to weighted cases.

Edge weighting is not a problem as long as the users know this.

So, how about you get your optimized unweighted-weighted approach and I keep the weighted edges in the app?

Keep them in the app. I don't have a problem, but we should export binary paths. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

Düben, Christian writes

In my opinion, the app is ready to be publicly released. Is there anything you would like to add or modify?

I would love to get rid of the CollEc we have now, but for that to happen we need something that covers its essential funcitonality. As CZ has pointed out, the path data is important. We need binary paths and reduce the number of multiples by elimating those that don't have a shorted weight length. I would love having other weigh in on this. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

Thanks. 1. Which browser do you use? With Firefox, Chrome and Edge I only see one footer on the intro page. 2. My RePEc profile is still rather empty. I, therefore, prefer to use my homepage for now. 3. I do not recommend setting the parameters - incl. author names - via the URL. It can be a source of errors in the application. As a test, I tried it in the closeness tab. The link to Daron Acemoglu without weighted edges: http://test.collec.repec.org/app_direct/collec_app/?_inputs_&d_author1_handl.... However, the app gets stuck when opening it via this URL rather than by clicking through the app. Even if I somehow fix this, the links only remain valid until I change the app and introduce further parameters or change the names of the current ones. Thus, links to specific authors' pages are not available at this point. 4. It is an interactive D3 JavaScript Network Graph. When you hover with your mouse over a vertex, it displays the respective author's name or short ID and highlights his or her co-authors (Co-Authors tab at 6:14 in tutorial video). Thus, it also illustrates the links between the blue co-author vertices. I can add a note on that in the text. 5. If you have access to the server, I can send you the path to the closeness and betweenness files. An external download functionality is yet to be implemented. 6. How long should the blog post be? 7. I have no idea how well this scales with potentially thousands of users. The app is designed to only use a minimum of computational resources when users interact with it. We will see if this one server can handle the load after the app is released. 8. The network only entails authors who published co-authored research. If you like, I can add a note on the overall number of people in RAS. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Christian Zimmermann <zimmermann@stlouisfed.org> Sent: Montag, 10. August 2020 17:31 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Thomas Krichel <krichel@openlib.org>; collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc Looks great! A couple of comments: 1- On the intro page, it looks like there are two footers. 2- Personal preference: I like links to RePEc profiles instead of homepages (more stable), in the footer. 3- Is there a way to link to a particular author on any of the pages? I would like to maintain a link from author profiles on IDEAS. 4- I am not sure what the cloud means in the co-author tab. Does it include co-author-links amongst co-authors? If so, say so more clearly. 5- I need a way to find the closeness and betweenness data to download for the author rankings. 6- As this is launched, it would be good to have an accompanying blog post. 7- I would announce this in the monthly mailing. Can this sustain a surge in traffic? 8- 47K authors are in the network, but RAS has 59K authors. Important distinction. Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist On Mon, 10 Aug 2020, D�ben, Christian wrote:

Great. Thanks.

The app can now be accessed at test.collec.repec.org.

In my opinion, the app is ready to be publicly released. Is there anything you would like to add or modify?

Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Montag, 10. August 2020 12:26 To: D�ben, Christian <Christian.Dueben@uni-hamburg.de> Cc: collec-run@lists.openlib.org Subject: Re: New CollEc

D�ben, Christian writes

...

Thanks.

Looks like it's not in a separate package

root@darni ~ # apt search ws_tunnel Sorting... Done Full Text Search... Done root@darni ~ # a2enmod proxy_wstunnel Considering dependency proxy for proxy_wstunnel: Module proxy already enabled Enabling module proxy_wstunnel. To activate the new configuration, you need to run: systemctl restart apache2 root@darni ~ # systemctl restart apache2

--

Cheers,

Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run

Sorry for the late reply. I currently do not have time for CollEc, because I need to work on two of my papers with upcoming conference presentations. An issue that I already fixed in a modification a week ago is point 8 on your list. The documentation now mentions the 59,000 people in RAS. To avoid regular manual updates of the number of people in CollEc and the number of authors in the main graph, the daily updating routine now automatically sets these values mentioned in the introduction and the documentation. I am going to work on the remaining CollEc issues in early September. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Christian Zimmermann <zimmermann@stlouisfed.org> Sent: Sonntag, 16. August 2020 00:04 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Thomas Krichel <krichel@openlib.org>; collec-run@lists.openlib.org Subject: RE: [CollEc] New CollEc Sorry for the delay. Answers below. Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist On Mon, 10 Aug 2020, D�ben, Christian wrote:

Thanks.

1. Which browser do you use? With Firefox, Chrome and Edge I only see one footer on the intro page.

There are twice lines starting with "CollEc was founded by..."

2. My RePEc profile is still rather empty. I, therefore, prefer to use my homepage for now. 3. I do not recommend setting the parameters - incl. author names - via the URL. It can be a source of errors in the application. As a test, I tried it in the closeness tab. The link to Daron Acemoglu without weighted edges: http://test.collec.repec.org/app_direct/collec_app/?_inputs_&d_author1 _handle=%22%22&p_author2_handle=%22%22&d_select_var=%22name%22&link_ta b_Documentation=0&b_select_var=%22name%22&c_weighting_var=%22inverse%2 2&b_weighting_var=%22inverse%22&d_author1_name=%22%22&c_select_var=%22 name%22&p_author2_name=%22%22&g_view=0&b_author_name=%22%22&d_view=0&p _author1_name=%22%22&p_weighting_yn=%22no%22&c_view=1&navbars=%22tab_C loseness%22&d_weighting_yn=%22no%22&p_select_var=%22name%22&c_author_n ame=%22Daron%20Acemoglu%22&g_select_var=%22name%22&d_author2_name=%22% 22&g_author_name=%22%22&g_author_handle=%22%22&p_weighting_var=%22inve rse%22&b_author_handle=%22%22&d_author2_handle=%22%22&d_weighting_var= %22inverse%22&c_weighting_yn=%22no%22&p_view=0&c_author_handle=%22%22& p _author1_handle=%22%22&b_view=0&b_weighting_yn=%22no%22. However, the app gets stuck when opening it via this URL rather than by clicking through the app. Even if I somehow fix this, the links only remain valid until I change the app and introduce further parameters or change the names of the current ones. Thus, links to specific authors' pages are not available at this point.

I would still aprreciate to point people to there landing page.

4. It is an interactive D3 JavaScript Network Graph. When you hover with your mouse over a vertex, it displays the respective author's name or short ID and highlights his or her co-authors (Co-Authors tab at 6:14 in tutorial video). Thus, it also illustrates the links between the blue co-author vertices. I can add a note on that in the text.

Yes please (somehow I always end up getting the questions).

5. If you have access to the server, I can send you the path to the closeness and betweenness files. An external download functionality is yet to be implemented.

If you can put a file somewhere through ftp ot http(s), that would be great.

6. How long should the blog post be?

As long as necessary. See past posts for reference. There is a lot of variation.

7. I have no idea how well this scales with potentially thousands of users. The app is designed to only use a minimum of computational resources when users interact with it. We will see if this one server can handle the load after the app is released.

The most traffic will likely come when I announce it in the monthly email. That is staggered over a week, because the RePEc Author Service cannot handle the traffic (working on it...).

8. The network only entails authors who published co-authored research. If you like, I can add a note on the overall number of people in RAS.

Yes please.

Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: Christian Zimmermann <zimmermann@stlouisfed.org> Sent: Montag, 10. August 2020 17:31 To: D�ben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Thomas Krichel <krichel@openlib.org>; collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

Looks great!

A couple of comments: 1- On the intro page, it looks like there are two footers. 2- Personal preference: I like links to RePEc profiles instead of homepages (more stable), in the footer. 3- Is there a way to link to a particular author on any of the pages? I would like to maintain a link from author profiles on IDEAS. 4- I am not sure what the cloud means in the co-author tab. Does it include co-author-links amongst co-authors? If so, say so more clearly. 5- I need a way to find the closeness and betweenness data to download for the author rankings. 6- As this is launched, it would be good to have an accompanying blog post. 7- I would announce this in the monthly mailing. Can this sustain a surge in traffic? 8- 47K authors are in the network, but RAS has 59K authors. Important distinction.

Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist

On Mon, 10 Aug 2020, D�ben, Christian wrote:

...

Great. Thanks.

The app can now be accessed at test.collec.repec.org.

In my opinion, the app is ready to be publicly released. Is there anything you would like to add or modify?

Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Montag, 10. August 2020 12:26 To: D�ben, Christian <Christian.Dueben@uni-hamburg.de> Cc: collec-run@lists.openlib.org Subject: Re: New CollEc

D�ben, Christian writes

...

Thanks.

Looks like it's not in a separate package

root@darni ~ # apt search ws_tunnel Sorting... Done Full Text Search... Done root@darni ~ # a2enmod proxy_wstunnel Considering dependency proxy for proxy_wstunnel: Module proxy already enabled Enabling module proxy_wstunnel. To activate the new configuration, you need to run: systemctl restart apache2 root@darni ~ # systemctl restart apache2

--

Cheers,

Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run

@ Christian Zimmermann: 3. The landing page issue has been dealt with. The documentation's section on entry points describes how to use this feature. 4. The text in the Co-Authors tab now contains an additional note on the displayed illustration. I am soon going to address the remaining issues. @ Thomas Krichel: - Where can I put the files that Christian Zimmermann wants to access through ftp? Should I add them to /mnt/helos/home/icanis/opt/ftp? - The daily updating routine syncs a local folder with the per archive. At least, this is what I thought my code does. Unfortunately, it does not add newly added files to the directory. This is my code: rsync -a --delete --include "*.xml" --exclude "*" "rsync://rsync.repec.org/RePEc-AMF/per/" "/home/icanis/opt/collec_data/Per_Archive/AMF" I am currently missing more than 400 xml files of what I assume to be recently added RePEc users. Any idea what is wrong? Do I need to add another --include argument? Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of Düben, Christian Sent: Donnerstag, 17. September 2020 21:15 To: Christian Zimmermann <zimmermann@stlouisfed.org> Cc: collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc A quick update on the new CollEc: I am working on the landing page issue (number 3 in your list). The hidden reactivity layers still cause problems when trying to enter the app at stages defined via the URL. But I found tools which I might fix this. It just takes quite a bit of time. My ongoing research projects do not leave a lot of space for other activities at moment. Nonetheless, I will try to fix the landing page issue within the next few weeks. Completing the remaining CollEc tasks until the end of the month would be great. I will notify you once it is done. Have a nice evening. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of Düben, Christian Sent: Donnerstag, 20. August 2020 15:08 To: Christian Zimmermann <zimmermann@stlouisfed.org> Cc: collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc Sorry for the late reply. I currently do not have time for CollEc, because I need to work on two of my papers with upcoming conference presentations. An issue that I already fixed in a modification a week ago is point 8 on your list. The documentation now mentions the 59,000 people in RAS. To avoid regular manual updates of the number of people in CollEc and the number of authors in the main graph, the daily updating routine now automatically sets these values mentioned in the introduction and the documentation. I am going to work on the remaining CollEc issues in early September. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Christian Zimmermann <zimmermann@stlouisfed.org> Sent: Sonntag, 16. August 2020 00:04 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Thomas Krichel <krichel@openlib.org>; collec-run@lists.openlib.org Subject: RE: [CollEc] New CollEc Sorry for the delay. Answers below. Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist On Mon, 10 Aug 2020, D�ben, Christian wrote:

Thanks.

1. Which browser do you use? With Firefox, Chrome and Edge I only see one footer on the intro page.

There are twice lines starting with "CollEc was founded by..."

2. My RePEc profile is still rather empty. I, therefore, prefer to use my homepage for now. 3. I do not recommend setting the parameters - incl. author names - via the URL. It can be a source of errors in the application. As a test, I tried it in the closeness tab. The link to Daron Acemoglu without weighted edges: http://test.collec.repec.org/app_direct/collec_app/?_inputs_&d_author1 _handle=%22%22&p_author2_handle=%22%22&d_select_var=%22name%22&link_ta b_Documentation=0&b_select_var=%22name%22&c_weighting_var=%22inverse%2 2&b_weighting_var=%22inverse%22&d_author1_name=%22%22&c_select_var=%22 name%22&p_author2_name=%22%22&g_view=0&b_author_name=%22%22&d_view=0&p _author1_name=%22%22&p_weighting_yn=%22no%22&c_view=1&navbars=%22tab_C loseness%22&d_weighting_yn=%22no%22&p_select_var=%22name%22&c_author_n ame=%22Daron%20Acemoglu%22&g_select_var=%22name%22&d_author2_name=%22% 22&g_author_name=%22%22&g_author_handle=%22%22&p_weighting_var=%22inve rse%22&b_author_handle=%22%22&d_author2_handle=%22%22&d_weighting_var= %22inverse%22&c_weighting_yn=%22no%22&p_view=0&c_author_handle=%22%22& p _author1_handle=%22%22&b_view=0&b_weighting_yn=%22no%22. However, the app gets stuck when opening it via this URL rather than by clicking through the app. Even if I somehow fix this, the links only remain valid until I change the app and introduce further parameters or change the names of the current ones. Thus, links to specific authors' pages are not available at this point.

I would still aprreciate to point people to there landing page.

4. It is an interactive D3 JavaScript Network Graph. When you hover with your mouse over a vertex, it displays the respective author's name or short ID and highlights his or her co-authors (Co-Authors tab at 6:14 in tutorial video). Thus, it also illustrates the links between the blue co-author vertices. I can add a note on that in the text.

Yes please (somehow I always end up getting the questions).

5. If you have access to the server, I can send you the path to the closeness and betweenness files. An external download functionality is yet to be implemented.

If you can put a file somewhere through ftp ot http(s), that would be great.

6. How long should the blog post be?

As long as necessary. See past posts for reference. There is a lot of variation.

7. I have no idea how well this scales with potentially thousands of users. The app is designed to only use a minimum of computational resources when users interact with it. We will see if this one server can handle the load after the app is released.

The most traffic will likely come when I announce it in the monthly email. That is staggered over a week, because the RePEc Author Service cannot handle the traffic (working on it...).

8. The network only entails authors who published co-authored research. If you like, I can add a note on the overall number of people in RAS.

Yes please.

Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: Christian Zimmermann <zimmermann@stlouisfed.org> Sent: Montag, 10. August 2020 17:31 To: D�ben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Thomas Krichel <krichel@openlib.org>; collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

Looks great!

A couple of comments: 1- On the intro page, it looks like there are two footers. 2- Personal preference: I like links to RePEc profiles instead of homepages (more stable), in the footer. 3- Is there a way to link to a particular author on any of the pages? I would like to maintain a link from author profiles on IDEAS. 4- I am not sure what the cloud means in the co-author tab. Does it include co-author-links amongst co-authors? If so, say so more clearly. 5- I need a way to find the closeness and betweenness data to download for the author rankings. 6- As this is launched, it would be good to have an accompanying blog post. 7- I would announce this in the monthly mailing. Can this sustain a surge in traffic? 8- 47K authors are in the network, but RAS has 59K authors. Important distinction.

Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist

On Mon, 10 Aug 2020, D�ben, Christian wrote:

...

Great. Thanks.

The app can now be accessed at test.collec.repec.org.

In my opinion, the app is ready to be publicly released. Is there anything you would like to add or modify?

Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Montag, 10. August 2020 12:26 To: D�ben, Christian <Christian.Dueben@uni-hamburg.de> Cc: collec-run@lists.openlib.org Subject: Re: New CollEc

D�ben, Christian writes

...

Thanks.

Looks like it's not in a separate package

root@darni ~ # apt search ws_tunnel Sorting... Done Full Text Search... Done root@darni ~ # a2enmod proxy_wstunnel Considering dependency proxy for proxy_wstunnel: Module proxy already enabled Enabling module proxy_wstunnel. To activate the new configuration, you need to run: systemctl restart apache2 root@darni ~ # systemctl restart apache2

--

Cheers,

Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run

_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run _______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run

The files meant for the ftp server are in /home/icanis/ftp/opt. The rank variable in these files handles ties with the "min" option. Assume we have four authors with closeness values of 0.5, 0.3, 0.3 and 0.2 respectively. With the "min" option their ranks are 1, 2, 2 and 4. You mentioned that your code fixes the rsync permission problem over time. What time frame did you have in mind? It does not work yet. Do I need to add anything to the shell script? I fixed the issues on Christian Zimmermann's list, except for the blog post which still needs another paragraph or so. The text in the Introduction tab, the Co-Authors tab, the Documentation tab and the footer is now partly different. I also added an image to the documentation on the Co-Authors tab. Before Christian Zimmermann links any IDEAS content to the web application's entry points, we should discuss the URL. The current test.collec.repec.org was suitable during testing. But now that the app is quasi officially released, I recommend changing the name to e.g. collec2.repec.org. Until we see how well the app performs when accessed by many users at the same time we should keep the old version of CollEc in service. And keeping the old one at collec.repec.org while running the new one at collec2.repec.org appears to be a good solution. What do you think? Could you add a placeholder HTML page at graphec.repec.org? Something like "GraphEc is currently publicly unavailable. Contact <a href="http://www.christian-dueben.com">Christian Düben</a> regarding the application's public release.". The new CollEc already links to GraphEc and I think that a placeholder page with a short informative text would look nicer than the Apache2 Debian default page. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Freitag, 25. September 2020 09:24 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Christian Zimmermann <zimmermann@stlouisfed.org>; collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc Düben, Christian writes

- Where can I put the files that Christian Zimmermann wants to access through ftp?

Just put them anywhere and let me know where they are, I will arrange for public rsync for you. Since, I presume, they are generated, they should go into a directory named 'opt' to avoid filling disk space at the backup servers.

Should I add them to /mnt/helos/home/icanis/opt/ftp?

No, that's an NFS mount of helos.

I am currently missing more than 400 xml files of what I assume to be recently added RePEc users. Any idea what is wrong? Do I need to add another --include argument?

adrepec@aigtu:~$ $HOME/perl/remi_round_the_clock export REDIFDIR=/home/adrepec/RePEc/zzz ; cd /home/adrepec/perl; /home/adrepec/perl/remi2.pl -conf remi.local.conf sh: 1: /home/adrepec/perl/remi2.pl: Permission denied oh that hurts! adrepec@aigtu:~$ cd perl adrepec@aigtu:~/perl$ chmod 755 remi2.pl adrepec@aigtu:~/perl$ $HOME/perl/remi_round_the_clock Should fix that one over time. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

Do you mean the shortest paths between any two authors in the network? Users can look at their shortest paths in the Shortest Paths tab. That functionality computes them directly from the graph. The distances and the closeness measures also rely on these shortest paths. But at no point are they written to disk. They are only stored in memory. I did some extensive testing and writing these paths to disk is incredibly inefficient. There is a reason why the new CollEc's daily updating routine takes only a tiny fraction of the time and computational resources required by the old CollEc. It is the eradication of inefficiencies along multiple dimensions. Unless there is a very good reason why these paths should be written to disk, I do not see a point of doing so. Are they used anywhere outside CollEc? app.collec.repec.org sounds good. Let me know when you changed it. I am then going to adjust the documentation on the entry points. It is not a problem that Helos is an Ubuntu system. In fact, I prefer Ubuntu over Debian. Nonetheless, I am really not looking forward to that migration as I have to set up and test the entire framework again. There should be easier ways to directly migrating the CollEc system as a whole instead of installing it again piece by piece. But that is beyond my current technical skills. The new CollEc requires rather little disk space. The only component that appears unexpectedly large is the containerized MariaDB. I am going to check tomorrow what might be wrong with it. The authors' full names are derived from that local copy of the Per archive. The hundreds of missing files set the names of hundreds of people to NA. I am going to sync the archive to another server today and am going to check whether the problem remains. I need to make sure that the files are really missing on Aigtu and there is not a glitch somewhere in my code. Have a nice day. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Samstag, 3. Oktober 2020 17:11 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Christian Zimmermann <zimmermann@stlouisfed.org>; collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc Düben, Christian writes

The files meant for the ftp server are in /home/icanis/ftp/opt. The rank variable in these files handles ties with the "min" option. Assume we have four authors with closeness values of 0.5, 0.3, 0.3 and 0.2 respectively. With the "min" option their ranks are 1, 2, 2 and 4.

Seems fine, but where is the path data? CollEc generates path data. As long as your system does does not produce path data, it is not a replacement for the existing CollEc.

You mentioned that your code fixes the rsync permission problem over time. What time frame did you have in mind? It does not work yet.

Then it does not work. The issue had nothing to do with permissions, but with the fact that remi at aigtu was completely broken. That is fixed.

Do I need to add anything to the shell script?

I doubt it. If the data does not update the problem is that aigtu does not have an up-to-date copy of RePEc:per.

Before Christian Zimmermann links any IDEAS content to the web application's entry points, we should discuss the URL.

Collec2 is no good. What about app.collec? It will become collec.repec.org when it can replace the functionality. I'm much looking forward to that as the current calculations block helos for any other talks. We need to move this to helos. There you will have more computing power.

Could you add a placeholder HTML page at graphec.repec.org? Something like "GraphEc is currently publicly unavailable. Contact <a href="http://www.christian-dueben.com">Christian Düben</a> regarding the application's public release.". The new CollEc already links to GraphEc and I think that a placeholder page with a short informative text would look nicer than the Apache2 Debian default page.

I suggest that if graphec is not available, you should not link to it. When helos was installed, debian had no controller for its NIC. Thus it's an ubuntu machine. Will that be a problem? I could reinstall helos, but I don't have disk space to at least temporarily store the data it houses now. Generally, my machines are running full. We are short of disk space. It has been difficult to get further resources because of the crisis. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

I reinstalled the containerized MariaDB which fixed the issue of the unexpectedly large ibdata1 file in it. A script is currently computing CollEc's results again and inserts the tables into the new containerized data base. The app does not work as intended until this process is complete, i.e. until around 11 pm CEST. This temporary "shutdown" of the app only happens because I set up a new containerized data base. The regular daily updating routine does not impede the app's availability. The cleanup freed around 50 GB of disk space. The new CollEc's total size is around 28 GB. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of Düben, Christian Sent: Montag, 5. Oktober 2020 19:04 To: Thomas Krichel <krichel@openlib.org> Cc: collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc I fixed the Per archive issue. The files are not missing on Aigtu. An intermediate step in the daily updating routine was incomplete. It works now. Sorry about that. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of Düben, Christian Sent: Sonntag, 4. Oktober 2020 13:12 To: Thomas Krichel <krichel@openlib.org> Cc: collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc Do you mean the shortest paths between any two authors in the network? Users can look at their shortest paths in the Shortest Paths tab. That functionality computes them directly from the graph. The distances and the closeness measures also rely on these shortest paths. But at no point are they written to disk. They are only stored in memory. I did some extensive testing and writing these paths to disk is incredibly inefficient. There is a reason why the new CollEc's daily updating routine takes only a tiny fraction of the time and computational resources required by the old CollEc. It is the eradication of inefficiencies along multiple dimensions. Unless there is a very good reason why these paths should be written to disk, I do not see a point of doing so. Are they used anywhere outside CollEc? app.collec.repec.org sounds good. Let me know when you changed it. I am then going to adjust the documentation on the entry points. It is not a problem that Helos is an Ubuntu system. In fact, I prefer Ubuntu over Debian. Nonetheless, I am really not looking forward to that migration as I have to set up and test the entire framework again. There should be easier ways to directly migrating the CollEc system as a whole instead of installing it again piece by piece. But that is beyond my current technical skills. The new CollEc requires rather little disk space. The only component that appears unexpectedly large is the containerized MariaDB. I am going to check tomorrow what might be wrong with it. The authors' full names are derived from that local copy of the Per archive. The hundreds of missing files set the names of hundreds of people to NA. I am going to sync the archive to another server today and am going to check whether the problem remains. I need to make sure that the files are really missing on Aigtu and there is not a glitch somewhere in my code. Have a nice day. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Samstag, 3. Oktober 2020 17:11 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Christian Zimmermann <zimmermann@stlouisfed.org>; collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc Düben, Christian writes

The files meant for the ftp server are in /home/icanis/ftp/opt. The rank variable in these files handles ties with the "min" option. Assume we have four authors with closeness values of 0.5, 0.3, 0.3 and 0.2 respectively. With the "min" option their ranks are 1, 2, 2 and 4.

Seems fine, but where is the path data? CollEc generates path data. As long as your system does does not produce path data, it is not a replacement for the existing CollEc.

You mentioned that your code fixes the rsync permission problem over time. What time frame did you have in mind? It does not work yet.

Then it does not work. The issue had nothing to do with permissions, but with the fact that remi at aigtu was completely broken. That is fixed.

Do I need to add anything to the shell script?

I doubt it. If the data does not update the problem is that aigtu does not have an up-to-date copy of RePEc:per.

Before Christian Zimmermann links any IDEAS content to the web application's entry points, we should discuss the URL.

Collec2 is no good. What about app.collec? It will become collec.repec.org when it can replace the functionality. I'm much looking forward to that as the current calculations block helos for any other talks. We need to move this to helos. There you will have more computing power.

Could you add a placeholder HTML page at graphec.repec.org? Something like "GraphEc is currently publicly unavailable. Contact <a href="http://www.christian-dueben.com">Christian Düben</a> regarding the application's public release.". The new CollEc already links to GraphEc and I think that a placeholder page with a short informative text would look nicer than the Apache2 Debian default page.

I suggest that if graphec is not available, you should not link to it. When helos was installed, debian had no controller for its NIC. Thus it's an ubuntu machine. Will that be a problem? I could reinstall helos, but I don't have disk space to at least temporarily store the data it houses now. Generally, my machines are running full. We are short of disk space. It has been difficult to get further resources because of the crisis. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel _______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run _______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run

Finally getting to it... Excellent post. I will try to massage it into the blog format. See me the proper links. Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist On Tue, 6 Oct 2020, Dï¿œben, Christian wrote:

As requested, I came up with a RePEc blog post. You can find a draft attached to this e-mail. How do you like it?

Do not post it yet. The hyperlinks already point to app.collec.repec.org which is yet to be connected to the app.

Christian Dï¿œben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of Dï¿œben, Christian Sent: Montag, 5. Oktober 2020 22:04 To: Thomas Krichel <krichel@openlib.org> Cc: collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

I reinstalled the containerized MariaDB which fixed the issue of the unexpectedly large ibdata1 file in it. A script is currently computing CollEc's results again and inserts the tables into the new containerized data base. The app does not work as intended until this process is complete, i.e. until around 11 pm CEST. This temporary "shutdown" of the app only happens because I set up a new containerized data base. The regular daily updating routine does not impede the app's availability.

The cleanup freed around 50 GB of disk space. The new CollEc's total size is around 28 GB.

Christian Dï¿œben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of Dï¿œben, Christian Sent: Montag, 5. Oktober 2020 19:04 To: Thomas Krichel <krichel@openlib.org> Cc: collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

I fixed the Per archive issue. The files are not missing on Aigtu. An intermediate step in the daily updating routine was incomplete. It works now. Sorry about that.

Christian Dï¿œben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of Dï¿œben, Christian Sent: Sonntag, 4. Oktober 2020 13:12 To: Thomas Krichel <krichel@openlib.org> Cc: collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

Do you mean the shortest paths between any two authors in the network? Users can look at their shortest paths in the Shortest Paths tab. That functionality computes them directly from the graph. The distances and the closeness measures also rely on these shortest paths. But at no point are they written to disk. They are only stored in memory. I did some extensive testing and writing these paths to disk is incredibly inefficient. There is a reason why the new CollEc's daily updating routine takes only a tiny fraction of the time and computational resources required by the old CollEc. It is the eradication of inefficiencies along multiple dimensions. Unless there is a very good reason why these paths should be written to disk, I do not see a point of doing so. Are they used anywhere outside CollEc?

app.collec.repec.org sounds good. Let me know when you changed it. I am then going to adjust the documentation on the entry points.

It is not a problem that Helos is an Ubuntu system. In fact, I prefer Ubuntu over Debian. Nonetheless, I am really not looking forward to that migration as I have to set up and test the entire framework again. There should be easier ways to directly migrating the CollEc system as a whole instead of installing it again piece by piece. But that is beyond my current technical skills.

The new CollEc requires rather little disk space. The only component that appears unexpectedly large is the containerized MariaDB. I am going to check tomorrow what might be wrong with it.

The authors' full names are derived from that local copy of the Per archive. The hundreds of missing files set the names of hundreds of people to NA. I am going to sync the archive to another server today and am going to check whether the problem remains. I need to make sure that the files are really missing on Aigtu and there is not a glitch somewhere in my code.

Have a nice day.

Christian Dï¿œben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Samstag, 3. Oktober 2020 17:11 To: Dï¿œben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Christian Zimmermann <zimmermann@stlouisfed.org>; collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

Dï¿œben, Christian writes

...

The files meant for the ftp server are in /home/icanis/ftp/opt. The rank variable in these files handles ties with the "min" option. Assume we have four authors with closeness values of 0.5, 0.3, 0.3 and 0.2 respectively. With the "min" option their ranks are 1, 2, 2 and 4.

Seems fine, but where is the path data?

CollEc generates path data. As long as your system does does not produce path data, it is not a replacement for the existing CollEc.

...

You mentioned that your code fixes the rsync permission problem over time. What time frame did you have in mind? It does not work yet.

Then it does not work. The issue had nothing to do with permissions, but with the fact that remi at aigtu was completely broken. That is fixed.

...

Do I need to add anything to the shell script?

I doubt it. If the data does not update the problem is that aigtu does not have an up-to-date copy of RePEc:per.

...

Before Christian Zimmermann links any IDEAS content to the web application's entry points, we should discuss the URL.

Collec2 is no good. What about app.collec? It will become collec.repec.org when it can replace the functionality. I'm much looking forward to that as the current calculations block helos for any other talks.

We need to move this to helos. There you will have more computing power.

...

Could you add a placeholder HTML page at graphec.repec.org? Something like "GraphEc is currently publicly unavailable. Contact <a href="http://www.christian-dueben.com">Christian Dï¿œben</a> regarding the application's public release.". The new CollEc already links to GraphEc and I think that a placeholder page with a short informative text would look nicer than the Apache2 Debian default page.

I suggest that if graphec is not available, you should not link to it.

When helos was installed, debian had no controller for its NIC. Thus it's an ubuntu machine. Will that be a problem? I could reinstall helos, but I don't have disk space to at least temporarily store the data it houses now.

Generally, my machines are running full. We are short of disk space. It has been difficult to get further resources because of the crisis.

--

Cheers,

Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run

_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run

_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run

I'll take the co-authors. app.collec.repec.org is not going to change, right? Also, authors will start complaining that the data in the new CollEc does not match the data in their statistics. Where can I find the new data? Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist On Wed, 25 Nov 2020, D�ben, Christian wrote:

Thanks. I am glad you like it.

Which links you use for the author pages depends on you. Which tab would you like to use as a landing page? Each of the output-generating tabs (Distances, Closeness, Betweenness, Co-Authors, Shortest Paths) has an entry point.

Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: Christian Zimmermann <zimmermann@stlouisfed.org> Sent: Mittwoch, 25. November 2020 20:09 To: D�ben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Thomas Krichel <krichel@openlib.org>; collec-run@lists.openlib.org Subject: RE: [CollEc] New CollEc

Finally getting to it...

Excellent post. I will try to massage it into the blog format. See me the proper links.

Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist

On Tue, 6 Oct 2020, Dï¿œben, Christian wrote:

...

As requested, I came up with a RePEc blog post. You can find a draft attached to this e-mail. How do you like it?

Do not post it yet. The hyperlinks already point to app.collec.repec.org which is yet to be connected to the app.

Christian Dï¿œben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of Dï¿œben, Christian Sent: Montag, 5. Oktober 2020 22:04 To: Thomas Krichel <krichel@openlib.org> Cc: collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

I reinstalled the containerized MariaDB which fixed the issue of the unexpectedly large ibdata1 file in it. A script is currently computing CollEc's results again and inserts the tables into the new containerized data base. The app does not work as intended until this process is complete, i.e. until around 11 pm CEST. This temporary "shutdown" of the app only happens because I set up a new containerized data base. The regular daily updating routine does not impede the app's availability.

The cleanup freed around 50 GB of disk space. The new CollEc's total size is around 28 GB.

Christian Dï¿œben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of Dï¿œben, Christian Sent: Montag, 5. Oktober 2020 19:04 To: Thomas Krichel <krichel@openlib.org> Cc: collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

I fixed the Per archive issue. The files are not missing on Aigtu. An intermediate step in the daily updating routine was incomplete. It works now. Sorry about that.

Christian Dï¿œben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of Dï¿œben, Christian Sent: Sonntag, 4. Oktober 2020 13:12 To: Thomas Krichel <krichel@openlib.org> Cc: collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

Do you mean the shortest paths between any two authors in the network? Users can look at their shortest paths in the Shortest Paths tab. That functionality computes them directly from the graph. The distances and the closeness measures also rely on these shortest paths. But at no point are they written to disk. They are only stored in memory. I did some extensive testing and writing these paths to disk is incredibly inefficient. There is a reason why the new CollEc's daily updating routine takes only a tiny fraction of the time and computational resources required by the old CollEc. It is the eradication of inefficiencies along multiple dimensions. Unless there is a very good reason why these paths should be written to disk, I do not see a point of doing so. Are they used anywhere outside CollEc?

app.collec.repec.org sounds good. Let me know when you changed it. I am then going to adjust the documentation on the entry points.

It is not a problem that Helos is an Ubuntu system. In fact, I prefer Ubuntu over Debian. Nonetheless, I am really not looking forward to that migration as I have to set up and test the entire framework again. There should be easier ways to directly migrating the CollEc system as a whole instead of installing it again piece by piece. But that is beyond my current technical skills.

The new CollEc requires rather little disk space. The only component that appears unexpectedly large is the containerized MariaDB. I am going to check tomorrow what might be wrong with it.

The authors' full names are derived from that local copy of the Per archive. The hundreds of missing files set the names of hundreds of people to NA. I am going to sync the archive to another server today and am going to check whether the problem remains. I need to make sure that the files are really missing on Aigtu and there is not a glitch somewhere in my code.

Have a nice day.

Christian Dï¿œben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Samstag, 3. Oktober 2020 17:11 To: Dï¿œben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Christian Zimmermann <zimmermann@stlouisfed.org>; collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

Dï¿œben, Christian writes

...

The files meant for the ftp server are in /home/icanis/ftp/opt. The rank variable in these files handles ties with the "min" option. Assume we have four authors with closeness values of 0.5, 0.3, 0.3 and 0.2 respectively. With the "min" option their ranks are 1, 2, 2 and 4.

Seems fine, but where is the path data?

CollEc generates path data. As long as your system does does not produce path data, it is not a replacement for the existing CollEc.

...

You mentioned that your code fixes the rsync permission problem over time. What time frame did you have in mind? It does not work yet.

Then it does not work. The issue had nothing to do with permissions, but with the fact that remi at aigtu was completely broken. That is fixed.

...

Do I need to add anything to the shell script?

I doubt it. If the data does not update the problem is that aigtu does not have an up-to-date copy of RePEc:per.

...

Before Christian Zimmermann links any IDEAS content to the web application's entry points, we should discuss the URL.

Collec2 is no good. What about app.collec? It will become collec.repec.org when it can replace the functionality. I'm much looking forward to that as the current calculations block helos for any other talks.

We need to move this to helos. There you will have more computing power.

...

Could you add a placeholder HTML page at graphec.repec.org? Something like "GraphEc is currently publicly unavailable. Contact <a href="http://www.christian-dueben.com">Christian Dï¿œben</a> regarding the application's public release.". The new CollEc already links to GraphEc and I think that a placeholder page with a short informative text would look nicer than the Apache2 Debian default page.

I suggest that if graphec is not available, you should not link to it.

When helos was installed, debian had no controller for its NIC. Thus it's an ubuntu machine. Will that be a problem? I could reinstall helos, but I don't have disk space to at least temporarily store the data it houses now.

Generally, my machines are running full. We are short of disk space. It has been difficult to get further resources because of the crisis.

--

Cheers,

Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run

_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run

_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run

Yes, I mean the app data. Blog post is almost ready. I cannot find a way to share it with you without publishing it... but I think I copied everyting over and only changed links a little and what you suggested below. I wanted ot make a fresh and clear picture of the graph you included in your pdf. Unfortunately, I cannot recreate it, as you do not seem to be in the database... Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist On Wed, 25 Nov 2020, D�ben, Christian wrote:

As far as I am concerned, the app is going to remain at app.collec.repec.org.

Do you mean the app's data that Thomas made available through the ftp server?

Could you change "Co-authored research has been on the rise over the past decades forming collaborations over vast geographic distances and fields of research" in the blog post's second paragraph to "Co-authored research has been on the rise over the past decades, forming collaborations over enormous geographic distances and many fields of research"?

Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: Christian Zimmermann <zimmermann@stlouisfed.org> Sent: Mittwoch, 25. November 2020 20:21 To: D�ben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Thomas Krichel <krichel@openlib.org>; collec-run@lists.openlib.org Subject: RE: [CollEc] New CollEc

I'll take the co-authors. app.collec.repec.org is not going to change, right?

Also, authors will start complaining that the data in the new CollEc does not match the data in their statistics. Where can I find the new data?

Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist

On Wed, 25 Nov 2020, D�ben, Christian wrote:

...

Thanks. I am glad you like it.

Which links you use for the author pages depends on you. Which tab would you like to use as a landing page? Each of the output-generating tabs (Distances, Closeness, Betweenness, Co-Authors, Shortest Paths) has an entry point.

Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: Christian Zimmermann <zimmermann@stlouisfed.org> Sent: Mittwoch, 25. November 2020 20:09 To: D�ben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Thomas Krichel <krichel@openlib.org>; collec-run@lists.openlib.org Subject: RE: [CollEc] New CollEc

Finally getting to it...

Excellent post. I will try to massage it into the blog format. See me the proper links.

Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist

On Tue, 6 Oct 2020, Dï¿œben, Christian wrote:

...

As requested, I came up with a RePEc blog post. You can find a draft attached to this e-mail. How do you like it?

Do not post it yet. The hyperlinks already point to app.collec.repec.org which is yet to be connected to the app.

Christian Dï¿œben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of Dï¿œben, Christian Sent: Montag, 5. Oktober 2020 22:04 To: Thomas Krichel <krichel@openlib.org> Cc: collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

I reinstalled the containerized MariaDB which fixed the issue of the unexpectedly large ibdata1 file in it. A script is currently computing CollEc's results again and inserts the tables into the new containerized data base. The app does not work as intended until this process is complete, i.e. until around 11 pm CEST. This temporary "shutdown" of the app only happens because I set up a new containerized data base. The regular daily updating routine does not impede the app's availability.

The cleanup freed around 50 GB of disk space. The new CollEc's total size is around 28 GB.

Christian Dï¿œben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of Dï¿œben, Christian Sent: Montag, 5. Oktober 2020 19:04 To: Thomas Krichel <krichel@openlib.org> Cc: collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

I fixed the Per archive issue. The files are not missing on Aigtu. An intermediate step in the daily updating routine was incomplete. It works now. Sorry about that.

Christian Dï¿œben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of Dï¿œben, Christian Sent: Sonntag, 4. Oktober 2020 13:12 To: Thomas Krichel <krichel@openlib.org> Cc: collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

Do you mean the shortest paths between any two authors in the network? Users can look at their shortest paths in the Shortest Paths tab. That functionality computes them directly from the graph. The distances and the closeness measures also rely on these shortest paths. But at no point are they written to disk. They are only stored in memory. I did some extensive testing and writing these paths to disk is incredibly inefficient. There is a reason why the new CollEc's daily updating routine takes only a tiny fraction of the time and computational resources required by the old CollEc. It is the eradication of inefficiencies along multiple dimensions. Unless there is a very good reason why these paths should be written to disk, I do not see a point of doing so. Are they used anywhere outside CollEc?

app.collec.repec.org sounds good. Let me know when you changed it. I am then going to adjust the documentation on the entry points.

It is not a problem that Helos is an Ubuntu system. In fact, I prefer Ubuntu over Debian. Nonetheless, I am really not looking forward to that migration as I have to set up and test the entire framework again. There should be easier ways to directly migrating the CollEc system as a whole instead of installing it again piece by piece. But that is beyond my current technical skills.

The new CollEc requires rather little disk space. The only component that appears unexpectedly large is the containerized MariaDB. I am going to check tomorrow what might be wrong with it.

The authors' full names are derived from that local copy of the Per archive. The hundreds of missing files set the names of hundreds of people to NA. I am going to sync the archive to another server today and am going to check whether the problem remains. I need to make sure that the files are really missing on Aigtu and there is not a glitch somewhere in my code.

Have a nice day.

Christian Dï¿œben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Samstag, 3. Oktober 2020 17:11 To: Dï¿œben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Christian Zimmermann <zimmermann@stlouisfed.org>; collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

Dï¿œben, Christian writes

...

The files meant for the ftp server are in /home/icanis/ftp/opt. The rank variable in these files handles ties with the "min" option. Assume we have four authors with closeness values of 0.5, 0.3, 0.3 and 0.2 respectively. With the "min" option their ranks are 1, 2, 2 and 4.

Seems fine, but where is the path data?

CollEc generates path data. As long as your system does does not produce path data, it is not a replacement for the existing CollEc.

...

You mentioned that your code fixes the rsync permission problem over time. What time frame did you have in mind? It does not work yet.

Then it does not work. The issue had nothing to do with permissions, but with the fact that remi at aigtu was completely broken. That is fixed.

...

Do I need to add anything to the shell script?

I doubt it. If the data does not update the problem is that aigtu does not have an up-to-date copy of RePEc:per.

...

Before Christian Zimmermann links any IDEAS content to the web application's entry points, we should discuss the URL.

Collec2 is no good. What about app.collec? It will become collec.repec.org when it can replace the functionality. I'm much looking forward to that as the current calculations block helos for any other talks.

We need to move this to helos. There you will have more computing power.

...

Could you add a placeholder HTML page at graphec.repec.org? Something like "GraphEc is currently publicly unavailable. Contact <a href="http://www.christian-dueben.com">Christian Dï¿œben</a> regarding the application's public release.". The new CollEc already links to GraphEc and I think that a placeholder page with a short informative text would look nicer than the Apache2 Debian default page.

I suggest that if graphec is not available, you should not link to it.

When helos was installed, debian had no controller for its NIC. Thus it's an ubuntu machine. Will that be a problem? I could reinstall helos, but I don't have disk space to at least temporarily store the data it houses now.

Generally, my machines are running full. We are short of disk space. It has been difficult to get further resources because of the crisis.

--

Cheers,

Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run

_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run

_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run

All I needed was this image file. Blog post is published. First batch of IDEAS user profiles has links to new CollEc. When I search for "Christian D" your name does not appear (No umlaut on the current keyword) Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist On Thu, 26 Nov 2020, D�ben, Christian wrote:

I do not know the externally available app data's ftp address. I simply put it in /home/icanis/ftp/opt on darni. And Thomas takes care of the ftp part.

Why am I not in the database? When I enter my name in the distances tab, which the plot is based on, it works.

You mention that you want a fresh and clear picture. If you tell me how exactly you want to change the plot, I can generate it and send it to you. Do you need a higher resolution? See the figure attached to this e-mail, i.e. the plot used in the pdf, as a reference.

Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: Christian Zimmermann <zimmermann@stlouisfed.org> Sent: Donnerstag, 26. November 2020 05:01 To: D�ben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Thomas Krichel <krichel@openlib.org>; collec-run@lists.openlib.org Subject: RE: [CollEc] New CollEc

Yes, I mean the app data.

Blog post is almost ready. I cannot find a way to share it with you without publishing it... but I think I copied everyting over and only changed links a little and what you suggested below.

I wanted ot make a fresh and clear picture of the graph you included in your pdf. Unfortunately, I cannot recreate it, as you do not seem to be in the database...

Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist

On Wed, 25 Nov 2020, D�ben, Christian wrote:

...

As far as I am concerned, the app is going to remain at app.collec.repec.org.

Do you mean the app's data that Thomas made available through the ftp server?

Could you change "Co-authored research has been on the rise over the past decades forming collaborations over vast geographic distances and fields of research" in the blog post's second paragraph to "Co-authored research has been on the rise over the past decades, forming collaborations over enormous geographic distances and many fields of research"?

Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: Christian Zimmermann <zimmermann@stlouisfed.org> Sent: Mittwoch, 25. November 2020 20:21 To: D�ben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Thomas Krichel <krichel@openlib.org>; collec-run@lists.openlib.org Subject: RE: [CollEc] New CollEc

I'll take the co-authors. app.collec.repec.org is not going to change, right?

Also, authors will start complaining that the data in the new CollEc does not match the data in their statistics. Where can I find the new data?

Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist

On Wed, 25 Nov 2020, D�ben, Christian wrote:

...

Thanks. I am glad you like it.

Which links you use for the author pages depends on you. Which tab would you like to use as a landing page? Each of the output-generating tabs (Distances, Closeness, Betweenness, Co-Authors, Shortest Paths) has an entry point.

Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: Christian Zimmermann <zimmermann@stlouisfed.org> Sent: Mittwoch, 25. November 2020 20:09 To: D�ben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Thomas Krichel <krichel@openlib.org>; collec-run@lists.openlib.org Subject: RE: [CollEc] New CollEc

Finally getting to it...

Excellent post. I will try to massage it into the blog format. See me the proper links.

Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist

On Tue, 6 Oct 2020, Dï¿œben, Christian wrote:

...

As requested, I came up with a RePEc blog post. You can find a draft attached to this e-mail. How do you like it?

Do not post it yet. The hyperlinks already point to app.collec.repec.org which is yet to be connected to the app.

Christian Dï¿œben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of Dï¿œben, Christian Sent: Montag, 5. Oktober 2020 22:04 To: Thomas Krichel <krichel@openlib.org> Cc: collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

I reinstalled the containerized MariaDB which fixed the issue of the unexpectedly large ibdata1 file in it. A script is currently computing CollEc's results again and inserts the tables into the new containerized data base. The app does not work as intended until this process is complete, i.e. until around 11 pm CEST. This temporary "shutdown" of the app only happens because I set up a new containerized data base. The regular daily updating routine does not impede the app's availability.

The cleanup freed around 50 GB of disk space. The new CollEc's total size is around 28 GB.

Christian Dï¿œben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of Dï¿œben, Christian Sent: Montag, 5. Oktober 2020 19:04 To: Thomas Krichel <krichel@openlib.org> Cc: collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

I fixed the Per archive issue. The files are not missing on Aigtu. An intermediate step in the daily updating routine was incomplete. It works now. Sorry about that.

Christian Dï¿œben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of Dï¿œben, Christian Sent: Sonntag, 4. Oktober 2020 13:12 To: Thomas Krichel <krichel@openlib.org> Cc: collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

Do you mean the shortest paths between any two authors in the network? Users can look at their shortest paths in the Shortest Paths tab. That functionality computes them directly from the graph. The distances and the closeness measures also rely on these shortest paths. But at no point are they written to disk. They are only stored in memory. I did some extensive testing and writing these paths to disk is incredibly inefficient. There is a reason why the new CollEc's daily updating routine takes only a tiny fraction of the time and computational resources required by the old CollEc. It is the eradication of inefficiencies along multiple dimensions. Unless there is a very good reason why these paths should be written to disk, I do not see a point of doing so. Are they used anywhere outside CollEc?

app.collec.repec.org sounds good. Let me know when you changed it. I am then going to adjust the documentation on the entry points.

It is not a problem that Helos is an Ubuntu system. In fact, I prefer Ubuntu over Debian. Nonetheless, I am really not looking forward to that migration as I have to set up and test the entire framework again. There should be easier ways to directly migrating the CollEc system as a whole instead of installing it again piece by piece. But that is beyond my current technical skills.

The new CollEc requires rather little disk space. The only component that appears unexpectedly large is the containerized MariaDB. I am going to check tomorrow what might be wrong with it.

The authors' full names are derived from that local copy of the Per archive. The hundreds of missing files set the names of hundreds of people to NA. I am going to sync the archive to another server today and am going to check whether the problem remains. I need to make sure that the files are really missing on Aigtu and there is not a glitch somewhere in my code.

Have a nice day.

Christian Dï¿œben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Samstag, 3. Oktober 2020 17:11 To: Dï¿œben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Christian Zimmermann <zimmermann@stlouisfed.org>; collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

Dï¿œben, Christian writes

...

The files meant for the ftp server are in /home/icanis/ftp/opt. The rank variable in these files handles ties with the "min" option. Assume we have four authors with closeness values of 0.5, 0.3, 0.3 and 0.2 respectively. With the "min" option their ranks are 1, 2, 2 and 4.

Seems fine, but where is the path data?

CollEc generates path data. As long as your system does does not produce path data, it is not a replacement for the existing CollEc.

...

You mentioned that your code fixes the rsync permission problem over time. What time frame did you have in mind? It does not work yet.

Then it does not work. The issue had nothing to do with permissions, but with the fact that remi at aigtu was completely broken. That is fixed.

...

Do I need to add anything to the shell script?

I doubt it. If the data does not update the problem is that aigtu does not have an up-to-date copy of RePEc:per.

...

Before Christian Zimmermann links any IDEAS content to the web application's entry points, we should discuss the URL.

Collec2 is no good. What about app.collec? It will become collec.repec.org when it can replace the functionality. I'm much looking forward to that as the current calculations block helos for any other talks.

We need to move this to helos. There you will have more computing power.

...

Could you add a placeholder HTML page at graphec.repec.org? Something like "GraphEc is currently publicly unavailable. Contact <a href="http://www.christian-dueben.com">Christian Dï¿œben</a> regarding the application's public release.". The new CollEc already links to GraphEc and I think that a placeholder page with a short informative text would look nicer than the Apache2 Debian default page.

I suggest that if graphec is not available, you should not link to it.

When helos was installed, debian had no controller for its NIC. Thus it's an ubuntu machine. Will that be a problem? I could reinstall helos, but I don't have disk space to at least temporarily store the data it houses now.

Generally, my machines are running full. We are short of disk space. It has been difficult to get further resources because of the crisis.

--

Cheers,

Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run

_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run

_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run

There is just a little error in "volunteering options" link, which is why it does not work. It is "https://ideas.repec.org/volunteers.htmnl", but should be "https://ideas.repec.org/volunteers.html". Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Christian Zimmermann <zimmermann@stlouisfed.org> Sent: Donnerstag, 26. November 2020 15:45 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Thomas Krichel <krichel@openlib.org>; collec-run@lists.openlib.org Subject: RE: [CollEc] New CollEc All I needed was this image file. Blog post is published. First batch of IDEAS user profiles has links to new CollEc. When I search for "Christian D" your name does not appear (No umlaut on the current keyword) Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist On Thu, 26 Nov 2020, D�ben, Christian wrote:

I do not know the externally available app data's ftp address. I simply put it in /home/icanis/ftp/opt on darni. And Thomas takes care of the ftp part.

Why am I not in the database? When I enter my name in the distances tab, which the plot is based on, it works.

You mention that you want a fresh and clear picture. If you tell me how exactly you want to change the plot, I can generate it and send it to you. Do you need a higher resolution? See the figure attached to this e-mail, i.e. the plot used in the pdf, as a reference.

Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: Christian Zimmermann <zimmermann@stlouisfed.org> Sent: Donnerstag, 26. November 2020 05:01 To: D�ben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Thomas Krichel <krichel@openlib.org>; collec-run@lists.openlib.org Subject: RE: [CollEc] New CollEc

Yes, I mean the app data.

Blog post is almost ready. I cannot find a way to share it with you without publishing it... but I think I copied everyting over and only changed links a little and what you suggested below.

I wanted ot make a fresh and clear picture of the graph you included in your pdf. Unfortunately, I cannot recreate it, as you do not seem to be in the database...

Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist

On Wed, 25 Nov 2020, D�ben, Christian wrote:

...

As far as I am concerned, the app is going to remain at app.collec.repec.org.

Do you mean the app's data that Thomas made available through the ftp server?

Could you change "Co-authored research has been on the rise over the past decades forming collaborations over vast geographic distances and fields of research" in the blog post's second paragraph to "Co-authored research has been on the rise over the past decades, forming collaborations over enormous geographic distances and many fields of research"?

Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: Christian Zimmermann <zimmermann@stlouisfed.org> Sent: Mittwoch, 25. November 2020 20:21 To: D�ben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Thomas Krichel <krichel@openlib.org>; collec-run@lists.openlib.org Subject: RE: [CollEc] New CollEc

I'll take the co-authors. app.collec.repec.org is not going to change, right?

Also, authors will start complaining that the data in the new CollEc does not match the data in their statistics. Where can I find the new data?

Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist

On Wed, 25 Nov 2020, D�ben, Christian wrote:

...

Thanks. I am glad you like it.

Which links you use for the author pages depends on you. Which tab would you like to use as a landing page? Each of the output-generating tabs (Distances, Closeness, Betweenness, Co-Authors, Shortest Paths) has an entry point.

Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: Christian Zimmermann <zimmermann@stlouisfed.org> Sent: Mittwoch, 25. November 2020 20:09 To: D�ben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Thomas Krichel <krichel@openlib.org>; collec-run@lists.openlib.org Subject: RE: [CollEc] New CollEc

Finally getting to it...

Excellent post. I will try to massage it into the blog format. See me the proper links.

Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist

On Tue, 6 Oct 2020, Dï¿œben, Christian wrote:

...

As requested, I came up with a RePEc blog post. You can find a draft attached to this e-mail. How do you like it?

Do not post it yet. The hyperlinks already point to app.collec.repec.org which is yet to be connected to the app.

Christian Dï¿œben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of Dï¿œben, Christian Sent: Montag, 5. Oktober 2020 22:04 To: Thomas Krichel <krichel@openlib.org> Cc: collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

I reinstalled the containerized MariaDB which fixed the issue of the unexpectedly large ibdata1 file in it. A script is currently computing CollEc's results again and inserts the tables into the new containerized data base. The app does not work as intended until this process is complete, i.e. until around 11 pm CEST. This temporary "shutdown" of the app only happens because I set up a new containerized data base. The regular daily updating routine does not impede the app's availability.

The cleanup freed around 50 GB of disk space. The new CollEc's total size is around 28 GB.

Christian Dï¿œben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of Dï¿œben, Christian Sent: Montag, 5. Oktober 2020 19:04 To: Thomas Krichel <krichel@openlib.org> Cc: collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

I fixed the Per archive issue. The files are not missing on Aigtu. An intermediate step in the daily updating routine was incomplete. It works now. Sorry about that.

Christian Dï¿œben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of Dï¿œben, Christian Sent: Sonntag, 4. Oktober 2020 13:12 To: Thomas Krichel <krichel@openlib.org> Cc: collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

Do you mean the shortest paths between any two authors in the network? Users can look at their shortest paths in the Shortest Paths tab. That functionality computes them directly from the graph. The distances and the closeness measures also rely on these shortest paths. But at no point are they written to disk. They are only stored in memory. I did some extensive testing and writing these paths to disk is incredibly inefficient. There is a reason why the new CollEc's daily updating routine takes only a tiny fraction of the time and computational resources required by the old CollEc. It is the eradication of inefficiencies along multiple dimensions. Unless there is a very good reason why these paths should be written to disk, I do not see a point of doing so. Are they used anywhere outside CollEc?

app.collec.repec.org sounds good. Let me know when you changed it. I am then going to adjust the documentation on the entry points.

It is not a problem that Helos is an Ubuntu system. In fact, I prefer Ubuntu over Debian. Nonetheless, I am really not looking forward to that migration as I have to set up and test the entire framework again. There should be easier ways to directly migrating the CollEc system as a whole instead of installing it again piece by piece. But that is beyond my current technical skills.

The new CollEc requires rather little disk space. The only component that appears unexpectedly large is the containerized MariaDB. I am going to check tomorrow what might be wrong with it.

The authors' full names are derived from that local copy of the Per archive. The hundreds of missing files set the names of hundreds of people to NA. I am going to sync the archive to another server today and am going to check whether the problem remains. I need to make sure that the files are really missing on Aigtu and there is not a glitch somewhere in my code.

Have a nice day.

Christian Dï¿œben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Samstag, 3. Oktober 2020 17:11 To: Dï¿œben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Christian Zimmermann <zimmermann@stlouisfed.org>; collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

Dï¿œben, Christian writes

...

The files meant for the ftp server are in /home/icanis/ftp/opt. The rank variable in these files handles ties with the "min" option. Assume we have four authors with closeness values of 0.5, 0.3, 0.3 and 0.2 respectively. With the "min" option their ranks are 1, 2, 2 and 4.

Seems fine, but where is the path data?

CollEc generates path data. As long as your system does does not produce path data, it is not a replacement for the existing CollEc.

...

You mentioned that your code fixes the rsync permission problem over time. What time frame did you have in mind? It does not work yet.

Then it does not work. The issue had nothing to do with permissions, but with the fact that remi at aigtu was completely broken. That is fixed.

...

Do I need to add anything to the shell script?

I doubt it. If the data does not update the problem is that aigtu does not have an up-to-date copy of RePEc:per.

...

Before Christian Zimmermann links any IDEAS content to the web application's entry points, we should discuss the URL.

Collec2 is no good. What about app.collec? It will become collec.repec.org when it can replace the functionality. I'm much looking forward to that as the current calculations block helos for any other talks.

We need to move this to helos. There you will have more computing power.

...

Could you add a placeholder HTML page at graphec.repec.org? Something like "GraphEc is currently publicly unavailable. Contact <a href="http://www.christian-dueben.com">Christian Dï¿œben</a> regarding the application's public release.". The new CollEc already links to GraphEc and I think that a placeholder page with a short informative text would look nicer than the Apache2 Debian default page.

I suggest that if graphec is not available, you should not link to it.

When helos was installed, debian had no controller for its NIC. Thus it's an ubuntu machine. Will that be a problem? I could reinstall helos, but I don't have disk space to at least temporarily store the data it houses now.

Generally, my machines are running full. We are short of disk space. It has been difficult to get further resources because of the crisis.

--

Cheers,

Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run

_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run

_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run

I guess, Thomas will clarify this. Christian Düben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com -----Original Message----- From: Christian Zimmermann <zimmermann@stlouisfed.org> Sent: Donnerstag, 26. November 2020 16:50 To: Düben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Thomas Krichel <krichel@openlib.org>; collec-run@lists.openlib.org Subject: RE: [CollEc] New CollEc Corrected. Now all I need is the new data feed. Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist On Thu, 26 Nov 2020, D�ben, Christian wrote:

There is just a little error in "volunteering options" link, which is why it does not work. It is "https://ideas.repec.org/volunteers.htmnl", but should be "https://ideas.repec.org/volunteers.html".

Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: Christian Zimmermann <zimmermann@stlouisfed.org> Sent: Donnerstag, 26. November 2020 15:45 To: D�ben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Thomas Krichel <krichel@openlib.org>; collec-run@lists.openlib.org Subject: RE: [CollEc] New CollEc

All I needed was this image file.

Blog post is published.

First batch of IDEAS user profiles has links to new CollEc.

When I search for "Christian D" your name does not appear (No umlaut on the current keyword)

Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist

On Thu, 26 Nov 2020, D�ben, Christian wrote:

...

I do not know the externally available app data's ftp address. I simply put it in /home/icanis/ftp/opt on darni. And Thomas takes care of the ftp part.

Why am I not in the database? When I enter my name in the distances tab, which the plot is based on, it works.

You mention that you want a fresh and clear picture. If you tell me how exactly you want to change the plot, I can generate it and send it to you. Do you need a higher resolution? See the figure attached to this e-mail, i.e. the plot used in the pdf, as a reference.

Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: Christian Zimmermann <zimmermann@stlouisfed.org> Sent: Donnerstag, 26. November 2020 05:01 To: D�ben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Thomas Krichel <krichel@openlib.org>; collec-run@lists.openlib.org Subject: RE: [CollEc] New CollEc

Yes, I mean the app data.

Blog post is almost ready. I cannot find a way to share it with you without publishing it... but I think I copied everyting over and only changed links a little and what you suggested below.

I wanted ot make a fresh and clear picture of the graph you included in your pdf. Unfortunately, I cannot recreate it, as you do not seem to be in the database...

Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist

On Wed, 25 Nov 2020, D�ben, Christian wrote:

...

As far as I am concerned, the app is going to remain at app.collec.repec.org.

Do you mean the app's data that Thomas made available through the ftp server?

Could you change "Co-authored research has been on the rise over the past decades forming collaborations over vast geographic distances and fields of research" in the blog post's second paragraph to "Co-authored research has been on the rise over the past decades, forming collaborations over enormous geographic distances and many fields of research"?

Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: Christian Zimmermann <zimmermann@stlouisfed.org> Sent: Mittwoch, 25. November 2020 20:21 To: D�ben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Thomas Krichel <krichel@openlib.org>; collec-run@lists.openlib.org Subject: RE: [CollEc] New CollEc

I'll take the co-authors. app.collec.repec.org is not going to change, right?

Also, authors will start complaining that the data in the new CollEc does not match the data in their statistics. Where can I find the new data?

Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist

On Wed, 25 Nov 2020, D�ben, Christian wrote:

...

Thanks. I am glad you like it.

Which links you use for the author pages depends on you. Which tab would you like to use as a landing page? Each of the output-generating tabs (Distances, Closeness, Betweenness, Co-Authors, Shortest Paths) has an entry point.

Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: Christian Zimmermann <zimmermann@stlouisfed.org> Sent: Mittwoch, 25. November 2020 20:09 To: D�ben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Thomas Krichel <krichel@openlib.org>; collec-run@lists.openlib.org Subject: RE: [CollEc] New CollEc

Finally getting to it...

Excellent post. I will try to massage it into the blog format. See me the proper links.

Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist

On Tue, 6 Oct 2020, Dï¿œben, Christian wrote:

...

As requested, I came up with a RePEc blog post. You can find a draft attached to this e-mail. How do you like it?

Do not post it yet. The hyperlinks already point to app.collec.repec.org which is yet to be connected to the app.

Christian Dï¿œben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of Dï¿œben, Christian Sent: Montag, 5. Oktober 2020 22:04 To: Thomas Krichel <krichel@openlib.org> Cc: collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

I reinstalled the containerized MariaDB which fixed the issue of the unexpectedly large ibdata1 file in it. A script is currently computing CollEc's results again and inserts the tables into the new containerized data base. The app does not work as intended until this process is complete, i.e. until around 11 pm CEST. This temporary "shutdown" of the app only happens because I set up a new containerized data base. The regular daily updating routine does not impede the app's availability.

The cleanup freed around 50 GB of disk space. The new CollEc's total size is around 28 GB.

Christian Dï¿œben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of Dï¿œben, Christian Sent: Montag, 5. Oktober 2020 19:04 To: Thomas Krichel <krichel@openlib.org> Cc: collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

I fixed the Per archive issue. The files are not missing on Aigtu. An intermediate step in the daily updating routine was incomplete. It works now. Sorry about that.

Christian Dï¿œben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of Dï¿œben, Christian Sent: Sonntag, 4. Oktober 2020 13:12 To: Thomas Krichel <krichel@openlib.org> Cc: collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

Do you mean the shortest paths between any two authors in the network? Users can look at their shortest paths in the Shortest Paths tab. That functionality computes them directly from the graph. The distances and the closeness measures also rely on these shortest paths. But at no point are they written to disk. They are only stored in memory. I did some extensive testing and writing these paths to disk is incredibly inefficient. There is a reason why the new CollEc's daily updating routine takes only a tiny fraction of the time and computational resources required by the old CollEc. It is the eradication of inefficiencies along multiple dimensions. Unless there is a very good reason why these paths should be written to disk, I do not see a point of doing so. Are they used anywhere outside CollEc?

app.collec.repec.org sounds good. Let me know when you changed it. I am then going to adjust the documentation on the entry points.

It is not a problem that Helos is an Ubuntu system. In fact, I prefer Ubuntu over Debian. Nonetheless, I am really not looking forward to that migration as I have to set up and test the entire framework again. There should be easier ways to directly migrating the CollEc system as a whole instead of installing it again piece by piece. But that is beyond my current technical skills.

The new CollEc requires rather little disk space. The only component that appears unexpectedly large is the containerized MariaDB. I am going to check tomorrow what might be wrong with it.

The authors' full names are derived from that local copy of the Per archive. The hundreds of missing files set the names of hundreds of people to NA. I am going to sync the archive to another server today and am going to check whether the problem remains. I need to make sure that the files are really missing on Aigtu and there is not a glitch somewhere in my code.

Have a nice day.

Christian Dï¿œben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Samstag, 3. Oktober 2020 17:11 To: Dï¿œben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Christian Zimmermann <zimmermann@stlouisfed.org>; collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

Dï¿œben, Christian writes

...

The files meant for the ftp server are in /home/icanis/ftp/opt. The rank variable in these files handles ties with the "min" option. Assume we have four authors with closeness values of 0.5, 0.3, 0.3 and 0.2 respectively. With the "min" option their ranks are 1, 2, 2 and 4.

Seems fine, but where is the path data?

CollEc generates path data. As long as your system does does not produce path data, it is not a replacement for the existing CollEc.

...

You mentioned that your code fixes the rsync permission problem over time. What time frame did you have in mind? It does not work yet.

Then it does not work. The issue had nothing to do with permissions, but with the fact that remi at aigtu was completely broken. That is fixed.

...

Do I need to add anything to the shell script?

I doubt it. If the data does not update the problem is that aigtu does not have an up-to-date copy of RePEc:per.

...

Before Christian Zimmermann links any IDEAS content to the web application's entry points, we should discuss the URL.

Collec2 is no good. What about app.collec? It will become collec.repec.org when it can replace the functionality. I'm much looking forward to that as the current calculations block helos for any other talks.

We need to move this to helos. There you will have more computing power.

...

Could you add a placeholder HTML page at graphec.repec.org? Something like "GraphEc is currently publicly unavailable. Contact <a href="http://www.christian-dueben.com">Christian Dï¿œben</a> regarding the application's public release.". The new CollEc already links to GraphEc and I think that a placeholder page with a short informative text would look nicer than the Apache2 Debian default page.

I suggest that if graphec is not available, you should not link to it.

When helos was installed, debian had no controller for its NIC. Thus it's an ubuntu machine. Will that be a problem? I could reinstall helos, but I don't have disk space to at least temporarily store the data it houses now.

Generally, my machines are running full. We are short of disk space. It has been difficult to get further resources because of the crisis.

--

Cheers,

Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run

_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run

_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run

Almost through a big RePEc backlog, will get back to you soon. Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist On Fri, 30 Oct 2020, D�ben, Christian wrote:

Are there any updates regarding the new CollEc? Do you have any comments on the blog post draft I sent a few weeks ago?

Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: D�ben, Christian Sent: Dienstag, 6. Oktober 2020 19:54 To: Thomas Krichel <krichel@openlib.org>; Christian Zimmermann <zimmermann@stlouisfed.org> Cc: collec-run@lists.openlib.org Subject: RE: [CollEc] New CollEc

As requested, I came up with a RePEc blog post. You can find a draft attached to this e-mail. How do you like it?

Do not post it yet. The hyperlinks already point to app.collec.repec.org which is yet to be connected to the app.

Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of D�ben, Christian Sent: Montag, 5. Oktober 2020 22:04 To: Thomas Krichel <krichel@openlib.org> Cc: collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

I reinstalled the containerized MariaDB which fixed the issue of the unexpectedly large ibdata1 file in it. A script is currently computing CollEc's results again and inserts the tables into the new containerized data base. The app does not work as intended until this process is complete, i.e. until around 11 pm CEST. This temporary "shutdown" of the app only happens because I set up a new containerized data base. The regular daily updating routine does not impede the app's availability.

The cleanup freed around 50 GB of disk space. The new CollEc's total size is around 28 GB.

Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of D�ben, Christian Sent: Montag, 5. Oktober 2020 19:04 To: Thomas Krichel <krichel@openlib.org> Cc: collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

I fixed the Per archive issue. The files are not missing on Aigtu. An intermediate step in the daily updating routine was incomplete. It works now. Sorry about that.

Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: CollEc-run <collec-run-bounces@lists.openlib.org> On Behalf Of D�ben, Christian Sent: Sonntag, 4. Oktober 2020 13:12 To: Thomas Krichel <krichel@openlib.org> Cc: collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

Do you mean the shortest paths between any two authors in the network? Users can look at their shortest paths in the Shortest Paths tab. That functionality computes them directly from the graph. The distances and the closeness measures also rely on these shortest paths. But at no point are they written to disk. They are only stored in memory. I did some extensive testing and writing these paths to disk is incredibly inefficient. There is a reason why the new CollEc's daily updating routine takes only a tiny fraction of the time and computational resources required by the old CollEc. It is the eradication of inefficiencies along multiple dimensions. Unless there is a very good reason why these paths should be written to disk, I do not see a point of doing so. Are they used anywhere outside CollEc?

app.collec.repec.org sounds good. Let me know when you changed it. I am then going to adjust the documentation on the entry points.

It is not a problem that Helos is an Ubuntu system. In fact, I prefer Ubuntu over Debian. Nonetheless, I am really not looking forward to that migration as I have to set up and test the entire framework again. There should be easier ways to directly migrating the CollEc system as a whole instead of installing it again piece by piece. But that is beyond my current technical skills.

The new CollEc requires rather little disk space. The only component that appears unexpectedly large is the containerized MariaDB. I am going to check tomorrow what might be wrong with it.

The authors' full names are derived from that local copy of the Per archive. The hundreds of missing files set the names of hundreds of people to NA. I am going to sync the archive to another server today and am going to check whether the problem remains. I need to make sure that the files are really missing on Aigtu and there is not a glitch somewhere in my code.

Have a nice day.

Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Samstag, 3. Oktober 2020 17:11 To: D�ben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Christian Zimmermann <zimmermann@stlouisfed.org>; collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

D�ben, Christian writes

...

The files meant for the ftp server are in /home/icanis/ftp/opt. The rank variable in these files handles ties with the "min" option. Assume we have four authors with closeness values of 0.5, 0.3, 0.3 and 0.2 respectively. With the "min" option their ranks are 1, 2, 2 and 4.

Seems fine, but where is the path data?

CollEc generates path data. As long as your system does does not produce path data, it is not a replacement for the existing CollEc.

...

You mentioned that your code fixes the rsync permission problem over time. What time frame did you have in mind? It does not work yet.

Then it does not work. The issue had nothing to do with permissions, but with the fact that remi at aigtu was completely broken. That is fixed.

...

Do I need to add anything to the shell script?

I doubt it. If the data does not update the problem is that aigtu does not have an up-to-date copy of RePEc:per.

...

Before Christian Zimmermann links any IDEAS content to the web application's entry points, we should discuss the URL.

Collec2 is no good. What about app.collec? It will become collec.repec.org when it can replace the functionality. I'm much looking forward to that as the current calculations block helos for any other talks.

We need to move this to helos. There you will have more computing power.

...

Could you add a placeholder HTML page at graphec.repec.org? Something like "GraphEc is currently publicly unavailable. Contact <a href="http://www.christian-dueben.com">Christian D�ben</a> regarding the application's public release.". The new CollEc already links to GraphEc and I think that a placeholder page with a short informative text would look nicer than the Apache2 Debian default page.

I suggest that if graphec is not available, you should not link to it.

When helos was installed, debian had no controller for its NIC. Thus it's an ubuntu machine. Will that be a problem? I could reinstall helos, but I don't have disk space to at least temporarily store the data it houses now.

Generally, my machines are running full. We are short of disk space. It has been difficult to get further resources because of the crisis.

--

Cheers,

Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run

_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run

_______________________________________________ CollEc-run mailing list CollEc-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run

Düben, Christian writes

Do you mean the shortest paths between any two authors in the network? Users can look at their shortest paths in the Shortest Paths tab. That functionality computes them directly from the graph. The distances and the closeness measures also rely on these shortest paths. But at no point are they written to disk. They are only stored in memory. I did some extensive testing and writing these paths to disk is incredibly inefficient. There is a reason why the new CollEc's daily updating routine takes only a tiny fraction of the time and computational resources required by the old CollEc. It is the eradication of inefficiencies along multiple dimensions.

I'm not sure I understand, but it's clear that the approach I took recalculated the same paths over over again. I did not know better.

Unless there is a very good reason why these paths should be written to disk, I do not see a point of doing so. Are they used anywhere outside CollEc?

They have been exported to IZA. But I have not read Nikos complaining. He is on the list.

app.collec.repec.org sounds good. Let me know when you changed it. I am then going to adjust the documentation on the entry points.

It looks like this is done, with app pointing to darni.

It is not a problem that Helos is an Ubuntu system. In fact, I prefer Ubuntu over Debian. Nonetheless, I am really not looking forward to that migration as I have to set up and test the entire framework again. There should be easier ways to directly migrating the CollEc system as a whole instead of installing it again piece by piece. But that is beyond my current technical skills.

I can get migrations done quite easily with my ways of working. It involves writing all in the user space with minimum root involvement, and relying on o/s packages as much as possible. There is nothing we can do about the migration. Helos is sponsored to do CollEc. We can't run it on darni. On the upside, helos is more powerful than darni.

The new CollEc requires rather little disk space. The only component that appears unexpectedly large is the containerized MariaDB. I am going to check tomorrow what might be wrong with it.

helos has more space. It's used for the RePEc snapshot. That requires huge space. It is also used for backup, since aigtu is so full it can't handle it all. The plan is to migrate to migrate the snapshot to archec. But that may take a few years more. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

So, what links should I use for the author pages? Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist On Fri, 2 Oct 2020, D�ben, Christian wrote:

The files meant for the ftp server are in /home/icanis/ftp/opt. The rank variable in these files handles ties with the "min" option. Assume we have four authors with closeness values of 0.5, 0.3, 0.3 and 0.2 respectively. With the "min" option their ranks are 1, 2, 2 and 4.

You mentioned that your code fixes the rsync permission problem over time. What time frame did you have in mind? It does not work yet. Do I need to add anything to the shell script?

I fixed the issues on Christian Zimmermann's list, except for the blog post which still needs another paragraph or so. The text in the Introduction tab, the Co-Authors tab, the Documentation tab and the footer is now partly different. I also added an image to the documentation on the Co-Authors tab.

Before Christian Zimmermann links any IDEAS content to the web application's entry points, we should discuss the URL. The current test.collec.repec.org was suitable during testing. But now that the app is quasi officially released, I recommend changing the name to e.g. collec2.repec.org. Until we see how well the app performs when accessed by many users at the same time we should keep the old version of CollEc in service. And keeping the old one at collec.repec.org while running the new one at collec2.repec.org appears to be a good solution. What do you think?

Could you add a placeholder HTML page at graphec.repec.org? Something like "GraphEc is currently publicly unavailable. Contact <a href="http://www.christian-dueben.com">Christian D�ben</a> regarding the application's public release.". The new CollEc already links to GraphEc and I think that a placeholder page with a short informative text would look nicer than the Apache2 Debian default page.

Christian D�ben Research Associate Chair of Macroeconomics Hamburg University Von-Melle-Park 5, Room 3102 20146 Hamburg Germany +49 40 42838 1898 christian.dueben@uni-hamburg.de http://www.christian-dueben.com

-----Original Message----- From: Thomas Krichel <krichel@openlib.org> Sent: Freitag, 25. September 2020 09:24 To: D�ben, Christian <Christian.Dueben@uni-hamburg.de> Cc: Christian Zimmermann <zimmermann@stlouisfed.org>; collec-run@lists.openlib.org Subject: Re: [CollEc] New CollEc

D�ben, Christian writes

...

- Where can I put the files that Christian Zimmermann wants to access through ftp?

Just put them anywhere and let me know where they are, I will arrange for public rsync for you. Since, I presume, they are generated, they should go into a directory named 'opt' to avoid filling disk space at the backup servers.

...

Should I add them to /mnt/helos/home/icanis/opt/ftp?

No, that's an NFS mount of helos.

...

I am currently missing more than 400 xml files of what I assume to be recently added RePEc users. Any idea what is wrong? Do I need to add another --include argument?

adrepec@aigtu:~$ $HOME/perl/remi_round_the_clock export REDIFDIR=/home/adrepec/RePEc/zzz ; cd /home/adrepec/perl; /home/adrepec/perl/remi2.pl -conf remi.local.conf sh: 1: /home/adrepec/perl/remi2.pl: Permission denied

oh that hurts!

adrepec@aigtu:~$ cd perl adrepec@aigtu:~/perl$ chmod 755 remi2.pl adrepec@aigtu:~/perl$ $HOME/perl/remi_round_the_clock

Should fix that one over time.

--

Cheers,

Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel