When trying to get the group catalog from snapshot 67 (z = 0.5) using wget, I'm getting a lot of "bad file descriptor" errors, and some "unknown error" failures as well. This has caused me to need to get the files each separately, which is nearly impossible since there are 600. And I did try to do this for the missing files, but when I try to load it, it appears some files have been truncated. Is there an issue with the server?
Thanks.
Dylan Nelson
24 Oct '19
Hi,
I'm not sure which simulation? Can you provide some specific links which fail? Thanks!
Well it's happened with TNG300-1 for snapshot 67, as well as TNG100-1 for snapshot 0. As for which links specifically, it's hit-and-miss: when I wget them again, ones that failed previously will come down the next time. But there's only so many times one can redownload an entire group catalog to get all the files...
I'm also wondering if it's specific to my account?? Since it's happening for different catalogs in different simulations, and it's not happened before.
Sophia Nasr
24 Oct '19
Just as an update, I'm now downloading snapshot 84 group catalog from TNG300-1, and getting a lot of "bad file descriptor" errors as well. As I said, I was able to download the specific ones i couldn't get via wget by clicking the links themselves (sometimes multiple clicks were required as I got "access denied" messages), but this is not sustainable in the long run.
Dylan Nelson
24 Oct '19
Hi Sophia,
Not quite sure, if you can change the -nv to -v (to turn on verbose), and then paste an error message you get, that would be helpful.
If there's something else you'd like me to post, please let me know.
Dylan Nelson
24 Oct '19
Yes too bad, seems to be a problem with the "uswest" server. I've disabled this for now until we can get it improved, you should see all requests going to the "eu" server.
If you hit any more errors here, please let me know.
Sophia Nasr
24 Oct '19
It seems to have solved the problem so far, I'm not getting any errors! Thank you so much, Dylan! I'll net you know if there are other problems.
Hello,
When trying to get the group catalog from snapshot 67 (z = 0.5) using wget, I'm getting a lot of "bad file descriptor" errors, and some "unknown error" failures as well. This has caused me to need to get the files each separately, which is nearly impossible since there are 600. And I did try to do this for the missing files, but when I try to load it, it appears some files have been truncated. Is there an issue with the server?
Thanks.
Hi,
I'm not sure which simulation? Can you provide some specific links which fail? Thanks!
Well it's happened with TNG300-1 for snapshot 67, as well as TNG100-1 for snapshot 0. As for which links specifically, it's hit-and-miss: when I wget them again, ones that failed previously will come down the next time. But there's only so many times one can redownload an entire group catalog to get all the files...
Here's the wget for TNG300-1: wget -nd -nc -nv -e robots=off -l 1 -r -A hdf5 --content-disposition --header="API-Key: _" "http://www.tng-project.org/api/TNG300-1/files/groupcat-67/?format=api"
and for TNG100-1: wget -nd -nc -nv -e robots=off -l 1 -r -A hdf5 --content-disposition --header="API-Key: __" "http://www.tng-project.org/api/TNG100-1/files/groupcat-0/?format=api"
I'm also wondering if it's specific to my account?? Since it's happening for different catalogs in different simulations, and it's not happened before.
Just as an update, I'm now downloading snapshot 84 group catalog from TNG300-1, and getting a lot of "bad file descriptor" errors as well. As I said, I was able to download the specific ones i couldn't get via wget by clicking the links themselves (sometimes multiple clicks were required as I got "access denied" messages), but this is not sustainable in the long run.
Hi Sophia,
Not quite sure, if you can change the
-nvto-v(to turn on verbose), and then paste an error message you get, that would be helpful.Okay, here's one example:
--2019-10-24 12:41:04-- https://www.tng-project.org/api/TNG300-1/files/groupcat-84.6.hdf5
Connecting to www.tng-project.org (www.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://data-uswest.tng-project.org/file/groupcat/L205n2500TNG/84/6/?token=______ [following]
--2019-10-24 12:41:05-- https://data-uswest.tng-project.org/file/groupcat/L205n2500TNG/84/6/?token=_______
Connecting to data-uswest.tng-project.org (data-uswest.tng-project.org)|132.249.245.246|:443... failed: Bad file descriptor.
URL transformed to HTTPS due to an HSTS policy
If there's something else you'd like me to post, please let me know.
Yes too bad, seems to be a problem with the "uswest" server. I've disabled this for now until we can get it improved, you should see all requests going to the "eu" server.
If you hit any more errors here, please let me know.
It seems to have solved the problem so far, I'm not getting any errors! Thank you so much, Dylan! I'll net you know if there are other problems.