When I downloaded SubLink of Illustris-3 (following the example script), it did not show any kinda error.
So I can not understand what is the problem.
How can I fix it? Any comment will be helpful.
I realized that the above mentioned method (downloading the chunk files one by one) also does NOT work.
Some files are downloaded completely, but some others are failed.
And they show also 'ERROR 403: Forbidden',
I attach all of massages from above method as below.
Hi all,
I tried to get the SubLink merger tree of TNG50-1, but encountered the below error:
It says 'ERROR 403: Forbidden'.
FYI, the command was the same in the download page.
wget -nd -nc -nv -e robots=off -l 1 -r -A hdf5 --content-disposition --header="API-Key: (I put my API-Key here)" "http://www.tng-project.org/api/TNG50-1/files/sublink/?format=api"
When I downloaded SubLink of Illustris-3 (following the example script), it did not show any kinda error.
So I can not understand what is the problem.
How can I fix it? Any comment will be helpful.
Cheers,
Gain
Now I'm downloading the chunk files one by one, using these commands :
wget -nc --content-disposition --header="API-Key: (my key)" "http://www.tng-project.org/api/TNG50-1/files/sublink.0.hdf5"
wget -nc --content-disposition --header="API-Key: (my key)" "http://www.tng-project.org/api/TNG50-1/files/sublink.1.hdf5"
wget -nc --content-disposition --header="API-Key: (my key)" "http://www.tng-project.org/api/TNG50-1/files/sublink.2.hdf5"
...
I think it works, but the speed is quite slow!
I realized that the above mentioned method (downloading the chunk files one by one) also does NOT work.
Some files are downloaded completely, but some others are failed.
And they show also 'ERROR 403: Forbidden',
I attach all of massages from above method as below.
URL transformed to HTTPS due to an HSTS policy
--2022-07-20 16:24:59-- https://www.tng-project.org/api/TNG50-1/files/sublink.0.hdf5
Resolving www.tng-project.org (www.tng-project.org)... 130.183.17.94
Connecting to www.tng-project.org (www.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://data-eu.tng-project.org/file/sublink/L35n2160TNG/0/?token=41f1df26a4e5f54506b1 [following]
--2022-07-20 16:25:00-- https://data-eu.tng-project.org/file/sublink/L35n2160TNG/0/?token=41f1df26a4e5f54506b1
Resolving data-eu.tng-project.org (data-eu.tng-project.org)... 130.183.17.94
Connecting to data-eu.tng-project.org (data-eu.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 219851497910 (205G) [application/octet-stream]
Saving to: 'tree_extended.0.hdf5'
tree_extended.0.hdf5 22%[===================> tree_extended.0.hdf5 22%[===================> tree_extended.0.hdf5 36%[=====================> ] 75.59G 3.06MB/s in 6h 39m
2022-07-20 23:04:31 (3.23 MB/s) - Read error at byte 81166889173/219851497910 (Error in the pull function.). Retrying.
--2022-07-20 23:04:32-- (try: 2) https://data-eu.tng-project.org/file/sublink/L35n2160TNG/0/?token=41f1df26a4e5f54506b1
Connecting to data-eu.tng-project.org (data-eu.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2022-07-20 23:04:33 ERROR 403: Forbidden.
URL transformed to HTTPS due to an HSTS policy
--2022-07-20 16:36:42-- https://www.tng-project.org/api/TNG50-1/files/sublink.1.hdf5
Resolving www.tng-project.org (www.tng-project.org)... 130.183.17.94
Connecting to www.tng-project.org (www.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://data-eu.tng-project.org/file/sublink/L35n2160TNG/1/?token=d33cec18f9e5dae2686d [following]
--2022-07-20 16:36:43-- https://data-eu.tng-project.org/file/sublink/L35n2160TNG/1/?token=d33cec18f9e5dae2686d
Resolving data-eu.tng-project.org (data-eu.tng-project.org)... 130.183.17.94
Connecting to data-eu.tng-project.org (data-eu.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 43931159238 (41G) [application/octet-stream]
Saving to: 'tree_extended.1.hdf5'
tree_extended.1.hdf5 100%[========================================================================================>] 40.91G 3.51MB/s in 3h 39m
2022-07-20 20:16:39 (3.18 MB/s) - 'tree_extended.1.hdf5' saved [43931159238/43931159238]
URL transformed to HTTPS due to an HSTS policy
--2022-07-20 20:16:39-- https://www.tng-project.org/api/TNG50-1/files/sublink.2.hdf5
Resolving www.tng-project.org (www.tng-project.org)... 130.183.17.94
Connecting to www.tng-project.org (www.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://data-eu.tng-project.org/file/sublink/L35n2160TNG/2/?token=5d32c9f1b87ed7a12e21 [following]
--2022-07-20 20:16:41-- https://data-eu.tng-project.org/file/sublink/L35n2160TNG/2/?token=5d32c9f1b87ed7a12e21
Resolving data-eu.tng-project.org (data-eu.tng-project.org)... 130.183.17.94
Connecting to data-eu.tng-project.org (data-eu.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 43953269106 (41G) [application/octet-stream]
Saving to: 'tree_extended.2.hdf5'
tree_extended.2.hdf5 10%[========> tree_extended.2.hdf5 10%[========> tree_extended.2.hdf5 100%[===========================================================>] 40.93G 4.33MB/s in 3h 10m
2022-07-20 23:27:04 (3.67 MB/s) - 'tree_extended.2.hdf5' saved [43953269106/43953269106]
URL transformed to HTTPS due to an HSTS policy
--2022-07-20 23:27:04-- https://www.tng-project.org/api/TNG50-1/files/sublink.3.hdf5
Resolving www.tng-project.org (www.tng-project.org)... 130.183.17.94
Connecting to www.tng-project.org (www.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://data-eu.tng-project.org/file/sublink/L35n2160TNG/3/?token=ec8a3e0e45ffd397d767 [following]
--2022-07-20 23:27:05-- https://data-eu.tng-project.org/file/sublink/L35n2160TNG/3/?token=ec8a3e0e45ffd397d767
Resolving data-eu.tng-project.org (data-eu.tng-project.org)... 130.183.17.94
Connecting to data-eu.tng-project.org (data-eu.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 43970304710 (41G) [application/octet-stream]
Saving to: 'tree_extended.3.hdf5'
tree_extended.3.hdf5 11%[======> ] 4.85G 4.98MB/s in 19m 28s
2022-07-20 23:46:34 (4.25 MB/s) - Read error at byte 5210636288/43970304710 (Error decoding the received TLS packet.). Retrying.
--2022-07-20 23:46:35-- (try: 2) https://data-eu.tng-project.org/file/sublink/L35n2160TNG/3/?token=ec8a3e0e45ffd397d767
Connecting to data-eu.tng-project.org (data-eu.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2022-07-20 23:46:37 ERROR 403: Forbidden.
URL transformed to HTTPS due to an HSTS policy
--2022-07-20 23:46:37-- https://www.tng-project.org/api/TNG50-1/files/sublink.4.hdf5
Resolving www.tng-project.org (www.tng-project.org)... 130.183.17.94
Connecting to www.tng-project.org (www.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://data-eu.tng-project.org/file/sublink/L35n2160TNG/4/?token=ed7385c8365d5a9ec480 [following]
--2022-07-20 23:46:38-- https://data-eu.tng-project.org/file/sublink/L35n2160TNG/4/?token=ed7385c8365d5a9ec480
Resolving data-eu.tng-project.org (data-eu.tng-project.org)... 130.183.17.94
Connecting to data-eu.tng-project.org (data-eu.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 43969921424 (41G) [application/octet-stream]
Saving to: 'tree_extended.4.hdf5'
tree_extended.4.hdf5 100%[===========================================================>] 40.95G 6.35MB/s in 2h 42m
2022-07-21 02:29:11 (4.30 MB/s) - 'tree_extended.4.hdf5' saved [43969921424/43969921424]
URL transformed to HTTPS due to an HSTS policy
--2022-07-21 02:29:11-- https://www.tng-project.org/api/TNG50-1/files/sublink.5.hdf5
Resolving www.tng-project.org (www.tng-project.org)... 130.183.17.94
Connecting to www.tng-project.org (www.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://data-eu.tng-project.org/file/sublink/L35n2160TNG/5/?token=f06de10ba625e8532e8f [following]
--2022-07-21 02:29:13-- https://data-eu.tng-project.org/file/sublink/L35n2160TNG/5/?token=f06de10ba625e8532e8f
Resolving data-eu.tng-project.org (data-eu.tng-project.org)... 130.183.17.94
Connecting to data-eu.tng-project.org (data-eu.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 43970245356 (41G) [application/octet-stream]
Saving to: 'tree_extended.5.hdf5'
tree_extended.5.hdf5 0%[ ] 137.28M 5.01MB/s in 42s
2022-07-21 02:29:57 (3.29 MB/s) - Read error at byte 143952647/43970245356 (Error decoding the received TLS packet.). Retrying.
--2022-07-21 02:29:58-- (try: 2) https://data-eu.tng-project.org/file/sublink/L35n2160TNG/5/?token=f06de10ba625e8532e8f
Connecting to data-eu.tng-project.org (data-eu.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2022-07-21 02:29:59 ERROR 403: Forbidden.
URL transformed to HTTPS due to an HSTS policy
--2022-07-21 02:29:59-- https://www.tng-project.org/api/TNG50-1/files/sublink.6.hdf5
Resolving www.tng-project.org (www.tng-project.org)... 130.183.17.94
Connecting to www.tng-project.org (www.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://data-eu.tng-project.org/file/sublink/L35n2160TNG/6/?token=21e1fc4ec063971b1408 [following]
--2022-07-21 02:30:00-- https://data-eu.tng-project.org/file/sublink/L35n2160TNG/6/?token=21e1fc4ec063971b1408
Resolving data-eu.tng-project.org (data-eu.tng-project.org)... 130.183.17.94
Connecting to data-eu.tng-project.org (data-eu.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 43970316782 (41G) [application/octet-stream]
Saving to: 'tree_extended.6.hdf5'
tree_extended.6.hdf5 100%[===========================================================>] 40.95G 3.75MB/s in 2h 43m
2022-07-21 05:13:06 (4.29 MB/s) - 'tree_extended.6.hdf5' saved [43970316782/43970316782]
URL transformed to HTTPS due to an HSTS policy
--2022-07-21 05:13:06-- https://www.tng-project.org/api/TNG50-1/files/sublink.7.hdf5
Resolving www.tng-project.org (www.tng-project.org)... 130.183.17.94
Connecting to www.tng-project.org (www.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://data-eu.tng-project.org/file/sublink/L35n2160TNG/7/?token=da96980a696020a57a90 [following]
--2022-07-21 05:13:08-- https://data-eu.tng-project.org/file/sublink/L35n2160TNG/7/?token=da96980a696020a57a90
Resolving data-eu.tng-project.org (data-eu.tng-project.org)... 130.183.17.94
Connecting to data-eu.tng-project.org (data-eu.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 43970284590 (41G) [application/octet-stream]
Saving to: 'tree_extended.7.hdf5'
tree_extended.7.hdf5 6%[===> ] 2.76G 5.30MB/s in 11m 30s
2022-07-21 05:24:40 (4.09 MB/s) - Read error at byte 2961597695/43970284590 (Error in the pull function.). Retrying.
--2022-07-21 05:24:41-- (try: 2) https://data-eu.tng-project.org/file/sublink/L35n2160TNG/7/?token=da96980a696020a57a90
Connecting to data-eu.tng-project.org (data-eu.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2022-07-21 05:24:42 ERROR 403: Forbidden.
URL transformed to HTTPS due to an HSTS policy
--2022-07-21 05:24:42-- https://www.tng-project.org/api/TNG50-1/files/sublink.8.hdf5
Resolving www.tng-project.org (www.tng-project.org)... 130.183.17.94
Connecting to www.tng-project.org (www.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://data-eu.tng-project.org/file/sublink/L35n2160TNG/8/?token=32b981a808d0c0c57d07 [following]
--2022-07-21 05:24:43-- https://data-eu.tng-project.org/file/sublink/L35n2160TNG/8/?token=32b981a808d0c0c57d07
Resolving data-eu.tng-project.org (data-eu.tng-project.org)... 130.183.17.94
Connecting to data-eu.tng-project.org (data-eu.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 43970304710 (41G) [application/octet-stream]
Saving to: 'tree_extended.8.hdf5'
tree_extended.8.hdf5 100%[===========================================================>] 40.95G 3.85MB/s in 2h 43m
2022-07-21 08:08:11 (4.28 MB/s) - 'tree_extended.8.hdf5' saved [43970304710/43970304710]
URL transformed to HTTPS due to an HSTS policy
--2022-07-21 08:08:11-- https://www.tng-project.org/api/TNG50-1/files/sublink.9.hdf5
Resolving www.tng-project.org (www.tng-project.org)... 130.183.17.94
Connecting to www.tng-project.org (www.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://data-eu.tng-project.org/file/sublink/L35n2160TNG/9/?token=918f56be8903b1351719 [following]
--2022-07-21 08:08:12-- https://data-eu.tng-project.org/file/sublink/L35n2160TNG/9/?token=918f56be8903b1351719
Resolving data-eu.tng-project.org (data-eu.tng-project.org)... 130.183.17.94
Connecting to data-eu.tng-project.org (data-eu.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 43970310746 (41G) [application/octet-stream]
Saving to: 'tree_extended.9.hdf5'
tree_extended.9.hdf5 18%[==========> ] 7.77G 2.97MB/s in 31m 51s
2022-07-21 08:40:04 (4.16 MB/s) - Read error at byte 8341669118/43970310746 (Error decoding the received TLS packet.). Retrying.
--2022-07-21 08:40:05-- (try: 2) https://data-eu.tng-project.org/file/sublink/L35n2160TNG/9/?token=918f56be8903b1351719
Connecting to data-eu.tng-project.org (data-eu.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2022-07-21 08:40:06 ERROR 403: Forbidden.
URL transformed to HTTPS due to an HSTS policy
--2022-07-21 08:40:06-- https://www.tng-project.org/api/TNG50-1/files/sublink.10.hdf5
Resolving www.tng-project.org (www.tng-project.org)... 130.183.17.94
Connecting to www.tng-project.org (www.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://data-eu.tng-project.org/file/sublink/L35n2160TNG/10/?token=e05e77223e96c3568d4d [following]
--2022-07-21 08:40:08-- https://data-eu.tng-project.org/file/sublink/L35n2160TNG/10/?token=e05e77223e96c3568d4d
Resolving data-eu.tng-project.org (data-eu.tng-project.org)... 130.183.17.94
Connecting to data-eu.tng-project.org (data-eu.tng-project.org)|130.183.17.94|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 43266537290 (40G) [application/octet-stream]
Saving to: 'tree_extended.10.hdf5'
tree_extended.10.hdf5 100%[========================================================================================>] 40.29G 5.39MB/s in 2h 22m
2022-07-21 11:02:31 (4.83 MB/s) - 'tree_extended.10.hdf5' saved [43266537290/43266537290]