Illustris Web API - performance in accessing a single field per subhalo
Connor Bottrell
20 May '15
Is there a fast way to get a single field about each subhalo for a long list of subhalos?
For example, when I do:
import requests,string
halfmassrad_stars = {}
for subhalo in subhalo_list:
halfmassrad_stars[str(subhalo)] = get(urlbase_for_Illustris1_135s + str(subhalo) + '/')['halfmassrad_stars']
This takes a considerable amount of time since it is parsing all of the field info for each subhalo. Or is it because of the url grabs?
Connor
Dylan Nelson
20 May '15
Hi Connor,
As you suspect, the time here (probably 0.1-0.5 sec or so per subhalo) is mainly just the roundtrip latency for the page retrieval, and the time for the DB to grab the values for the particular subhalo. It wouldn't be any faster if there was some shortcut to get just a single value (there isn't).
If you want to do this for many subhalos (>1000) or many times, you should just download the group catalog (this is only 4.5 GB for snapshot 135). Then these same values will be found in e.g. SubhaloHalfmassRadType[:,4].
Is there a fast way to get a single field about each subhalo for a long list of subhalos?
For example, when I do:
This takes a considerable amount of time since it is parsing all of the field info for each subhalo. Or is it because of the url grabs?
Connor
Hi Connor,
As you suspect, the time here (probably 0.1-0.5 sec or so per subhalo) is mainly just the roundtrip latency for the page retrieval, and the time for the DB to grab the values for the particular subhalo. It wouldn't be any faster if there was some shortcut to get just a single value (there isn't).
If you want to do this for many subhalos (>1000) or many times, you should just download the group catalog (this is only 4.5 GB for snapshot 135). Then these same values will be found in e.g.
SubhaloHalfmassRadType[:,4].Alright. Thanks for your response, Dylan.