EOL Data services

#75

No worries. Now I’ve got the API sorted a bit better we’re seeing various bits and pieces of text, rather than an ominous silence.

#76

I work with @charvolant and it appears we sometime get blocked due to exceeding some maximum rate limit (e.g. with https://eol.org/api/pages/1.0 or https://eol.org/api/search/1.0 request). I’m happy to add a rate limiter to our code and would like to know what the threshold is before we’re blocked (requests per second) so I can prevent us hitting that limit.

FYI, I noticed the “Data services” page makes mention of:

we ask that you observe query rate limits

but I couldn’t find any mention of what the actual limits value was, on any of the pages linked off that page (including the various terms of use pages).

#77

The rate is currently 1rps, which should be mentioned in the robots.txt file. …We may increase that in the months to come, so it could be worth re-checking. Thanks!

#78

thanks for your patience, @charvolant- filter_by_string should return results now.

#79

Sorry for the delay, @divinglog! We’ve got the FishBase resource downloaded; those broken images should be working now.

Jen

#80

Thank you @hammockj, seems to be working fine!

#81

@hammockj @jrice Hey, just checking in on sounds in the pages API?

#82

Thanks for checking, @Natasha! Alas, we’re still crawling down the to do list. I’m hoping for February, now…

#83

@hammockj Okay! Thank you for the update!

#84

For a number of years I have been maintaining an internal catalog for the Colorado State University Herbarium. The base taxonomy used by the Herbarium is ITIS. As you know, terminology changes over time and taxonomies shift so we map the name given on the specimen, which may have been collected in 1890, to the modern terminology.

EOL provides a valuable service through “page” or “taxonConcept”. This is really a numeric identifier for something that is normally referenced by name. One of the interesting features in the old user interface was looking at how the taxon was classified in different taxonomies (and they vary substantially).

When a specimen is added to the Herbarium we automatically check with ITIS to find the preferred name (as well as common names, synonyms…). The preferred name is then used to search EOL to find the taxonConcept. This gives an invariant ID and and also allows us to provide the user a link to EOL page for the species. In mapping between ITIS and EOL I have tried to be careful that the EOL page actually corresponds to the ITIS entry.

This does not seem possible in the re-implemented EOL APIs. The taxonomic name providers have not only changed in sources, but in character. The new providers are species lists while the old providers were taxonomy maintainers. Unless I have some way to map from existing taxonomy (ITIS) to the EOL taxonConcept, I will have to drop EOL references entirely. That is a shame because EOL is a fine resource and maintaining links between the various taxonomic sources is valuable.


New ‘nameAccordingTo’ from page 1127944 (Vanilla Pompona)
EOL Dynamic Hierarchy
South Atlantic Species List
USDA Plants data
North Atlantic Species List
North Pacific Species List
Sweden Species List
Venezuela Species List
Nicaragua Species List
Panama Species List
Peru Species List
Mexico Species List

old nameAccordingTo providers (on my development system):

AntWeb (Ant Species)
WORMS Species Information (Marine Species)
FishBase (Fish Species)
IUCN Red List (Species Assessed for Global Conservation)
Index Fungorum
Cephaloleia LifeDesk resource
The Reptile Database
GBIF Nub Taxonomy
Avibase - IOC World Bird Names (2011)
Integrated Taxonomic Information System (ITIS)
NCBI Taxonomy
Species 2000 & ITIS Catalogue of Life: April 2013

#85

Hi Colin,

The situation you describe is temporary. We have not yet loaded the major classification providers onto the new platform but will do so over the next few weeks. Please bear with us as our new names infrastructure takes shape. Once we have ITIS, Catalogue of Life, NCBI, etc. loaded, the system should function as it did previously. In addition, we expect improved data mappings, i.e., less homonym confusion and fewer doubtful names in the hierarchy.

#86

It is good to hear that the old name providers will be restored.

It seems like eol has been going through quite the transformation lately. A quick look at git doesn’t indicate to me exactly which areas are changing. The underlying rails code looks like it has been stable for a while. The UI has changed and this thread indicates the API is being re-written.

What are the main areas of change and why?

Colin

#87

hoping the CORS issue is resolved.

3 Likes
#88

Sorry for the delay! Our sysadmin is still looking into the CORS problem.

Jen

2 Likes
#89

@hammockj thanks for explain me that problem :+1:

2 Likes
#90

This is a good thing [quote=“hammockj, post:56, topic:236”]

This is a good thing [quote=“hammockj, post:56, topic:236”]
https://eol.org/api/pages/1.0.json?id=46577088&images_per_page=1&images_page=1
[/quote]

2 Likes
#91

Ah, that looks like success. Still an excessively long header of canonical names, though. We’re hoping to get that trimmed down by default to only sources we consider “classification providers” so it won’t be so long. But you got what you needed?

2 Likes
#92

Any idea when ITIS taxonomic data will be added back into eol?

Colin

#93

I’m not sure we can get it up by the end of this month, but an ETA in the first couple of weeks of April is realistic.

#94

@hammockj Hey, checking in on the status of sounds in the Pages API again, is there any ETA? We’re currently using EOL as a sort of media service to retrieve images and audio for users while they create educational content. I’m wondering if we should start looking into some additional alternatives if the restoration of sounds is going to take awhile? Thanks