EOL Data services

**hammockj** · December 6, 2018, 3:56pm

Thanks for reporting, @divinglog. It looks like the media download for that partner didn’t complete. We should be able to retrieve those; stand by.

**charvolant** · December 10, 2018, 3:48am

Hi,

I’ve been using the EoL API for summary data and I’ve been having a few problems.

Firstly, the filter_by_string option on /search seems to generate a 500 error. For example https://eol.org/api/search/1.0.json?q=Acacia+dealbata&filter_by_string=Fabaceae

Secondly, I don’t seem to be able to get any English overview or summary text entries. All the summary texts that I can find seem to be from the Italian or Spanish wikipedias. For example, https://eol.org/api/pages/1.0/122672.json?texts_per_page=75&subjects=overview&vetted=0&language=en&details=true

Doug

**hammockj** · December 10, 2018, 3:41pm

Thanks for reporting, @charvolant ! I’m sorry about the gap in text object service. We hit a delay in migrating in our largest sources of text. We’re working on it.

Paging @jrice to investigate the /search string filter issue…

Jen

**charvolant** · December 10, 2018, 11:35pm

No worries. Now I’ve got the API sorted a bit better we’re seeing various bits and pieces of text, rather than an ominous silence.

**nickdos** · December 11, 2018, 3:17am

I work with @charvolant and it appears we sometime get blocked due to exceeding some maximum rate limit (e.g. with https://eol.org/api/pages/1.0 or https://eol.org/api/search/1.0 request). I’m happy to add a rate limiter to our code and would like to know what the threshold is before we’re blocked (requests per second) so I can prevent us hitting that limit.

FYI, I noticed the “Data services” page makes mention of:

we ask that you observe query rate limits

but I couldn’t find any mention of what the actual limits value was, on any of the pages linked off that page (including the various terms of use pages).

**jrice** · December 11, 2018, 4:06pm

The rate is currently 1rps, which should be mentioned in the robots.txt file. …We may increase that in the months to come, so it could be worth re-checking. Thanks!

**hammockj** · December 18, 2018, 4:35pm

thanks for your patience, @charvolant- filter_by_string should return results now.

**hammockj** · January 8, 2019, 6:45pm

Sorry for the delay, @divinglog! We’ve got the FishBase resource downloaded; those broken images should be working now.

Jen

**divinglog** · January 8, 2019, 8:09pm

Thank you @hammockj, seems to be working fine!

**Natasha** · January 18, 2019, 7:06pm

@hammockj @jrice Hey, just checking in on sounds in the pages API?

**hammockj** · January 22, 2019, 7:17pm

Thanks for checking, @Natasha! Alas, we’re still crawling down the to do list. I’m hoping for February, now…

**Natasha** · January 22, 2019, 7:29pm

@hammockj Okay! Thank you for the update!

**Colin** · February 10, 2019, 10:46pm

For a number of years I have been maintaining an internal catalog for the Colorado State University Herbarium. The base taxonomy used by the Herbarium is ITIS. As you know, terminology changes over time and taxonomies shift so we map the name given on the specimen, which may have been collected in 1890, to the modern terminology.

EOL provides a valuable service through “page” or “taxonConcept”. This is really a numeric identifier for something that is normally referenced by name. One of the interesting features in the old user interface was looking at how the taxon was classified in different taxonomies (and they vary substantially).

When a specimen is added to the Herbarium we automatically check with ITIS to find the preferred name (as well as common names, synonyms…). The preferred name is then used to search EOL to find the taxonConcept. This gives an invariant ID and and also allows us to provide the user a link to EOL page for the species. In mapping between ITIS and EOL I have tried to be careful that the EOL page actually corresponds to the ITIS entry.

This does not seem possible in the re-implemented EOL APIs. The taxonomic name providers have not only changed in sources, but in character. The new providers are species lists while the old providers were taxonomy maintainers. Unless I have some way to map from existing taxonomy (ITIS) to the EOL taxonConcept, I will have to drop EOL references entirely. That is a shame because EOL is a fine resource and maintaining links between the various taxonomic sources is valuable.

New ‘nameAccordingTo’ from page 1127944 (Vanilla Pompona)
EOL Dynamic Hierarchy
South Atlantic Species List
USDA Plants data
North Atlantic Species List
North Pacific Species List
Sweden Species List
Venezuela Species List
Nicaragua Species List
Panama Species List
Peru Species List
Mexico Species List
…

old nameAccordingTo providers (on my development system):

AntWeb (Ant Species)
WORMS Species Information (Marine Species)
FishBase (Fish Species)
IUCN Red List (Species Assessed for Global Conservation)
Index Fungorum
Cephaloleia LifeDesk resource
The Reptile Database
GBIF Nub Taxonomy
Avibase - IOC World Bird Names (2011)
Integrated Taxonomic Information System (ITIS)
NCBI Taxonomy
Species 2000 & ITIS Catalogue of Life: April 2013
…

**schulzk** · February 12, 2019, 4:15pm

Hi Colin,

The situation you describe is temporary. We have not yet loaded the major classification providers onto the new platform but will do so over the next few weeks. Please bear with us as our new names infrastructure takes shape. Once we have ITIS, Catalogue of Life, NCBI, etc. loaded, the system should function as it did previously. In addition, we expect improved data mappings, i.e., less homonym confusion and fewer doubtful names in the hierarchy.

**Colin** · February 12, 2019, 4:57pm

It is good to hear that the old name providers will be restored.

It seems like eol has been going through quite the transformation lately. A quick look at git doesn’t indicate to me exactly which areas are changing. The underlying rails code looks like it has been stable for a while. The UI has changed and this thread indicates the API is being re-written.

What are the main areas of change and why?

Colin

**Tao** · February 13, 2019, 5:25pm

hoping the CORS issue is resolved.

**hammockj** · February 13, 2019, 5:27pm

Sorry for the delay! Our sysadmin is still looking into the CORS problem.

Jen

**Tao** · February 19, 2019, 7:06pm

@hammockj thanks for explain me that problem

**Tao** · February 21, 2019, 9:14am

This is a good thing [quote=“hammockj, post:56, topic:236”]

This is a good thing [quote=“hammockj, post:56, topic:236”]
https://eol.org/api/pages/1.0.json?id=46577088&images_per_page=1&images_page=1
[/quote]

**hammockj** · February 20, 2019, 5:02pm

Ah, that looks like success. Still an excessively long header of canonical names, though. We’re hoping to get that trimmed down by default to only sources we consider “classification providers” so it won’t be so long. But you got what you needed?