EOL Data services


#21

Hi Jen,

Yes, the previously included query does show the issue.

Cheers,
Greg


#22

fixed- all licenses now means all licenses. Thanks for reporting!


#23

hoping the CORS issue is resolved. Could you try again?


#24

We allow users to choose default images to represent taxa and to choose those images from iNat observations, Flickr, EOL, and Wikimedia Commons. We don’t necessarily encourage one source over another, but iNat obs are the first choice presented if only because it prevents problems with third parties (like the one we’re discussing right now). Even so, iNat users have chosen EOL photos for many of our taxa because there are no available iNat observation photos to use, so it doesn’t really matter what we encourage or how people behave on iNat now, we still have a ton of taxa with broken images. If you folks could resurrect those data object IDs and get the data_objects endpoint to work with them again we could at least refresh the URLs for all those photos.

Is EOL in the loop for the selected iNat images also?

Not sure what this means.

So, what I would like to know is

  1. Are these old identifiers ever going to work with the data_objects API endpoint again? Sounds like the answer is “maybe”?
  2. Can EOL commit to stable identifiers for photos from now on? B/c if you cannot we need to stop allowing our users to choose EOL photos for iNat taxa b/c this problem will probably just keep happening.

#25

I see; that makes it clearer, thanks. If you’d like to put a bandaid on this and can send a list of the image file URLs, we can send you a mapping; it may take awhile and I expect it will cover most but not all of the images. Sources have been reharvested, and we have seen some media disappear at the source.

We could also try mapping back to the file URL from which we harvested them- many may have come originally from your other two sources.

Going forward, the need for another reharvest event like this one is unlikely with our new codebase. That being said, EOL is (still) not a media repository. I cannot offer the commitment you describe.


#26

@hammockj Thank you but the problem still remains.
Error message is still present but this morning there is additional message about CORB which wasn’t there before.

Access to XMLHttpRequest at ‘https://eol.org/api/pages/1.0.json?id=1745&details=true&images_per_page=30&common_names=false’ from origin ‘http://wiki.mydns.jp’ has been blocked by CORS policy: No ‘Access-Control-Allow-Origin’ header is present on the requested resource.
jquery-3.3.1.min.js:2 Cross-Origin Read Blocking (CORB) blocked cross-origin response https://eol.org/api/pages/1.0.json?id=1745&details=true&images_per_page=30&common_names=false with MIME type application/json. See https://www.chromestatus.com/feature/5629709824032768 for more details.


#27

I’m getting an unexpected result format from batch pages api calls. I believe the old format, at the top level, was an object mapping
{ <identifier>: <result>, … }
but I am now getting an array
[ <result>, … ]
Is this change intentional?


#28

For single-identifier pages api calls (non-batch), there is now a top level taxonConcept attribute which didn’t previously exist – it’s a bit confusing, since its value also contains a taxonConcepts array. Is the format expected to stay this way?

To be clear, I’m referring to the JSON api in these comments.


#29

@gregm, apologies, we have a slightly different data model now, requiring a slightly different construction for text objects, eg:

https://eol.org/api/data_objects_articles/1.0/12749603.json?taxonomy=false&cache_ttl=&language=en

can you manage?


#30

#31

@OLDY we’re finding CORS issues difficult to test, but have tried a couple of fixes per enable-cors.org. Could you report what you’re seeing now, please?


#32

Thank Jen,

Yes, this API change is easy for me adapt to, no problem on that front.

In my efforts to test this change though, I noticed that a number of other things changed yesterday afternoon (around the time you posted on this topic) that break other parts of the API. The most significant issue is affecting my use of the “pages” API. No mater what identifier I use (which I get from the search API) it always returns the same (wrong) results. For example if I search for a “Yellow Tang” and pick the search result with the identifier: 46577088 I get the same (or at least very similar) results as if I search for a “Boxfish” and pick the result with the identifier: 46570907.

In both cases I get a result with a scientific name of: “Metazoa” and (the same) images related to spiders and reptiles. Here is the pages command for the “Reticulated Boxfish” example:

https://eol.org/api/pages/1.0.json?batch=false&id=46570907&images_per_page=10&images_page=1&videos_per_page=0&videos_page=0&sounds_per_page=0&sounds_page=0&maps_per_page=0&maps_page=0&texts_per_page=10&texts_page=1&subjects=overview&licenses=all&details=true&common_names=true&synonyms=true&references=true&taxonomy=true&vetted=0&cache_ttl=&language=en

my previous post’s pages command was for the Yellow Tang example and it use to give correct results (I didn’t reproduce it here).

The second issue is yet another change to the JSON results structure that is returned. Before v3 the pages API returned a dictionary object at the top level that contained the “taxonconcepts” dictionary. Then when v3 was first releases the top level was a dictionary that had a dictionary “taxonconcept” (no s) that contained the v2 version of the dictionary. Now it appears that the “taxonconcept” dictionary contains a two element array with the second element being the original dictionary that contains the “taxonconcepts” dictionary.

This later change is fine, but it would be good to know if this will stay this way or if it will continue to evolve or if this is an error and it will revert to its previous structure.

The third issue is that the “text” data objects returned from the pages lookups that I’ve cited do not result in text results (they appears to be large arrays of dictionaries containing identical information related to classifications). If I use the object identifiers that I had from before this change (as with your example) then I get appropriate results. Of course, this issue may be related to the fact that the result of the pages API call itself is wrong.

Thanks for all your help getting this to work for everyone again.

Cheers,

Greg


#33

Hi @gregm,

I am still learning how to use APIs and everything for EOL, but for the first part of your problem I noticed that you are using “…/1.0.json?..” and that is why you always get “Metazoa” which has taxon id of 1. If you change it to “…/46577088.json?..” or “…/46570907.json?..” you will get the result for those taxon ids. I am not sure how accurate I am but I hope it helps and If I am wrong please let me know.

Thanks,
Navid


#34

Hi Navid,

Thanks … that is a very helpful. The API use to work (prior to yesterday) the way I had it (with 1.0.json and the id=XXXX), which may have been a coincidence, but you are right that as of now, removing the id=XXXX and putting the ID before the “.json” is working. This form of the API is certainly more consistent with some of the other API. I just reviewed the current EOL.org Classic API documentation and it also confirms your interpretation.

Thanks again, great catch.

Cheers,

Greg


#35

Thank goodness! @gregm are your 2nd and/or 3rd issues still issues or did they descend from the 1.0 as taxon ID business?


#36

Also wondering about the JSON result format change that @gregm and @mmvitale have mentioned. Will this new format be a permanent change or is this an incorrect result? Changing to this new format seems to make everything work again but I don’t want to push a fix for it if it’s changing again soon.


#37

Hi! It is fixed! No CORS error any more.
Thank you very much for working on this issue.


#38

It’s probably worth documenting the change @gregm and @navidz discussed as a difference from the v2 API on the ‘classic APIs’ page. I just bumped into that issue too.

edit: if that’s the new format, what’s the correct way to call pages with a batch of ids?


#39

Hi Jen,

The second issue (the change to the structure of the returned JSON) is still an “issue”. As I and other have mentioned, the change is okay, but it would be good to know if this is stable change as it is hard to code for all possible return structures. I’ve modified my app so it will work with all three of the structures that we’ve seen to date so if the results stay within those bounds I’m happy.

The third issue is not really a problem for me, it just represents a change from the v2 results of the original data_objects API. This call for example:

https://eol.org/api/data_objects_articles/1.0/12657377.json?taxonomy=false&cache_ttl=&language=en

results in an additional taxonConcepts dictionary that (I think) the “taxonomy=false” should eliminate. It is not an issue other than the extra web traffic that it creates so I mostly mention it so you are aware of it.

Probably the bigger issue for me is that many of the “descriptions” that were available in v2 no longer seem to be returned by the current “pages” API (and so I can’t find them to use the data_objects_articles API). It seems that many of the original English descriptions are no longer returned and mostly we are seeing Italian (language=it) and Spanish (language=es) results. There are still a few cases where the language is returned as NULL and these tend to be English usually. In v2 it seemed that an English description was returned for almost all items (in fact, I use to use text_per_page=1 with the pages API and always got an English text object, now I use text_per_page=10 and usually get multiple Spanish and Italian results and the occasional NULL result).

Sorry about the original error in my pages API call vs the new API structure.

Cheers,

Greg


#40

Understood, thanks. The inconsistencies in Pages are next on the list after hierarchy_entries is returned to service; we’ll make those explicit one way or another.

You’re not the only one waiting on missing content. Alas, some of our oldest and most idiosyncratic resources are text object datasets; we’ve had trouble reharvesting some of them and are addressing the errors case by case. It may be a long winter, but everything should be back in eventually.