Derpibooru
Filters (Default) Settings

Breaking API changes

Site and Policy » Breaking API changes Search Posts
Started by liamwhite1
Posted
17 replies
Subscribe
liamwhite1
Perfect Pony Plot Provider - 10+ s with over 350 upvotes or more (Questionable/Explicit)
A Perfectly Normal Pony - <%byte[]> I keep logs up my ass

As of commit dd939f3 the Derpibooru API has introduced a major change with the introduction of interactions (ie, votes and faves) in the response.
 
 
• For the /search.json endpoint, nothing should be actually broken. interactions is just an extra field in the response hash. The same array has been added to /images/:id.json .
 
• The /images/page/*.json have been rewritten to return a dictionary {"images":[images], "interactions":[interactions] } instead of an array with the images. The same thing has been done with the /lists/*.json pages.
 
• Finally, a new route has been added to the interactions API that allows callbacks to show te ’s interactions on a given set of images (limit 50). The route can be accessed via:
 
GET /api/v2/interactions/interacted.json?class=Image&ids=image,ids&key=API_KEY
 
Sample requests:  
~@/api/v2/interactions/interacted.json?class=Image&ids=556feeed636872357b110d00  
~@/api/v2/interactions/interacted.json?class=Image&ids=5576d92463687248b0000a3f,54f6764d6368722f1e290200
 
 
This should provide the necessary data to developers that want access to existing vote and fave info without having to fall back to the poor solution of parsing the HTML of the pages.
Posted Report
p-a-t-a-r
Artist -

Prehistoric Brony-fish
I can’t decide whether to love you for implementing this or hate you for not saying anything until AFTER I’d hacked together the code to parse the HTML response. Ah well. Code time!
Posted Report
gallagher117
Platinum bit -
Heart Gem -
Wallet After Summer Sale -
Happy Derpy! -
Silver er -
The End wasn't The End - Found a new home after the great exodus of 2012

ｃａｎ'ｔ ｈａｖｅ ｎｉｃｅ ｔｈｉｎｇｓ
Fortunately, I only started work on my API binding this weekend and it’s only to the point of parsing an image result (and comment array, and fave array, etc.).
 
Once it’s done(ish), I’ll make a thread.
Posted Report
Taivastiuku
The End wasn't The End - Found a new home after the great exodus of 2012

@liamwhite1  
Thanks for implementing this and warning about the change with a notification. I’ll put it into use in Derpibooru Explorer.
 
 
Some thoughts:
 
For /images/page/ queries there could be a &interactions=true flag in case interactions are not needed. Just like the &fav=true flag.
 
I’d prefer to use images id_number rather than id for queries although there is probably a technical reason for using id.
 
I’d also prefer more straightforward response format. For example:
 
GET /api/v2/interactions/interacted.json?class=Image&ids=55769afb6368721b4b003412,557ad21463687247b600190e
 
Instead of:  
[@   {@  
"_id": "55769ead63687268da000ce9",  
"interactable_id": "55769afb6368721b4b003412",  
"interactable_type": "Image",  
"interaction_type": "faved",  
"_id": "asdfasdfasdfasdfasdfasdf",  
"value": null  
},  
{  
"_id": "55769ead63687268da000ce8",  
"interactable_id": "55769afb6368721b4b003412",  
"interactable_type": "Image",  
"interaction_type": "voted",  
"_id": "asdfasdfasdfasdfasdfasdf",  
"value": "up"  
}  
@]@
 
The response could be something like:  
{"55769afb6368721b4b003412": ["faved", "voted_up"], "557ad21463687247b600190e": []}
 
I don’t know of what use _id, interactable_type and _id are for interaction queries. Not a big deal though since I know how to map().
Posted Report
liamwhite1
Perfect Pony Plot Provider - 10+ s with over 350 upvotes or more (Questionable/Explicit)
A Perfectly Normal Pony - <%byte[]> I keep logs up my ass

@Taivastiuku  
For /images/page/ queries there could be a &interactions=true flag in case interactions are not needed. Just like the &fav=true flag.
I think the primary reason is that we calculate @interactions regardless of whether the cares about having them, but maybe in the future this can be an option as to whether they are included in the reponse.  
I’d prefer to use images id_number rather than id for queries although there is probably a technical reason for using id.
I am not sure about this one, but I am more comfortable using the raw BSON ID to look up things myself. I’ve probably spent too much time in the console though.  
I’d also prefer more straightforward response format. For example:
GET /api/v2/interactions/interacted.json?class=Image&ids=55769afb6368721b4b003412,557ad21463687247b600190e
<snip>
The response could be something like:
{"55769afb6368721b4b003412": \["faved", "voted\_up"\], "557ad21463687247b600190e": \[\]}
I don’t know of what use _id, interactable_type and _id are for interaction queries. Not a big deal though since I know how to map().
 
This again has to do with the format our database (which is big, clunky and slow) stores interactions in, but has a fairly straightforward JSON representation. The _id field is Mongo’s object ID, interactable_type is AFAIK a remnant from the days of voting on tag changes, and _id is how we get your interactions only. It would be fairly straightforward for me to make a proper to_json method for interactions, but really I’m just lazy and afraid of breaking things.
Posted Report
Taivastiuku
The End wasn't The End - Found a new home after the great exodus of 2012

@liamwhite1  
For /images/page/ queries there could be a &interactions=true flag in case interactions are not needed. Just like the &fav=true flag.
I think the primary reason is that we calculate @interactions regardless of whether the cares about having them, but maybe in the future this can be an option as to whether they are included in the reponse.
&fav=true was exists for performance reasons. I don’t know whether interaction queries are heavy for the database but I was thinking that leaving them out might help with the site resources.
 
I’d prefer to use images id_number rather than id for queries although there is probably a technical reason for using id.
I am not sure about this one, but I am more comfortable using the raw BSON ID to look up things myself. I’ve probably spent too much time in the console though.
Can you tell id_number of 4f139079945b7003540000a7 without using /developer privileges? Using id_number feels more natural to me because it can be easily used to fetch image information and it is present in all derpibooru image filenames.
Posted Report
liamwhite1
Perfect Pony Plot Provider - 10+ s with over 350 upvotes or more (Questionable/Explicit)
A Perfectly Normal Pony - <%byte[]> I keep logs up my ass

@Taivastiuku  
&fav=true was exists for performance reasons. I don’t know whether interaction queries are heavy for the database but I was thinking that leaving them out might help with the site resources.
That wasn’t my point though; we calculate interactions for every list regardless of whether they are used or not. Also, the &fav=true param was added because looking up the faves for every on a particular image is a relatively expensive operation, but faves+votes for the current is not quite as expensive (AFAIK). The relevant code looks something like  
@interactions = \#\<complex mongo query\> if current\_ respond\_to do \|format\|   format.html   format.json { render json: {images: @images, interactions: \(@interactions \|\| \[\]\)}.to\_json } } end  
Can you tell id_number of 4f139079945b7003540000a7 without using /developer privileges? Using id_number feels more natural to me because it can be easily used to fetch image information and it is present in all derpibooru image filenames.
 
Yes:  
GET /api/v2/images/show.json?ids=4f139079945b7003540000a7  
GET /api/v2/images/show.json?ids=5576d92463687248b0000a3f,54f6764d6368722f1e290200
Posted Report
Clover the Clever
Princess of Love - Extra special version for those who participated in the Canterlot Wedding 10th anniversary event by contributing art.
A Perfectly Normal Pony - <@CloverTheClever> I'd pay to see Carcer in a fursuit
Always Codes Drunk - It explains a lot
From the Night - I have technically banned myself a bunch of times...
Since the Beginning - number zero

Lord and Saviour
@Taivastiuku
 
fav=true and comment=true in particular are expensive because they hit the favs array but then need to load s to show names etc. This can get very slow on images with hundreds of comments/faves.
 
Interactions on the other hand are a simple $in lookup, which are fast, and well-bounded by the number of objects being queried. They also have no dependent objects for use.
 
As for IDs, it’s trivial for us to add an option to query by either id or id_number, whichever is provided, for images.
Posted Report
Background Pony #3CCA
When we wish to access images beyond the ones visible with the “Default” filter, are we still advised to create a new and use the “Everything”-filter? Are there any changes coming up (such as a filter parameter) to make this unnecessary?
Posted Report
Background Pony #3CCA
@OrionFOTL
 
I believe there is a slight misunderstanding. I am posting this in a thread concerning the API. I do not use my browser to access the API.
Posted Report
Taivastiuku
The End wasn't The End - Found a new home after the great exodus of 2012

@Clover the Clever  
As for IDs, it’s trivial for us to add an option to query by either id or id_number, whichever is provided, for images.
GET /api/v2/images/show.json?id_numbers=841846,913527
 
This would be really useful for scripts. With this id_number would work both as route to an image (like when wishes to go “next”) and a handle for querying image information from the API if thumbnails need to be generated. Now id_number works only for routing and id only for API access.
Posted Report
Interested in advertising on Derpibooru? Click here for information!
Midnight, Equestria - A Roleplaying MUD

Help fund the $15 daily operational cost of Derpibooru - us financially!

Edit Preview
Syntax quick reference: **bold** *italic* ||hide text|| `code` __underline__ ~~strike~~ ^sup^ ~sub~

Detailed syntax guide