JSON API Limits

[MGWben]

Working with a large 3rd party product feed coming into our Miva store, I have setup some server-side cron jobs to manage the import process.

Basically, after some segregated DB tables are made to manage info regarding the feed items, I have an API call created for each product being pulled in to either update the SKU's info in Miva or to insert a new product. There is more going on than just the one API call per product - new items require another call to capture its' newly made Product_ID to then assign images, etc.

With nearly 7k SKUs that get updated or created - or deleted near the end of the process - that's a lot of API calls.

Would grouping Product_Update and Product_Insert calls be more efficient? Is there a limit to the call size for the JSON API?

If I were to group these requests, the standard Product_Update API could have nearly 6k+ iterations. My common sense tells me this is excessive (probably close to 50MB of data), but can the API handle this or is there a better way?

Also, running API calls/responses in a per-item method takes a fair bit of time - hours. Which is no good. :/

[ILoveHostasaurus]

The ideal option for this would be the MultiCall functionality: https://docs.miva.com/json-api/multicall

That would let you batch it on your side to make fewer requests with more data per request; should be faster too. The request size, connection duration, etc. we can probably work with you on, so if you do attempt a certain size that would be ideal and encounter issues, definitely reach out to see if we can tune things. If you're running with a CDN in front, we can probably make some changes for that as well to bypass the CDN.

[MGWben]

Thanks ILoveHostasaurus, I'll give it a shot running a big 'ole push - at least for part of the process. I'll see if that speeds it up any.

If you're running with a CDN in front, we can probably make some changes for that as well to bypass the CDN.

We currently do not have a CDN going, though it is something we are looking into implementing - if we can, but does the JSON API get affected by a CDN in some way?
-Asking for the future :)

[ILoveHostasaurus]

The issue is many CDN's have hard limits on any number of these variables, connection duration, POST size, type of request, they may see heavy API usage from one end point as abusive and block it, etc. So there can sometimes be necessary tuning, addition of rules for the API endpoint, or bypass if it makes sense. Cloudflare, for example, will not allow a request to extend beyond 65 seconds.

[MGWben]

Oh I see. Definitely good to know for the future!

A CDN wouldn't affect a server-side cron job though would it?

[ILoveHostasaurus]

It would without special modification on the server-side to prevent the request from following the domain's normal DNS outbound to the CDN and then back to the server; but that can be worked around.