Source documents

Source store and documents

The URL below show GET request to retrieve the first 100 source documents from the source store.

GET https://<base-url>/stores-active/source/filter?offset=0&limit=100&api_key=<api-key>

Paramenter	description
offset=0	instructs the API to skip zero document in the store
limit=100	instructs the API to return 100 documents

For more details and options try this in the API explorer

This is the basis of pagination. By systematically adjusting the “offset” and “limit” query parameters, we can download the documents in a store “n” number of items at a time until all documents in the store are downloaded.

GET source documents in chunks

So, if we want to retrieve all source documents but only 100 source documents at a time, we would submit the request above in a loop while adjusting the parameter values in each iteration like in the pseudocode below:

var curr_offset = 0, doc_count = 100;
var response = https://<base-url>/stores-active/source/filter?offset={curr_offset}&limit={doc_count};
while (status code of response is 200) {
        curr_offset = curr_offset + doc_count;
        response = https://<base-url>/stores-active/source/filter?offset={curr_offset}&limit={doc_count};
        /* process response here */
}

Or recursively like below:

 function GetData(curr_offset, doc_count) {
            var response = https://<base-url>/stores-active/source/filter?offset={curr_offset}&limit={doc_count};
            if (status code of response is 200) {
                    /* process response here */
                    GetData(curr_offset+doc_count, doc_count);
            } else {
                    return;
            }
    }

Currently a value of 10,000 is the maximum allowed value for the parameter limit due to on the backend system requirements.

Now that we know how to paginate and use the request to retrieve all documents in the source store in chunks, we can apply the same tehnique to retrieve other types of documents from other stores as well.