Last Update: 04.12.2023

What is paging/pagination?

Paging generally describes a numbering of pages, usually located at the top or bottom of the website pages (Google search results). API paging is nothing more than retrieving all records per page based on an HTTP request.

It is used in environments where many records can be provided or fetched, and ensures dividing result lists into manageable sections. Depending on the implementation, there can be different types of paging. We will discuss the two most common variants.

A detailed description and further variants can be found here: https://dev.to/pragativerma18/unlocking-the-power-of-api-pagination-best-practices-and-strategies-4b49

General setup

With version 4.6.6 there is a new checkbox in the time-driven Input Agent HTTP (Activate paging). This enables paging within a profile without the need for a preceding profile.

The prerequisite is that the variable VAR_SYS_HTTP_PAGING_URL is used in phase 3. This variable is used to call up the profile again until no more input data can be fetched.

Usage

Cursor paging: RECOMMENDED - since only synchronous call possible.

Offset paging: DEPENDS - the offset can also be calculated using a logic in phase 3. For larger datasets, the asynchronous profile chain (example below), which performs parallel processing, is more suitable for offset paging.

Example profile: Profile-paging_checkbox_phase1.pak

Note: Profiles with the active paging checkbox are called synchronously!

Offset paging

In offset paging, the data to be determined is limited by limit and offset. The limit has a fixed maximum (50 for Spotify).

Example: https://api.spotify.com/v1/artists/43ZHCT0cAZBISjO8DG9PnE/albums?include_groups=single&market=US&limit=50&offset=0

The Spotify API returns 50 Michael Jackson albums (the first page).

Response

{
  "href": "https://api.spotify.com/v1/artists/3fMbdgg4jU18AjLCKBhRSm/albums?include_groups=album,single,compilation,appears_on&offset=0&limit=1&market=DE&locale=de,en-US;q=0.7,en;q=0.3",
  "limit": 50,
  "next": "https://api.spotify.com/v1/artists/3fMbdgg4jU18AjLCKBhRSm/albums?include_groups=album,single,compilation,appears_on&offset=1&limit=1&market=DE&locale=de,en-US;q=0.7,en;q=0.3",
  "offset": 0,
  "previous": null,
  "total": 274,
  "items": [
              ....data details (cut off)...
  ]
}

In the response, you can see that a key value total is included (line 7) which indicates how many records exist.

If we want to query all records, we can divide the total by the limit to get the number of requests needed: 274 / 50 = 5.48.

Therefore, to get all the albums, we need 6 requests.

Setup in profiles

There are several possibilities to implement this:

Profile chain - fast and efficient
In the first profile, a single request is made so that the total number of requests can be calculated. Then the offset for each iteration is calculated and the data profile is triggered.
Sample profiles: Profile-paging_tutorial_start_profile.pak and Profile-paging_tutorial_write_data.pak. Warning: The paging_tutorial_write_data profile is triggered asynchronously (very fast in parallel), which can lead to rate limitations with some APIs, or could possibly trigger an intrusion detection in firewalls!
Single profile - for smaller amounts of data
Point 1 can also be implemented within one profile using the http function. Due to the complex mapping, this is usually only suitable for smaller amounts of data!

Cursor paging

Cursor paging behaves somewhat differently than offset paging. The next page is given by a subsequent link/UniqueID or token in the previous request and cannot be calculated in advance.

Example: https://graph.microsoft.com/v1.0/users?$top=2

The Microsoft Graph API thus provides us with two Microsoft users per page.

Response

{
    "@odata.context": "https://graph.microsoft.com/v1.0/$metadata#users",
    "@odata.nextLink": "https://graph.microsoft.com/v1.0/users?$top=2&$skiptoken=RFNwdAIAAQAAACs6YWRtX3NjaG90d...i00MmQ5LTkxMTQtYzNkMDMxMGY1YWRjuQAAAAAAAAAAAAA",
    "@microsoft.graph.tips": "This request only returns a subset of the resource's properties. Your app will need to use $select to return non-default properties. ...",
    "value": [
        {
            "businessPhones": [],
            "displayName": "Daniel Mueller(ADM-Account)",
            "givenName": "Daniel",
            "jobTitle": null,
            "mail": null,
            "mobilePhone": null,
            "officeLocation": null,
            "preferredLanguage": null,
            "surname": "Mueller",
            "userPrincipalName": "adm_kaufmann@xxxxxx.onmicrosoft.com",
            "id": "f33db0bc-e5bf-4f39-bec7-e55aa2c60169"
        },
        {
            "businessPhones": [],
            "displayName": "Andreas Meier(admin)",
            "givenName": "Andreas",
            "jobTitle": null,
            "mail": null,
            "mobilePhone": null,
            "officeLocation": null,
            "preferredLanguage": null,
            "surname": "Meier",
            "userPrincipalName": "adm_schott@xxxxxx.onmicrosoft.com",
            "id": "c6e4974c-f1a2-42d9-9122-c3d0310d5adc"
        }
    ]
} 

In line 3 of the response, we get back a value for key @odata.nextLink. That is an automatically generated link, which returns the following two records.

The subsequent links are returned, until the last record has been queried.

To get all records via this method, we have to call the next links of @odata.nextLink until there is no key value @odata.nextLink in the JSON response or the value is empty.

Setup in profiles

There are several possibilities to implement this:

Profile chain - fast and efficient
In the first profile, a single request is made, which only passes the URL to the subsequent profile. The subsequent profile calls itself in the Response and automatically passes itself the new variable for phase 1.
In phase 3 the value of @odata.nextLink is checked and in the case of an empty content, the last job is aborted without error using the abort function.
Example profiles: Profile-paging_tutorial_start_graph_api.pak and Profile-paging_tutorial_graph_api.pak.
Warning: Profile loops can very quickly cause a crash if they are not terminated correctly or the logic is faulty. Please do some tests with single requests beforehand to see whether the abort logic works correctly!
Single profile - for smaller amounts of data
Point 1 can also be implemented within a profile using the http function. This is usually only suitable for smaller amounts of data due to the complex mapping! Advantage: No profile loop is required and therefore it is somewhat safer to implement.