Some Microsoft Graph queries can return a large number of entities, more than can practically be included in a single JSON payload. In those cases, Graph paginates responses to improve performance as well as developer convenience and flexibility.
This repo contains Python-based examples of how to work with Graph's paginated responses. For a high-level overview of how pagination works in Microsoft Graph, see Paging Microsoft Graph data in your app.
The samples in this repo use messages to illustrate how pagination works, but the same concepts can be applied to any Graph API that uses pagination, including messages, contacts, users, groups, and others.
Before installing the samples, verify that you have these prerequisites in place:
- Install Python from https://www.python.org/. We've tested the code with Python 3.6.2, but any Python 3.x version should work fine. If your code base is running under Python 2.7, you may find it helpful to use the 3to2 tools to port the code to Python 2.7.
- To register your application for access to Microsoft Graph, you'll need either a Microsoft account or an Office 365 for business account. If you don't have one of these, you can create a Microsoft account for free at outlook.com.
Then follow these steps to install the sample code:
- Clone this repo:
git clone https://github.com/microsoftgraph/python-sample-pagination.git. - Create and activate a virtual environment (optional). If you're new to Python virtual environments, Miniconda is a great place to start.
- In the root folder of your cloned repo, install the dependencies for the sample as listed in requirements.txt with this command:
pip install -r requirements.txt.
To complete the configuration of the samples, you'll need to register a new application in the Microsoft Application Registration Portal. You only need to do this once, and then any Microsoft identity can be used to run the samples.
Follow these steps to register a new application:
-
Sign in to the Application Registration Portal using either your personal or work or school account.
-
Under My applications, choose Add an app. If you're using an Office 365 account and see two categories listed (Converged or Azure AD only), choose Add an app for the Converged applications section.
-
Enter an application name, and choose Create. (Do not choose Guided Setup.)
-
Next you'll see the registration page for your app. Copy and save the Application Id field.You will need it later to complete the configuration process.
-
Under Application Secrets, choose Generate New Password. A new password will be displayed in the New password generated dialog. Copy this password. You will need it later to complete the configuration process.
-
Under Platforms, choose Add platform > Web.
-
Under Delegated Permissions, add the Mail.Read permission.
-
Enter
http://localhost:5000/login/authorizedas the Redirect URL, and then choose Save.
As the final step in configuring the sample, modify the config.py file in the root folder of your cloned repo, and follow the instructions to enter your Client ID and Client Secret (which are referred to as Application Id and Password in the app registration portal). Then save the change, and you're ready to run the samples.
Graph's approach to pagination for potentially large result sets uses the odata.context and odata.nextLink annotations that are defined in OData JSON Format Version 4.0.
When you query a paginated Graph API (for example, me/messages), you'll get back a JSON payload that contains these top-level elements:
@odata.contextcontains a URI that identifies the type of data being returned. This value is the same for every page of the result set.@odata.nextLinkcontains a link to the next page of results. You can do a GET against that endpoint to return the next page, which will contain a link to the next page after that, and you can repeat this process until the final page, which will not have this element.valuecontains the returned data, as a list of JSON elements. In theme/messagesexample, this would be a list of email messages. The number of items returned is based on the page size. Each paginated API has a default page size (for example, theme/messagesdefault is 10 messages), and you can specify a different page size if desired through use of the$topparameter. Note that the default page size and maximum page size may vary for different Graph APIs — see Paging Microsoft Graph data in your app for more information.
The following diagram shows how this works in practice, using the me/messages endpoint as an example.
The pagination.py sample in this repo provides an interactive demonstration of how it works. Follow the Installation instructions to install the sample, and then run it as follows:
- At the command prompt:
python pagination.py - Navigate a browser to http://localhost:5000
- Choose Connect and authenticate with a Microsoft identity (organization account or Microsoft Account)
You'll then see the following page listing your most recent 10 messages:
The @odata.nextLink value links to the next page of results. Each time you click on the Next Page button, the next page of results is loaded. This is the fundamental behavior of paginated responses from Graph APIs.
Some Graph APIs return all of the requested entities by default, and in that case the @odata.nextLink element is not provided. The absense of this element tells you that the value element contains all entities in a single page.
For example, if there are less than 250 items in your OneDrive root folder, you will see this JSON response when you request all of the DriveItems in the folder by doing a GET to the https://graph.microsoft.com/v1.0/me/drive/root/children endpoint:
Since there is no @odata.nextLink element, you know that this is a complete result set that contains all of the requested DriveItems. The default page size for this API is 250 items, so they all fit within a single page of results.
But the same API can also return paginated responses, if the result set is parger than the page size. For example, here we're using the $top query string parameter to return only the first 10 items from the same set:
In this case, we've received the first 10 DriveItems, and there is an @odata.nextLink value which we can use to query the next page of 10 items.
As a best practice, your code should allow for the fact that @odata.nextLink may be missing, in which case there is no pagination to be handled. There is an example of this in the generator sample below.
The Graph API returns pages of results, as demonstrated in pagination.py. But in your application or service, you may want to work with a single non-paginated collection of items such as messages, users, or files. In this sample, we create a Python generator that hides the pagination details so that your application code can simply ask for a collection of messages and then iterate through them using standard Python idioms such as for messages in messages or next(message).
The generator.py sample in this repo provides an interactive demonstration of how it works. Follow the Installation instructions to install the sample, and then run it as follows:
- At the command prompt:
python generator.py - Navigate a browser to http://localhost:5000
- Choose Connect and authenticate with a Microsoft identity (organization account or Microsoft Account)
You'll then see the most recent message you've received:
Each time you click the Next Message button, you'll see the next message. The generator() function in generator.py handles the details as shown here:
def graph_generator(session, endpoint=None):
"""Generator for paginated result sets returned by Microsoft Graph.
session = authenticated Graph session object
endpoint = the Graph endpoint (for example, 'me/messages' for messages,
or 'me/drive/root/children' for OneDrive drive items)
"""
while endpoint:
print('Retrieving next page ...')
response = session.get(endpoint).json()
for item in response.get('value', None):
yield item # return next item from this page
endpoint = response.get('@odata.nextLink', None)The key concept to understand in a Python generator is the yield statement, which returns a value but also retains the state of the generator function for the next call. We have an outer loop that steps through the paginated results (while endpoint:) and an inner loop (for item in ...) that returns the items from within each page.
To create a generator at runtime, we pass the Graph session connection object and the API endpoint for retrieving messages:
MSG_GENERATOR = messages(MSGRAPH, 'me/messages')Then the calling code simply uses Python's built-in next() function to retrieve messages:
def generator():
"""Example of using a Python generator to return items from paginated data."""
return {'graphdata': next(MSG_GENERATOR)}We call next(MSG_GENERATOR) whenever we need the next message, and we don't need to be aware of the fact that paginated results are coming from Graph. As a practical matter, you may notice a slightly longer response time whenever a new page is retrieved (every 10th message, with the default page size of 10 messages in the sample), but the individual items within each page are returned immediately without any need to call Graph, because they're in the page of results that is being retained in the state of the generator function after each yield statement.
Python generators are recommended for working with all paginated results from Microsoft Graph. You can use the generator() function in this sample for users, groups, drive items, and other paginated responses from Graph APIs.
These samples are open source, released under the MIT License. Issues (including feature requests and/or questions about this sample) and pull requests are welcome. If there's another Python sample you'd like to see for Microsoft Graph, we're interested in that feedback as well — please log an issue and let us know!
This project has adopted the Microsoft Open Source Code of Conduct. For more information, see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.