May 2013

Volume 28 Number 05

SharePoint 2013 - Understanding and Using the SharePoint 2013 REST Interface

By Jim Crowley | May 2013

The Basics

Before it can do anything with SharePoint, your remote Web or mobile application must obtain authorized access. In SharePoint 2013 there are two general approaches to authorizing access to a SharePoint site (regardless of whether you’re using REST). The first approach involves authenticating a SharePoint user, in which case your application has the same access to SharePoint data and capabilities as the user. The different ways you can authenticate a user from a Web or mobile application are outside the scope of this article, but we’ll briefly mention two new options in SharePoint 2013 because they affect the way you construct your REST requests:

• If you’re calling into SharePoint from a remotely hosted application that can’t use client-side code (HTML and JavaScript) exclusively, and there’s no firewall between SharePoint and your application, you can use OAuth 2.0 tokens with Microsoft Access Control Service (ACS) as the secure token server.
• If client-side code and the permissions of a user who’s logged in to SharePoint are sufficient, then the JavaScript cross-domain library (bit.ly/12kFwSP) is a good alternative to OAuth. The cross-domain library is also a convenient alternative to OAuth whenever you’re making remote calls through a firewall. The MSDN Library article, “Data access options for SharePoint 2013” (bit.ly/WGvt9L), describes these options in detail.

Using OAuth The MSDN Library article, “Authorization and authentication for apps in SharePoint 2013” (bit.ly/XAyv28), covers the details of how OAuth works in SharePoint 2013 and how you can get an access token for your application. Once you have a token, you need to pass it with each REST request. You do this by adding an Authorization header that passes the access token as its value, preceded by “Bearer”:

Authorization: Bearer access_token

For examples of C# and JavaScript code that do this, see the “Reading data with the SharePoint 2013 REST interface” section of the MSDN Library article, “How to: Complete basic operations using SharePoint 2013 REST endpoints” (bit.ly/13fjqFn). This example, which retrieves all of the lists from a SharePoint site, shows you what a basic REST request that passes an OAuth token looks like:

HttpWebRequest endpointRequest =
  (HttpWebRequest)HttpWebRequest.Create(
  "https:// <https:///> <site url>/_api/web/lists");
endpointRequest.Method = "GET";
endpointRequest.Accept = "application/json;odata=verbose";
endpointRequest.Headers.Add("Authorization", 
  "Bearer " + accessToken);
HttpWebResponse endpointResponse =
  (HttpWebResponse)endpointRequest.GetResponse();

Using the Cross-Domain JavaScript Library The SharePoint cross-domain library is contained in the file SP.RequestExecutor.js, which is located in the virtual /_layouts/15/ directory on SharePoint servers. To use it, load this file on a remote Web page. In JavaScript, create an SP.RequestExecutor object and then call its executeAsync method. As a parameter to this method you pass the information it needs to construct an HTTP request to the REST service. The main differences between REST calls using OAuth and REST calls using the cross-­domain library are that for the latter, you don’t need to pass an access token in the request, but you do need to specify (in the RESTful URL) the SharePoint Web site that will serve as the client context site. The MSDN Library article, “How to: Access SharePoint 2013 data from remote apps using the cross-domain library” (bit.ly/12kFwSP), discusses these details (such as the difference between an app Web and a host Web) more thoroughly. This example, which retrieves all of the lists from a SharePoint site, shows you what a REST request that uses the cross-domain library looks like:

var executor = new SP.RequestExecutor(appweburl);
  executor.executeAsync(
    {
      url:
        appweburl +
"/_api/SP.AppContextSite(@target)/web/lists?@target='" +
  hostweburl + "'",
  method: "GET",
  headers: { "Accept": "application/json; odata=verbose" },
  success: successHandler,
    error: errorHandler
    }
      );

Constructing RESTful URLs The SharePoint REST service is implemented in a client.svc file in the virtual folder /_vti_bin on the SharePoint Web site, but SharePoint 2013 supports the abbreviation “_api” as a substitute for “_vti_bin/client.svc.” This is the base URL for every endpoint:

https://<domain>/<site url>/_api/

The service-relative URLs of specific endpoints are appended to this base; for example, you can retrieve the lists from a SharePoint site with this URL:

https://<domain>/<site url>/_api/web/lists

You can get a reference to a particular list by specifying its ID or, as in the following example, by its title:

_api/web/lists/getByTitle('samplelist')/

The “web” in these examples is not a placeholder—it’s the name of an object of the Web class in the SharePoint client object model; “lists” is the name of a collection property and “getByTitle” is a method of that collection object. This paradigm enables Microsoft to combine the API reference for the endpoints and the JavaScript object model; for examples, see SP.Web.lists and SP.ListCollection.getByTitle at bit.ly/14a38wZ and bit.ly/WNtRMO, respectively. Also, the syntax roughly mirrors the structure of a SharePoint tenancy. You get information about a site collection at _api/site; information about a SharePoint Web site at _api/web; and information about all of the lists in a Web site at _api/web/lists. The last URL delivers a list collection that consists of all the lists on the SharePoint site. You can also take a look at how these objects are represented in XML by navigating to those URLs on your development site collection.

Every major class of the SharePoint content object model is available, including site collection, Web sites, lists, folders, fields and list items. You can get information about users through the SP.User (bit.ly/15M4fzo), SP.UserCollection (bit.ly/16TQTnW), SP.Group (bit.ly/X55Pga) and SP.GroupCollection (bit.ly/ZnFHbG) classes. The table in Figure 1 shows examples of various endpoints for read operations.

Figure 1 Endpoints for Read Operations

By default, the data is returned as XML in AtomPub format as extended by the OData format, but you can retrieve the data in JSON format by adding the following accept header to the HTTP request:

accept: application/json;odata=verbose

Whether to use JSON or Atom (XML) depends on your skill set, the programming platform you’re using and whether network latency will be an issue for your app. JSON uses far fewer characters, so it makes sense for smaller payloads over the network. On the other hand, most major platforms, including the Microsoft .NET Framework, have rich XML-parsing libraries.

We’ll describe later how you can use OData query operators to select, filter and order the data.

Writing to SharePoint All of the previous requests use the HTTP verb GET. When you write to SharePoint, your requests use the POST verb—though in some cases you’ll override this verb by adding an X-HTTP-Method header to the request and specifying a value of PUT, MERGE or DELETE. In general, POST is used when you’re creating an object such as a site, list or list item. MERGE is used when you’re updating certain properties of an object and you want the other properties to keep their current values. PUT is used when you want to replace an item; properties not specifically mentioned in the request are set to their default values. DELETE is used when you want to remove an item.

Every request that writes to SharePoint must include a form digest. Your code gets the digest as part of a set of information returned by the following endpoint:

_api/contextinfo

You must use the POST verb in this request (with an empty body) and include the Authorization header as described earlier. In some frameworks you’ll have to specify that the length of the POST request is 0. In the structure that’s returned there’s a property named FormDigestValue that contains the form digest. In all subsequent POST requests, you add an X-Request­Digest header with the digest as its value.

Note that if you’re working with a SharePoint-­hosted app and a page that uses the default master page for SharePoint, the digest is already on the page in an element with the ID “__REQUESTDIGEST” (with two underscore characters). So, instead of calling the contextinfo endpoint, you can simply read the value with script such as this jQuery code:

var formDigestValue = $("__REQUESTDIGEST").val()

Of course, you need to add to the body of the request the data you want to write, or an identification of the data you want to delete. You can use either AtomPub/OData format or JSON format. If you choose the latter, you must add a content-type header to the request like the following:

content-type: application/json;odata=verbose

For a full set of concrete examples of Create, Read, Update and Delete (CRUD) operations on SharePoint objects, see “How to: Complete basic operations using SharePoint 2013 REST endpoints” at bit.ly/13fjqFn.  

Advanced Operations

A certain degree of complexity comes along with the power of the SharePoint 2013 REST interface. The interface supports operations for sorting, filtering and ordering the data that it returns. It also supports a large number of SharePoint-specific operations. These additional capabilities add features and benefits that you don’t always see in a standard REST implementation. The next sections discuss some of the most important factors you’ll encounter when working with REST and SharePoint.

Filtering, Selecting and Sorting You can use OData system query options to control what data is returned and how it’s sorted. Figure 2 shows the supported options.

Figure 2 Options for Filtering and Sorting Data

To return the author, title and ISBN from a list called Books, for example, you’d use the following:

_api/web/lists/getByTitle('Books')/items?$select=Author,Title,ISBN

If the $select option isn’t used, all fields are returned except fields that would be resource-intensive for the server to return. If you need these fields, you need to use the $select option and specify them by name. To get all fields, use $select=‘*’.

To get all the books by Mark Twain, use:

_api/web/lists/getByTitle('Books')/items?$filter=Author eq 'Mark Twain'

For a list of all the operators supported for the $filter option, see the MSDN Library article, “Programming using the SharePoint 2013 REST service,” at bit.ly/Zlqf3e.

To sort the books by title in ascending order, use:

_api/web/lists/getByTitle('Books')/items?$orderby=Title asc

Use “desc” in place of “asc” to specify descending order. To sort by multiple fields, specify a comma-separated list of fields.

You can conjoin multiple options using the “&” operator. To get only the Title of the first two books by Mark Twain, use:

_api/web/lists/getByTitle(
  'Books')/items?$select=Title&$filter=Author eq 'Mark Twain'&$top=2

The service will completely resolve each option before it applies the next one. So each option only applies to the data set that’s produced by the options to its left in the URL. Thus, the order in which you apply the options matters. For example, the following URL returns items 3-10:

_api/web/lists/getByTitle('Books')/items?$top=10&$skip=2

But reversing the two options returns items 3-12:

_api/web/lists/getByTitle('Books')/items?$skip=2&$top=10

You can get the bottom n items by using a descending $orderby and a $top option (in that order). The following URL gets the bottom two items:

_api/web/lists/getByTitle('Books')/items?$orderby=ID desc&$top=2

When a SharePoint list has a lookup field to another list, this effectively serves as a join of the two lists. You can use the $expand option to return projected fields from the joined list. For example, if the Books list has a PublishedBy field that looks up to the Name field of a Publisher list, you can return those names with this URL:

_api/web/lists/getByTitle(
  'Books')/items?$select=Title,PublishedBy/Name&$expand=PublishedBy

Notice that you reference the column in the foreign list by using the syntax lookup_column_display_name/foreign_column_name, not foreign_list_name/foreign_column_name. It’s also important to note that you can’t select a lookup field name without also expanding it.

Working with Files and Folders The best way to reach a document library is by taking advantage of the GetFolderByServerRelativeUrl method that’s available at /_api/web. When you add a file to a document library, you send the contents of your file in the request body, and you pass the name of the file in the URL:

https://<site url>/_api/web/GetFolderByServerRelativeUrl(
  '/Shared Documents')/Files/add(url='a.txt',overwrite=true)

An important consideration when updating files is that you can only use the PUT HTTP method. You can’t, therefore, merge the contents of a file into one that’s already stored in a document library. Instead, you replace one version of a file with another. You also need to make sure to use the $value operator in your URL so you get access to the contents of the file itself, instead of the metadata associated with the file:

https://<site url>/_api/web/GetFileByServerRelativeUrl(
  '/Shared Documents/a.txt')/$value

It’s a best practice in SharePoint to check out files before making any changes to them, so you should check a file out before updating it, and then check it back in when you’re done. The following operations require you to make POST requests to these URLs, with empty request bodies:

https://<site url>/_api/web/GetFileByServerRelativeUrl(
  '/Shared Documents/a.txt')/CheckOut()
https://<site url>/_api/web/GetFileByServerRelativeUrl(
  '/Shared Documents/a.txt')/CheckIn(comment='Comment', checkintype=0)

The CheckIn method takes two parameters. The comment parameter enables you to add a comment to the check-in, and the checkintype parameter specifies whether this is a minor (0) or major (1) check-in.

One final consideration is that if you’re working with code (such as JavaScript) that runs in your browser client and you want to upload a file larger than 1.5MB, REST is your only option. This sort of operation with large files (larger than 1.5MB) is available only for Internet Explorer 10 (and later) and other modern browsers of equivalent vintage. The sample in Figure 3 shows how you might upload a binary file using the cross-domain library.

Figure 3 Uploading a Binary File Using the Cross-Domain Library

function uploadFileBinary() {
  var executor = new SP.RequestExecutor(appweburl);
  var body = "";
  for (var i = 0; i < 1000; i++) {
    var ch = i % 256;
    body = body + String.fromCharCode(ch);
  }
  var info = {
    url: "_api/web/lists/getByTitle('Shared Documents')/RootFolder/Files/Add(url='a.dat', overwrite=true)",  
    method: "POST",
    binaryStringRequestBody: true,  
    body: body,  
    success: success,  
    error: fail,  
    state: "Update"};
   executor.executeAsync(info);
}

Change Queries So far we’ve been describing how REST works with SharePoint entities you can reach with URLs that mimic the structure of a SharePoint site. Some SharePoint types can’t be reached or represented this way, however. In the context of REST, three of the most important of these are the ChangeQuery, ChangeLogItemQuery and ChangeToken types.

ChangeQuery objects make it possible for you to query the SharePoint change log for any updates that have been made to a SharePoint site collection, site or list. The REST interface exposes a getchanges method at each of these three locations:

• /_api/site (for site collections)
• /_api/web (for sites)
• /_api/web/lists/list(guid'<list id>') or /_api/web/lists/­getByTitle('list title') (for lists)

You pass a query to any of these locations by adding /getchanges to the relevant URL path, and then sending a ChangeQuery object through the POST body of your request. A simple change query that asks for all items that have been added to a list would look like this (in JSON):

{
  'query': {
    '__metadata': {
      'type': 'SP.ChangeQuery'
    }, 
    'Add': 'true',
    'Item': 'true'
  }
}

The getchanges method expects a request body that contains a representation of the ChangeQuery object inside the query parameter. You send this request to the URL for a specific list:

/_api/web/lists/list(guid'<list id>')/getchanges

or

/_api/web/lists/getByTitle('<list title>')/getchanges

The response returns a result that contains a collection of changes that matches the request. If the list has only one item, the response body would look like this:

{"d":
    {"results":[{
        "__metadata":{
            "id":"https://<site url>/_api/SP.ChangeItema7e7c6e9-2c41-47c3-aae9-2b4a63b7a087",
            "uri":"https://site url/_api/SP.ChangeItem",
            "type":"SP.ChangeItem"},
        "ChangeToken":{"__metadata":{"type":"SP.ChangeToken"},
            "StringValue":"1;3;482e418a-0900-414b-8902-02248c2e44e8;634955266749500000;5749111"},
        "ChangeType":1,
        "SiteId":"ce11bfbb-cf9d-4b2b-a642-8673bd48cceb",
        "Time":"2013-02-03T22:17:54Z",
        "ItemId":1,
        "ListId":"482e418a-0900-414b-8902-02248c2e44e8",
        "WebId":"a975b994-fc67-4203-a519-b160175ca967"}]
    }
}

This response tells you that a single list item (with an ItemId value of 1) was added at a time that’s represented in the change log by a ChangeToken. You can use the string value of this object to make your queries more precise. You can, for example, specify values for the ChangeTokenStart and ChangeTokenEnd properties of your ChangeQuery object to make sure you get changes that occurred before or after a single point in time, or between two points in time.

You can also use the value of the ChangeToken object when you use the getListItemChangesSinceToken method:

/_api/web/lists/list(guid'<list id>')/getListChangesSinceToken

This method exists only in the REST interface. If you want to know all of the changes to items in this list since the addition of the first item, you construct a ChangeLogItemQuery object that contains the change token:

{
    'query': {
        '__metadata': {
            'type': 'SP.ChangeLogItemQuery'
        }, 
    'ChangeToken':'1;3;482e418a-0900-414b-8902-02248c2e44e8;634955266749500000;5749111'
    }
}

SharePoint Server 2013 Feature Areas All of the operations we discuss in this article apply to both SharePoint Foundation 2013 and SharePoint Server 2013, because they involve core SharePoint capabilities. The SharePoint REST interface also exposes many of the capabilities of SharePoint Server 2013 feature areas. These feature areas lie outside the scope of this article, but you can refer to the following resources in the SDK for more information about using REST to work with them:

• “SharePoint 2013: Using the search REST service from an app for SharePoint” (bit.ly/Mt4szN)
• “Get started developing with social features in SharePoint 2013” (bit.ly/102qIGM)
• “BCS REST API reference for SharePoint 2013” (bit.ly/10FFMMu)

Debugging

The most important piece of information you need in order to perform a REST operation is obviously the correct URL. We’ve mentioned a lot of the most important ones, and for many of the others, you can refer to the SharePoint SDK. Because the REST interface is modeled on the client object model, you can refer to the JavaScript object model reference for information about REST endpoint URLs. For example, if you want to see what URLs are available for working with list collections, you can refer to the reference documentation for the SP.ListCollection object at bit.ly/108hI1e.

You can also navigate to a REST URL as a logged-in user and view the XML output of any GET request in order to see the available data at each endpoint and how it’s structured. This won’t help you with POST requests, but it can help you familiarize yourself with the different SharePoint entities and the information available at each endpoint.

It’s important that the HTTP requests you send from your code contain correctly encoded URLs. When you launch an app for SharePoint from SharePoint, you can retrieve an encoded URL from the SPHostUrl query string argument, but in other contexts you might have to encode the URL yourself.

When you’re doing more complex operations—especially when you’re performing operations that require the POST HTTP verb—you’ll need to use an HTTP tracing utility in order to debug your HTTP requests. SharePoint returns error messages whenever you make a bad request, and those messages can tell you a lot about what’s going wrong with your requests. For example, your application or user may simply not be authorized to get certain kinds of information from SharePoint. At other times, you may have constructed an invalid JSON object, or you might have assigned an invalid value to a property.

Some frameworks provide HTTP tracing utilities for you. You can use trace.axd (bit.ly/8bnst4) when you’re working with ASP.NET applications. If you’re sending requests directly from your browser, as with JavaScript, you can use Fiddler (fiddler2.com/fiddler2). We used Fiddler to generate the sample HTTP responses we included in this article.

Using REST to Talk to SharePoint in a PHP Application

As we said in our introduction, the REST interface allows you to interact with SharePoint from any of the standard languages and frameworks Web developers commonly use. In order to demonstrate this, we’ve published a sample PHP application that demonstrates how you can interact with a SharePoint site from a remote Web application written in PHP. This particular application is an app for SharePoint that’s designed to be launched from an Office 365 SharePoint site and run from a Windows Azure Web Site. This architecture simplifies some steps, such as publishing the Web site, but the PHP code in the sample can run on any architecture that supports PHP.

You can view and download the sample from the code gallery page at code.msdn.microsoft.com/office/SharePoint-2013-Perform-8a78b8ef. The sample demonstrates a number of things, including how to work with files and folders with REST, how to obtain a valid OAuth access token from a PHP application, and how to use the JavaScript cross-domain library. Most important for this context, it demonstrates how to retrieve files from and upload files to a SharePoint document library using REST and PHP.

Because the application writes data to a SharePoint site, one of the first things the application needs to do (after obtaining an access token) is request the form digest from _api/contextinfo. The request passes the access token in the headers and sets up a POST request to an SSL URL. The code, shown in Figure 4, will be familiar to anyone who has worked with PHP client URL objects.

Figure 4 Requesting the Form Digest

$opts = array (
    'Authorization: Bearer ' .
    $accToken);
$ch = curl_init();
$url = $appweburl . '/_api/contextinfo';
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HTTPHEADER, $opts);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, '');
curl_setopt ($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($ch);

After executing the request, the application parses the XML and stores the form digest value:

$root = new SimpleXmlElement($result);

$ns = $root->getNameSpaces(TRUE);
$childrenNodes = $root->children($ns['d']);
$formValue = $childrenNodes->FormDigestValue;

It also stores the access token in a cookie. When the user chooses to upload a file, the application passes those values to an HTML file that constructs the HTTP request for uploading files to a document library. Before setting up the request, it reads the data from the locally stored file into a string object that will be passed as the body of the POST request:

$accToken = $_COOKIE["moss_access_token"];

$url = $_REQUEST["target"] . 
$furl = $_FILES["file"]["tmp_name"];
$file = fopen($furl,"r");
$post_data = fread($file,filesize($furl));
fclose($file);

The lines in Figure 5 retrieve the form digest and access token values and set up and execute the HTTP request that uploads the file.

Figure 5 Executing the Request that Uploads the File

"/_api/web/GetFolderByServerRelativeUrl('Lists/SharedDoc')/Files/
add(url='" . $_FILES["file"]["name"] . "',overwrite=true)";
$opts = array (
'X-RequestDigest:' . $_REQUEST["digest"],
'Authorization: Bearer ' . $accToken);
// Initialize cURL
$ch = curl_init();
curl_setopt($ch, CURLOPT_HTTPHEADER, $opts);
// Set URL on which you want to post the Form and/or data
curl_setopt($ch, CURLOPT_URL, $url);
// Data+Files to be posted
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_data);
// Pass TRUE or 1 if you want to wait for and catch the 
// response against the request made
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// For Debug mode; shows up any error encountered during the operation
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt ($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, 0);
// Execute the request
$response = curl_exec($ch);

What Next?

Although it doesn’t have complete parity with the client object model, the SharePoint 2013 REST interface is extensive and powerful enough to provide most of what Web and mobile app developers will want to do, especially if they’re working with frameworks other than .NET. We’ve looked at many of the most important ways in which you can integrate SharePoint into your applications by using the REST interface, but there are many more possibilities.

The SharePoint 2013 SDK contains a collection of resources for REST development, and you can find links to all of them on the MSDN Library page, “SharePoint 2013 REST API, endpoints and samples” (bit.ly/137q9yk). This collection will grow, and it will contain an especially wide range of samples because, as the PHP sample demonstrates, the REST interface significantly expands the world of SharePoint development.

Jim Crowley is a senior programming writer in the Office Division. He writes developer documentation for SharePoint 2013. He also wrote the SharePoint Developer Documentation RSS Reader app (bit.ly/YIVbvY) for Windows Phone 7.

Ricky Kirkham is a senior programming writer in the Office Division and a Microsoft Certified Professional Developer (SharePoint 2010). Kirkham has been on the Microsoft developer documentation team for SharePoint since 2006.

THANKS to the following technical expert for reviewing this article: Jyoti Jacob (Microsoft)