Starting in the 2010 release, SharePoint has had support for consuming data via a REST interface. In SharePoint 2013, this functionality is exposed via the /_api/ relative path, which returns data following (quite loosely) the OData specification.
One of the stumbling blocks for SharePoint REST programming was the absence of the $metadata endpoint. The purpose of this endpoint is to describe the data (the XML schema and/or JSON objects) returned by the service. Most development tools in use today can use the $metadata endpoint to generate language objects to model the data. When SharePoint 2013 release for on-premises installations, the $metadata endpoint was not included.
As of the April 2013 CU for SharePoint 2013, the $metadata endpoint is now available. This whitepaper discusses consuming this REST API using the WCF Data Services library, highlighting similarities and differences with other approaches for consuming REST services as well as other approaches for consuming SharePoint data.
About the Sample Site
For my samples, I am using a SharePoint list that is a catalog used in Cross Site Publishing. This list is named ContosoProducts and contains several hundred items describing various electronics. This list includes a Taxonomy field that is used to group the items.
About the API
The /_api/ endpoint serves data for both the Client-Side Object Model API as well as the REST/OData API. So, the $metadata endpoint will provide information about the native SharePoint objects (Sites, Lists, Users, etc.) as well as Content Type-specific information about list items. When Visual Studio creates context classes based on the metadata, these two "personalities" of the endpoint are reflected in two different classes: ApiData and ListData.
Consuming the API
To consume the API in a programmer-friendly way, Visual Studio can generate a proxy class based on the metadata document provided by the service. When dealing with a REST data service, the proxy is generated by the WCF Data Services Client library.
The Microsoft patterns & practices group has an article that discusses the WCF Data Services Client library in the context of the SharePoint 2010 REST interface. The information in that article is still valid and reading that article is recommended.
Installing the WCF Data Services Client
The WCF Data Services Client library is available as a package from the NuGet Package Manager. The NuGet package system is a facility to update Visual Studio projects with libraries (both from Microsoft and from open source communities). There is a large NuGet gallery online at http://www.nuget.org/. The Package Manager in Visual Studio displays items from this gallery.
Once you have created a new project, click Tools > Library Package Manager > Manage NuGet Packages for Solution... (Figure 1).
Figure 1 - Menu option for managing NuGet packages in the current solution
In the Manage NuGet Packages dialog, there is a search box in the upper right. Enter 'wcf' to search for WCF packages. In the search results, select WCF Data Services Client (the current version for VS2012 is 5.6) as shown in Figure 2.
Figure 2 - Manage NuGet Packages dialog with search results for 'wcf'
Click the Install button to download and install the WCF Data Services Client package. This package has some prerequisites, which will also be downloaded and installed. Visual Studio will likely prompt you to accept license terms (Figure 3) during the download/install process.
Figure 3 - Accept License Terms dialog
After the NuGet packages are installed, we can add a service reference to the REST endpoint.
The WCF Data Services Client library will read the metadata of a service and generate many classes to aid in the use of that service. The communication pieces are encapsulated in a context class and the data structures are mapped to classes. To generate these classes in a proxy object, we add a Service Reference to our project. In the Service Reference dialog, enter the metadata endpoint for the applicable SharePoint site. This url is relative to the site. In Figure 4, the SharePoint site url is
http://wcm.wingtip.com/sites/products and the metadata endpoint is
Figure 4 - Add Service Reference dialog
The resulting proxy has hundreds of classes in SharePoint-related namespaces. The Object Viewer window in Visual Studio (Figure 5) can help locate these namespaces and classes.
Note: In the project that accompanies this article, the service reference was given a namespace of "SPProxy." Accordingly, all namespaces discussed in the remainder of this whitepaper use the SPProxy namespace in front of the namespace of the metadata document.
Figure 5 - SharePoint REST endpoint proxy in Object Viewer
SharePoint provides two APIs for accessing its resources via client-computers. The Client Side Object Model (CSOM) is best suited for accessing native SharePoint objects (Sites, Lists, Users, etc.) while the REST/OData endpoint is best suited for List Items. Both of these APIs talk to SharePoint using the same service (client.svc). Since the $metadata endpoint also uses this service, the resulting metadata covers both APIs. In our generated proxy, we will have two context classes:
- SPProxy.SP.ApiData - useful for CSOM-type operations
- SPProxy.SP.Data.ListData - useful for REST/OData-type operations
And, since the metadata document uses the schema from Entity Framework, the data context methods for creating, updating and deleting list items are generated as well. (However, due to the implementation of REST in SharePoint, we will not be able to use them. More details later.)
ApiData Data Context
The ApiData context object, which provides data and operations analogous to CSOM. Inspecting the metadata document (Figure 6) shows the Entities (EntitySet element) and Operations (FunctionImport element) available thru the ApiData object.
Figure 6 - Snippet of $metadata document showing ApiData
In typical Entity Framework coding conventions, referring to a property on the data context object causes the context to read the storage medium to retrieve the entities represented by that property. In SharePoint, however, it does not work that way. As you may recall, the CSOM paradigm requires us to call the Load() method of the client context to express our intent and the data is not pulled from SharePoint until the Execute() method is invoked. Similarly, in the REST interface, only the resource addressed in the url is retrieved, unless directed using the $select or $expand parameters.
The ApiData context object works in a similar fashion - it does not automatically load any data. However, it will cast the returned data into the appropriate type. We do not need to deserialize the response data. This provides a substantial reduction in the code we need to write.
ListData Data Context
The metadata document for the ListData data context (Figure 7) does not define operations (FunctionImport elements) but it does include list associations (AssociationSet elements). However, the associations (table relations) are not exposed thru the data context object.
Figure 7 - Snippet of $metadata document showing ListData
The ListData data context object performs closer to a typical entity framework context. The properties on this object represent EntitySets (list items) and can be queried using LINQ syntax. Just as in LINQ for SharePoint, the execution of the query is deferred until the query variable is used. And like LINQ for SharePoint, not all LINQ operations translate directly to the appropriate native query construct.
Another primary difference between ApiData and ListData is the types that are returned from the operations. While ApiData returns standard SharePoint types, ListData will return custom types that mimic the Content Types defined on the Lists. And, the lists that exist in the site are exposed - even hidden lists that would not typically be queried (e.g. TaxonomyHiddenList).
The proxy contains classes that represent the CSOM objects (Site, Web, User, etc.) in the SPProxy.SP namespace. In addition, classes are generated that model the content types found in the site. (These data-specific classes are in the SPProxy.SP.Data namespace.) This is very similar to the entity class generation in SPLINQ for server-side code. The classes also contain the typical Entity Framework events on field changes. For list data, the properties that represent columns are also strongly typed to the SharePoint field types. Taxonomy, User and Lookup columns are properly cast to the appropriate FieldValue classes. This is much better than SPLINQ!
The sample application is an MVC4 application written using Visual Studio 2012 against an on-premises SharePoint farm that has been updated with the April 2013 CU. The home page is shown in Figure 8.
Figure 8 - Sample Application Home page
The experience in managed code using the WCF library is similar to using the Microsoft.SharePoint.Client (CSOM) API. We have strongly typed objects, but the communication back to SharePoint needs to be managed, including specifying the objects and properties that we wish to read/write.
Exploring the ApiData Context class
Item 1 on the home page is a link to show site and list properties as "Native SharePoint Data." The application uses the ApiData context to retrieve this data into strongly-typed variables. The Site and List Properties page will query for the site title and description, and then query for all the non hidden lists in the site. The queries can be executed in the browser using the following:
http://wcm.wingtip.com/sites/products/_api/web/lists?$filter=Hidden eq false
The managed code equivalent of these requests is demonstrated in Figure 9. The Execute() method of the context class can be passed a relative Uri that specifies the resource address plus parameters. Line 20 will retrieve the site properties (SP.Web) and line 24 will retrieve the collection of lists (SP.List) in the site.
Figure 9 - Managed code to query for site and list properties
The two queries could be combined using the $expand parameter in native REST syntax. However, the ApiData context object does not put the collection of lists into the Lists property of the web object. Inspecting the raw XML returned from the combined query (Figure 10) shows that the lists are included as an inline
Figure 10 - Raw results of combine site and list query
Exploring the ListData Context class
Item 2 on the sample application home page is a link to show list data. Reading list data via REST/WCF Data Services is very similar to the LINQ to SharePoint experience in server-side code.
The data context has properties that represent the lists in the site. (Remember, the data context object is created when the service reference is added to the project. You need to ensure that the production site schema matches the site you used when setting the reference.) The properties that represent the lists are of type System.Data.Services.Client.DataServiceQuery
Figure 11 - Managed code to query for list data
The ContosoProducts List contains many columns, and the ListData context pull them all down. At first glance (Figure 12) there is hope that all the columns have rich data types. Look at the ProductCatalogItemCategory column. That column is a Managed Metadata column, and the data type in the data class is SP.TaxonomyFieldValue. However, the Label property of that class is set to the WSSID property - not exactly what we want!
Figure 12 - ContosoProducts list item data class as returned from REST service
The Author (Created By) and Editor (Modified By) fields, which are of type "Person or Group" in SharePoint, also suffer from this flattened structure. So, while the April 2013 CU brings an update to the REST API with the $metadata endpoint, the REST API still has its deficiencies in data handling.
The ListData context class derives from DataServiceContext, just as ApiData does. So the Execute
DataService Context operations
As noted earlier, the DataServiceContext class offers an Execute
In the sample application, the Native SharePoint Objects page show the title of the web and a grid of non-hidden lists. The page provides a link to edit the description of a list. When the list description edit page is display, the list data must be retrieved from SharePoint. Referring to the documentation on MSDN, the Uri to retrieve a list is as follows:
When rendering the grid of lists, the Native SharePoint Objects page is adding the Id of each list (which is a GUID) as a parameter on the link. Figure 13 shows the View code to accomplish the rendering of the grid. Line 30 renders the link with the list Id as a parameter (route values).
Figure 13 - MVC View to render grid of lists, including the list Id as a parameter on the edit link
In the corresponding controller action (Figure 14), the DataServiceContext object (in this case ApiData) is used to execute a request using the getById method. Line 30 defines the relative REST Uri (the getById method). Line 31 declares a parameter that will be added to the resulting Uri when executed. The parameter is passed to the Execute method (Line 33).
Figure 14 - MVC Controller Action demonstrating an operation with parameters
The DataServiceContext Class (named apiData) will append the parameter name and value to the Uri that we provide. For the ConstosoProducts list in our sample site, the resulting Uri that is called is the following:
While this Uri is not identical to the documentation, it does match one of the FunctionImport elements in the metadata document:
Figure 15 - Metadata document snippet showing the FunctionImport for the GetById method
Since the metadata document defines the return type of the call, and that type matches the type declaration of the Execute method (Figure 14, line 33), the resulting data is automatically cast to that type.
Default Behavior for Updating data
A very frequently-used capability of a DataServiceContext class (such as our ListData class) is to create, update and delete data (known as CRUD operations). The DataServiceContext class is written to track the state of each entity that is read from the remote service, provided that our code uses methods of the context to specify updates. Figure 16 shows one of those methods: UpdateObject. The other methods are AddObject and DeleteObject. (The generated context classes will often have AddToXXX methods, where XXX is the name of a type that the context provides. Those methods can be called instead of the AddObject method. In addition, the types typically have a static Create method to facilitate creating new objects.) Lastly, the DataServiceContext class has a method named SaveChanges that will gather all modified entities and perform the service operation requested. The context will automatically map these data operations to their HTTP method equivalent (e.g. Create = POST, Update = MERGE).
Figure 16 - Typical DataServiceContext code to update an entity
Adjusting the data update behavior
However, when calling a SharePoint REST endpoint, the code in Figure 16 will not work. SharePoint's REST interface works a bit differently. MSDN's "How to: Complete basic operations using SharePoint 2013 REST endpoints" document explains the requirements for SharePoint CRUD operations.
One key difference, however, is that you use a POST request. When you're updating entities, you also pass a PUT or MERGE HTTP request method by adding one of those terms to the headers of your request as the value of the X-HTTP-Method key.
This "key difference" is known as POST Tunneling and is enabled on the context via the DataServiceContext.UseTunneling property.
Even after enabling POST tunneling, the above update still fails. SharePoint returns an ArgumentNullException, indicating that the ServerRelativeUrl parameter cannot be null. Since the SP.List object being updated does not have a property with that name, this message is quite a puzzler. Digging further into the request being sent to SharePoint, the payload (request body) contains the entire SP.List entity. The OData semantics in all of the MSDN examples indicate that only the changed data should be sent on a MERGE operation. Fortunately the WCF Data Services Client allows "hooks" into the Request pipeline that allow us to modify the payload before it is sent. A blog post on the WCF Data Services team blog has all of the details: Using the new client hooks in WCF Data Services Client.
Lastly, the request headers need to include the FormDigest in order to update SharePoint. Again, this is demonstrated in the "How to" document referenced earlier. The RequestBuilding event of the DataServiceContext provides the ability to add headers to the request. (This would also be the extensibility point at which an Access Token would be passed, if the code is running in the context of a SharePoint provider-hosted app.)
The inclusion of the metadata endpoint in SharePoint's REST API allows the WCF Data Services Client library to provide definite benefits:
- DataServiceContext classes that handle the communication back to the service
- Strongly-typed objects that represent the SharePoint API and the content types and lists in the target site
- Appropriate extensibility points to handle the inconsistencies in SharePoint's OData implementation
These benefits outweigh the drawback present in the update payload issue. A client application is unlikely to provide a facility to change every property in a SharePoint object, so the need to specify which properties to include (or remove) from the payload is a tolerable requirement.