FTC to CAM – Advance HTTP remote operations for SP2013

SharePoint remote APIs are really powerful and can be used for modifying broad spectrum of the different options in site or in list level. There are however still some limitations compared to the functionalities which have been available using server side APIs. These limitations are being addressed gradually in the client side object model or in other remote techniques, but as a short term workaround we can quite often used so called HTTP remote operations patterns to achieve the needed functionality.

These needed functionalities could some something like following as a short example list

• Enable additional languages in site level
• Adjust audit logging capabilities in site collection level
• Adjust advance list settings
• Enable site request access functionality for some email address
• Control regional settings in site level
• Activate Sandbox solutions

DISCLAIMER: Microsoft does NOT recommend to use this pattern, since code can get broken with any future Office365 updates which will impact the html dom structure. This pattern is only to be used when native remote operation is not available and you have business critical need for required action. You should consider this route only when you are willing to maintain the code or when the operation is only for on-time execution. Microsoft will not take any responsibility of the updates to the Office365, which could break up this pattern. 

Update on 12th of June 2015 - Cookie name for authentication has been changed from FedAuth to SPOIDCRL. This change has to be done for the code to make provided code sample to work again. Notice that many of the settings mentioned above (like regional settings or language settings) are currently natively supported with Office 365 CSOM. See following blog post for more details: Latest API updates in Client Side Object Model (March 2015 CU for SP2013).

Introduction to advance HTTP remote operations

HTTP remote operations pattern is based on the model that we mimic what end users are doing while they operate SharePoint using broser. This means that we mimic following events using HttpWebRequest and HttpWebResponse objects.

1. Authenticated users requests the page from SharePoint
2. User modifies html controls in the browser
3. Modified set of properties are posted to server using HTTP Post operations

This means that we’ll need to first take care of the authentication, which is easy and straight forward within on-premises or with Office365-Dedicated, but does require additional considerations with Office365. In on-premises, we can simply use either default credentials of the process or impersonate with Network credentials as follows.

  1: HttpWebRequest request = (HttpWebRequest)WebRequest.Create(this.TargetSiteUrl);
  2: NetworkCredential credential = new NetworkCredential(this.User, this.Password, this.Domain);
  3: CredentialCache credentialCache = new CredentialCache();
  4: credentialCache.Add(new Uri(this.TargetSiteUrl), "NTLM", credential);
  5: request.Credentials = credentialCache;

With Office365, we’ll however need to do authentication slightly differently and to delegate authentication for the HttpWebRequest, we’ll need to set the FedAuth cookie for the http request. Here’s the reference code for doing that. In provided code example this is done automatically for the Office365 authentication style.

  1: // Convert password to secure string and create MSO Creds
  2: HttpWebRequest request = (HttpWebRequest)WebRequest.Create(this.TargetSiteUrl);
  3: var spoPassword = new SecureString();
  4: foreach (char c in this.Password)
  5: {
  6:     spoPassword.AppendChar(c);
  7: }
  8: SharePointOnlineCredentials Credentials = new SharePointOnlineCredentials(this.User, spoPassword);
  9: Uri tenantUrlUri = new Uri(this.TargetSiteUrl);
  10: string authCookieValue = Credentials.GetAuthenticationCookie(tenantUrlUri);
  11: // Create fed auth Cookie and set that to http request properly to access Office365 site
  12: Cookie fedAuth = new Cookie()
  13: {
  14:     Name = "FedAuth",
  15:     Value = authCookieValue.TrimStart("SPOIDCRL=".ToCharArray()), 
  16:     Path = "/",
  17:     Secure = true,
  18:     HttpOnly = true,
  19:     Domain = new Uri(this.TargetSiteUrl).Host
  20: };
  21: // Hookup authentication cookie to request
  22: request.CookieContainer = new CookieContainer();
  23: request.CookieContainer.Add(fedAuth);

After we have done the initial request of the page, we’ll need to store the request cookie view state and view validation values for posting and gather other elements which are posted back to server for adjusting the configuration. Following shows what elements are required for enabling access request.

  1: public override void SetPostVariables()
  2: {
  3:     // Set operation specific parameters
  4:     this.PostParameters.Add("__EVENTTARGET", "ctl00$PlaceHolderMain$ctl01$RptControls$btnSubmit");
  5:     this.PostParameters.Add("ctl00%24PlaceHolderMain%24ctl00%24txtEmail", this.EmailAddresses);
  6:     if (this.EnableAccessRequests)
  7:     {
  8:         // Present if we enable access request.
  9:         this.PostParameters.Add("ctl00%24PlaceHolderMain%24ctl00%24chkRequestAccess", "on");
  10:     }
  11: }

After the post parameters are collected, we can simply post in the information back to server in proper format. For additional details, please have a closer look on the code example and referenced video.

Video of the demo

Following video walks through the code and explains how provided example was implemented. Notice that used code in the demo is also available for download from following link.

• Code example used in video – SharePoint 2013 - AdvanceHttpRemoteCommands.zip – SkyDrive

Video will be soon available for download from my SkyDrive if needed for offline purposes.

Comments

• Anonymous
  February 02, 2014
  Hi, There seems to be a major problem using this approach in Auto-hosted apps. There is a dll file used in the creation of SharePointOnlineCredentials that is missing on Azure. Similar to this problem: stackoverflow.com/.../sharepoint-azure-msoidclil-dll-missing

• Anonymous
  February 06, 2014
  Hi Jesper, that could be indeed the case. I'm personally using always provider hosted pattern, since autohosted have some other challenges and can be still considered to be in beta status. I don't unfortunately enough experience on auto hosted to comment or help in detail.

• Anonymous
  April 01, 2014
  Hi Jesper, this is a great help if you want to convert calls straight into code. www.chadsowald.com/.../fiddler-extension-request-to-code

• Anonymous
  May 09, 2014
  Hello Vesku, I am extending your Remote provisioning App example for creating sites in O365 and want to provide user with functionality to select the locale while site creation. Since Post Pattern is only available option, can I somehow set the request.CookieContainer from the context I get from App, instead of giving user name and password explicitly?

• Anonymous
  May 11, 2014
  Hi Nanddeep, you can access the cookie container by using adding an delegate to ExecutingWebRequest event as follows. context.ExecutingWebRequest += delegate(object sender, WebRequestEventArgs e) {    e.WebRequestExecutor.WebRequest.CookieContainer = Cookies; }; This blog post is using the same model and above lines are directly copy from it - nicolaspeters.wordpress.com/.../using-csom-in-a-windows-8-app-for-sharepoint-online

• Anonymous
  May 12, 2014
  Hi Vesku, In continue with above query from Nanddeep, you provided solution for setting cookie. but here again for setting the cookie we require username and password. We are trying to use same credentials as logged in user without asking user to enter credential as we are using windows authentication. Is is possible to pass in the logged in user's credentials to o365 without having to popup a browser explicitly? Thanks in advance! Keval

• Anonymous
  May 12, 2014
  Hi K Solanki, understandable request, but have not looked to that personally too detailed. Theoretically you could use old Win32 APIs to access cookies, but that could be also blocked due cross process boundaries. I'm referring to low level access to Cookies using APIs which are mentioned for example in following page - www.codeproject.com/.../Retrieve-HttpOnly-Session-Cookie-in-WebBrowser

• Anonymous
  September 24, 2014
  Hello Vesku, One question that we are trying to create root site collection of O365 using post pattern but seems like the code breaks at getting authCookieValue "string authCookieValue = credentials.GetAuthenticationCookie(tenantUrlUri);"   this is coming as null until root site is already we set. Do you know any workaround or any other way to provision root site of O365? Thanks. //Keval

• Anonymous
  September 29, 2014
  Hi K Solanki, If you are talkig about deleting first the root site and then recreating that using the admin tools, that's something what I've not automated in this level, so unfortunately have not really got detail son how to address this issue. I would never actually delete the root site, if possible, rather update based on the business requirements with the feature activations etc. using remote provisioning, so that you'd not need http post model in first place.

• Anonymous
  December 18, 2014
  Hi Vesku,  Thanks for the article.  Is it possible to do this without explicitly using a username and password?  I have been following your provider hosted Site Provisioning solution for Office 365 where we get the context using the app's clientid and secret.  Can we integrate the example you have here using the same approach?  Thanks.

• Anonymous
  January 14, 2015
  HI Aesop, Sorry for missing your question. I would strongly recommend to use the Office 365 Developer Patterns and Practices group at the Office 365 network for any of these kind of questions. There's 1500 members already in that group with active support for these kind of questions.

• aka.ms/officedevpnpyammer I have not worked or tested this with the context model. We not also support the usage of this in production due possible issues with future changes in the page structure. I'm though interested to know what is the API what you are missing, so that we can ensure that it will be in our radar.

• Anonymous
  February 09, 2015
  Hi Vesa, Seems the authentication cookies generated for O365 environment is not getting right along with request. I get response for the GetRequest() method as a login page. Anything may go wrong in request?

• Anonymous
  February 10, 2015
  Hi Shankar, there has been small changes on the cookie handling. Previous we recommended SPOIDCRL cookie to be renamed as FedAuth, bt that is not needed no longer. We are working on updated documentation for this, but I won't be updating this http post example, since this is not really the preferred option.

• Anonymous
  February 10, 2015
  Hi Vesa, Thanks for the updates, Now we started relying on this http://spohelper.codeplex.com/ library to get authentication cookie, it works here. But please update the post if you know other simple way of getting the cookie like what you have already been demonstrated in this post.

• Anonymous
  June 11, 2015
  0: string authCookieValue = Credentials.GetAuthenticationCookie(tenantUrlUri); //remove all the lines 22: request.CookieContainer = new CookieContainer(); 23: request.CookieContainer.Add(authCookieValue );

• Anonymous
  June 17, 2015
  Thanks Vesa, This approach provides us some way out when we need such functionalities which are not exposed through client side APIs. I understand this is not recommended but at least a working solution. Thanks a lot for mentioning the change on 12th June. Just thought to put the changed code for future use. Change is in ModifyRequestBasedOnAuthPattern method of RemoteOperation class.   Cookie fedAuth = new Cookie()                    {                        Name = " FedAuth",                        Value = authCookieValue.TrimStart("SPOIDCRL=".ToCharArray()),                        Path = "/",                        Secure = true,                        HttpOnly = true,                        Domain = new Uri(this.TargetSiteUrl).Host                    }; To Cookie fedAuth = new Cookie()                    {                        Name = "SPOIDCRL",                        Value = authCookieValue.TrimStart("SPOIDCRL=".ToCharArray()),                        Path = "/",                        Secure = true,                        HttpOnly = true,                        Domain = new Uri(this.TargetSiteUrl).Host                    };

• Anonymous
  July 07, 2015
  Hello Vesa, I have a question in regard of SharePoint Online Authentication, is it available to use SharePoint App Token for the http request, because CSOM has limiation, so we need to access the SharePoint Online with http, but we don't know the tenant name and password, only the app token, is it possible to do this? Appreciated for your help. Best Regards, Long

• Anonymous
  August 03, 2015
  Hi Long, you can get access token for registered app using app id and app secret, if they have been pre-registered. Model is pretty similar as if you'd be using RESTful interfaces on calling SPO. Check following controller code for details from one of the Office 365 training packages: github.com/.../HomeController.cs If you have any other questions or comments, would suggest to use O365 Dev PnP Yammer group at aka.ms/OfficeDevPnPYammer, so that you get faster support and others in the community can follow up more easily on the discussions. Thx.