SharePoint 2010 Calendar Item Error – Item Does Not exist. It may have been deleted by another user

Today, I encountered an interesting issue with SharePoint calendar list items. My customer had created a recurring calendar entry using Outlook and then subsequently deleted the series in Outlook. Somewhere along the timeline they may have updated specific instances in the series. After deleting the series my customer noticed future events in the recurrence series remaining in the SharePoint Calendar list. Any attempt to delete these list items via the UI or PowerShell resulted in the error “Item Does Not exist. It may have been deleted by another user“.

SharePoint handles recurring events by creating child list items for every event in the series and tying them to one master list item – stored in the MasterSeriesItemID  field. The master list item is not shown via the UI, since users see the events in the series, especially when the series has exceptions.

Deleting the master list item with ID specified in MasterSeriesItemID  resulted in deleting the entire set of recurring events in the series, which was the desired outcome. I’m guessing Outlook should have deleted the series master item in SharePoint when the user deleted the series in Outlook, but that clearly didn’t happen.

SharePoint High-Trust App Only Context from PowerShell

Wictor Wilén wrote a great post on getting an “app-only” client context for SharePoint Add-ins using ACS – Low Trust Add-ins. I wanted to do something similar with High-Trust Server-2-Server Provider-Hosted Add-ins. I needed the app-only context so that I could use the Tenant API to create on-premises site collections from PowerShell. With the help of Fiddler and some mad PowerShell skills, I came up with the following function.

Function GetAppOnlyContext {
    Param([string]$clientId, [string]$issuerId, [string]$certPath, [string]$certPwd, [scriptblock]$GetAppOnlyContextCB);
    [System.Reflection.Assembly]::LoadWithPartialName("System") | Out-Null;
    [System.Reflection.Assembly]::LoadWithPartialName("System.Collections") | Out-Null;
    [System.Reflection.Assembly]::Load("System.IdentityModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089") | Out-Null;
    [System.Reflection.Assembly]::Load("Microsoft.IdentityModel, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35") | Out-Null;
    [System.Reflection.Assembly]::LoadWithPartialName("Microsoft.IdentityModel.Extensions") | Out-Null;
    # Get the realm.
    $realm = "";
    $headers = @{Authorization = "Bearer "};
    Try {
        $x = Invoke-WebRequest -Uri "$($tenantUrl)_vti_bin/client.svc" -Headers $headers -Method POST -UseBasicParsing;
    } Catch {
        # We expect a 401 here.
        $realm = $_.Exception.Response.Headers["WWW-Authenticate"].Substring(7).Split(",")[0].Split("=")[1].Trim("`"");
    }
    $issuer = "$($issuerId)@$($realm)";
    $nameId = "$($clientId)@$($realm)";
    $uri = New-Object System.Uri($tenantUrl);
    $audience = "00000003-0000-0ff1-ce00-000000000000/$($uri.Authority)@$($realm)";
    # Get the claims
    $actorClaims = New-Object System.Collections.Generic.List[Microsoft.IdentityModel.S2S.Tokens.JsonWebTokenClaim];
    $claim = New-Object Microsoft.IdentityModel.S2S.Tokens.JsonWebTokenClaim("nameid", $nameId);
    $actorClaims.Add($claim);
    # Get the signing credentials
    $cert = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2($certPath, $certPwd);
    $signingCreds = New-Object Microsoft.IdentityModel.SecurityTokenService.X509SigningCredentials(`
        $cert, [System.IdentityModel.Tokens.SecurityAlgorithms]::RsaSha256Signature, `
        [System.IdentityModel.Tokens.SecurityAlgorithms]::Sha256Digest);
    # Create the token.    
    $token = New-Object Microsoft.IdentityModel.S2S.Tokens.JsonWebSecurityToken(`
        $issuer, $audience, [System.DateTime]::UtcNow, [System.DateTime]::UtcNow.AddHours(12), $actorClaims, $signingCreds);
    $tokenString = (New-Object Microsoft.IdentityModel.S2S.Tokens.JsonWebSecurityTokenHandler).WriteTokenAsString($token);
    [Microsoft.SharePoint.Client.ClientContext]$clientContext = $null;
    Try {
        # Create the client context.
        $clientContext = New-Object Microsoft.SharePoint.Client.ClientContext($tenantUrl);
        $clientContext.AuthenticationMode = [Microsoft.SharePoint.Client.ClientAuthenticationMode]::Anonymous;
        $clientContext.FormDigestHandlingEnabled = $false;
        AddRequestHandler -clientContext $clientContext -token $tokenString;
        if ($GetAppOnlyContextCB -ne $null) { &$GetAppOnlyContextCB -clientContext $clientContext; }
    } Finally {
        if ($clientContext -ne $null) { $clientContext.Dispose(); }
    }
}

Function AddRequestHandler {
    Param([Microsoft.SharePoint.Client.ClientContext]$clientContext, [string]$token);
    Add-Type -TypeDefinition @"
using System;
using Microsoft.SharePoint.Client;
namespace SPHelper {
    public static class ClientContextHelper {
        private static string _token = "";
        public static void AddRequestHandler(ClientContext ctx, string token) {
            _token = token;
            ctx.ExecutingWebRequest += new EventHandler<WebRequestEventArgs>(RequestHandler);
        }
        private static void RequestHandler(object sender, WebRequestEventArgs e) {
            e.WebRequestExecutor.RequestHeaders["Authorization"] = "Bearer " + _token;
        }
    }
}
"@ -ReferencedAssemblies "$env:dp0\DLLs\Microsoft.SharePoint.Client.dll", "$env:dp0\DLLs\Microsoft.SharePoint.Client.Runtime.dll";
    [SPHelper.ClientContextHelper]::AddRequestHandler($clientContext, $token);
}

SharePoint Online AppRegNew via PowerShell

$clientId = "clientId GUID"
$appDomain = "App Domain"
$appName = "Friendly Name"
$appUrl = "App Url"
$newClientSecret = "Client Secret"

$servicePrincipalName = @("$clientID/$appDomain")
New-MsolServicePrincipal -ServicePrincipalNames $servicePrincipalName -AppPrincipalId $clientID -DisplayName $appName `
  -Type Symmetric -Usage Verify -StartDate "12/01/2016" -EndDate "12/01/2017" -Addresses (New-MsolServicePrincipalAddresses -Address $appUrl) 
New-MsolServicePrincipalCredential -AppPrincipalId $clientId -Type Symmetric -Usage Sign -Value $newClientSecret
New-MsolServicePrincipalCredential -AppPrincipalId $clientId -Type Symmetric -Usage Verify -Value $newClientSecret
New-MsolServicePrincipalCredential -AppPrincipalId $clientId -Type Password -Usage Verify -Value $newClientSecret

NodeJS Development Environment w/SharePoint Framework Support

Unless you’ve been hiding under a rock recently (or not interested in SharePoint development), you’ve probably heard/read the recent announcement about the preview release of the SharePoint Framework. You can read the announcement here:

http://dev.office.com/blogs/sharepoint-framework-developer-preview-release

The new SPFx adopts client-side development using Typescript (a superset language of JavaScript) and uses tools born from NodeJS development – such as Gulp and Yeoman. The aim of this post is not to go into the specifics of these tools, besides, there’s lots of information on the Internet.

If you’ve made the leap into client-side development (for SharePoint or otherwise) – congratulations and welcome to the new era of software development. Those of you embarking on the learning curve will soon learn that client-side development (and by extension SPFx development) requires installation of various tools for your development arsenal. The days of just installing a single IDE are fading away. At this point I shall mention that those die hard Visual Studio folks can develop NodeJS and SPFx projects with their IDE. However, you still need NodeJS and dependent modules installed to develop for SPFX. The following article details the steps:

https://github.com/SharePoint/sp-dev-docs/wiki

Like many JavaScript and Typescript developers before me, I have opted for the platform independent tools, using Visual Studio Code. VSCode is a lightweight code editor that embraces client-side and NodeJS development and runs on Windows, OSX and Linux (you can even run it on a Raspberry PI). Just as with it’s big brother, Visual Studio Code works with additional software to constitute a true development environment – alone it’s really just a JavaScript/Typescript editor. I’ll refer you to the previously mentioned article that speaks to installing all the necessary components for SPFx development.

By now, you’re probably thinking “I have to install Visual Studio Code or Visual Studio 2015, NodeJS, Yeoman, Gulp, Windows Build Tools, yada, yada, yada, just to get a development environment up and running?”. The short answer is “yes”. Luckily for you (those of you on Windows at least), I have created a PowerShell script that downloads all the tools and dependencies for you, available at the following location:

https://github.com/robgarrett/Study/blob/master/Install-JSDev.ps1

My script downloads the following binaries and installs them:

  • Visual Studio Code
  • NodeJS LTS
  • Windows GIT

After installation of the binaries, the script uses the Node Package Manager (NPM) to install:

  • Windows-Build-Tools (includes an installation of Python)
  • Yeoman
  • Gulp
  • The SPFx Yeoman Generator (this creates SPFx scaffolding)
  • Typescript 2.0

Typescript 2.0 isn’t necessarily required for SPFx development (I believe TSv1.0 installed as a dependency of one of the other packages), but Typescript is coming to stay so might as well get used to the next version.

Depending on the performance of your development machine and your Internet, the script can take some time installing all the necessary packages. So grab a coffee and let it do its stuff.

Finally, you’re ready to start developing. If you’re ready to dive into SPFx Web Part development you can create your first web-part using the instructions at the following location:

https://github.com/robgarrett/Study/blob/master/Install-JSDev.ps1

FYI – the @Microsoft/Generator-SharePoint downloads a ton of modules and it can take an absolute age. It might seem like a lot of waiting around to develop your first SPFx web part, but SPFx and the workbench rely on lots of modules. For subsequent projects you can always make a copy of the default web-part scaffolding (directory structure and files) save generating from scratch. At the very least the node_modules folder is good to keep because it contains all dependent NodeJS projects and libraries.

So, that’s it. Sorry, if you’re on OSX or Linux – you’ll have to download to the installs per the article instructions until I or someone else creates a bash script to do the same as my PowerShell script (note: PowerShell now runs on Linux, but my script is Windows specific). Hey, at least you seldom set up your development environment from scratch.

SharePoint 2013 Not Crawling Cold Fusion (CFM) Pages

It’s not every day that you find needle in a haystack, but when you do it’s worth blogging for prosperity…

My customer has set up SharePoint 2013 as the central search authority for their organization and uses it to crawl non-SharePoint sites as well as SharePoint. We noticed that SharePoint was not crawling links in Cold Fusion pages (.cfm). Instead, the crawler was treating CFM pages as text and stopping at the top-most levels without indexing lower-level pages.

We noticed in testing that renaming CFM to HTML extension fixed the issue, but wasn’t sustainable for the vast number of CF sites in the organization.

For the longest time I was messing around with the New-SPEnterpriseSearchFileFormat cmdlet, urging SharePoint to treat CFM pages the same as HTML. What I determined is this cmdlet is good for mapping custom extensions to Windows platform IFilters. What I wanted to do was to mimic the crawler indexing HTML pages, which does not use Windows IFilters. After much perseverance, I found the following information (included below, incase the link goes away):

# To check the current settings for filtering extensions, run the following command lines:
$ssa = Get-SPEnterpriseSearchServiceApplication "Search Service Application"
$ssa.GetProperty("ExtensionsToFilter")
# Here's the default output that you'll receive:
#;ascx;asp;aspx;htm;html;jhtml;jsp;mht;php;

#To add the .cfm extension to the property, run the following commands:
$ssa.SetProperty("ExtensionsToFilter", ";ascx;asp;aspx;htm;html;jhtml;jsp;mht;php;cfm;")
$ssa.Update()
# To restart the search functionality on a crawler when no crawling is occurring, run the following commands:
net stop osearch15
net start osearch15

https://support.microsoft.com/en-us/kb/2953907

SharePoint Crawling User Profiles (SPS3://) – Access Denied w/o HTTP

I stumbled across an interesting issue with People Search in SharePoint 2016. I was attempting to crawl the user profile store with URL: sps3://server-name and getting Access Denied in the crawl log. I checked the Administrators for the User Profile Service in Manage Service Applications and confirmed my default content access account (crawl account) had access to Retrieve People Data for Search Crawlers (see here).

Looking at the ULS I noticed errors about missing Alternate Access Mappings for an HTTP address, before seeing the Access Denied error. This caught my eye because I’ve configured my collaboration web application and my-site host as HTTPS.
For kicks, I added an IIS binding for HTTP://SERVER-NAME and added an AAM for the server name on HTTP, alongside my HTTPS FQDN. Lo-and-behold, after starting a full crawl the log reported successes for people data.

So, it appears that SharePoint takes the URL sps3://server-name and converts it to http://server-name to make some determination of access to the User Profile store. I’m not sure why this is the case (not yet anyway).

Lesson learned (for now): make sure SharePoint’s default content access account can access the same domain URL on HTTP as that of the SPS3 protocol. As mentioned at the top of this post, I found this out on SharePoint 2016, and I need to test to see if the results are the same on SharePoint 2013.

[Update 5/13/2016]: Turns out I should read the TechNet articles carefully. The following article indicates using sps3s://mysite-url, which then works correctly.

https://technet.microsoft.com/en-us/library/hh582311.aspx?f=255&MSPPError=-2147217396

Deploying a SharePoint Add-In from the Catalog via PowerShell

I recently came across a situation where I was asked if I could deploy a SharePoint App/Add-in from the corporate catalog to a sub-site, via PowerShell. With no surprise, there’s no single PowerShell Cmdlet that will perform this task. So, I took it upon myself to reverse engineer the SharePoint storefront and see how Microsoft does it within the platform.

The following code relies on .NET Reflection to invoke private and internal methods in the SharePoint server-side APIs. For this reason, I recommend taking caution in using this code, because we’re calling methods that Microsoft never intended developers to access. I highly recommend keeping this code away from production.

The code assumes the presence of custom add-ins/apps in the catalog and will iterate them. With each add-in/app, you have the option to add the app to the root site of the given site collection. You could easily change this code to suit your purpose. Note: if an app is already installed for a given web, it’ll not show up in the iteration.

[CmdletBinding()]param();

if ((Get-PSSnapin -Name "Microsoft.SharePoint.PowerShell" -ErrorAction SilentlyContinue) -eq $null) {
    Add-PSSnapin "Microsoft.SharePoint.PowerShell";
}

$yes = New-Object System.Management.Automation.Host.ChoiceDescription "&Yes","Description."
$no = New-Object System.Management.Automation.Host.ChoiceDescription "&No","Description."
$cancel = New-Object System.Management.Automation.Host.ChoiceDescription "&Cancel","Description."
$options = [System.Management.Automation.Host.ChoiceDescription[]]($yes, $no, $cancel)

$url = "site collection URL here";
$site = Get-SPSite $url;
$web = $site.RootWeb;

Write-Verbose "Getting apps from the catalog";
$json = Invoke-RestMethod -UseDefaultCredentials -Method Get -Uri $url + "/_layouts/15/addanapp.aspx?task=GetMyApps&sort=1&query=&myappscatalog=0&ci=1&vd=1";
$json | ? { $_.Catalog -eq 1 } | % {
    $appId = $_.ID;

    Write-Host -foreground Yellow "Title: $($_.Title)";
    Write-Host -foreground Yellow "AppID: $appId";

    $result = $host.ui.PromptForChoice("App Install", "Install App $($_.Title)", $options, 1)
    if ($result -eq 2) { break; }
    if ($result -eq 0) {

        Write-Verbose "Get the Corporate Catalog Accessor instance";
        $flags = [System.Reflection.BindingFlags]::NonPublic -bor [System.Reflection.BindingFlags]::Instance;
        $asm = [System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint");
        $ccaType = $asm.GetType("Microsoft.SharePoint.Marketplace.CorporateCuratedGallery.SPCorporateCatalogAccessor");
        $ccaCtor = $ccaType.GetConstructors($flags) | ? { $_.GetParameters().Count -eq 1; }
        $cca = $ccaCtor.Invoke(@($web));

        Write-Verbose "Getting App Package from the Catalog";
        $method = $ccaType.GetMethods($flags) | ? { $_.Name -ilike "GetAppPackage" -and ($_.GetParameters())[0].ParameterType.Name -eq "String" }
        $stream = $method.Invoke($cca, @($appId));

        Write-Verbose "Installing App from Catalog";
        $spAppType = $asm.GetType("Microsoft.SharePoint.Administration.SPApp");
        $method = $spAppType.GetMethod("CreateAppUsingPackageMetadata", [System.Reflection.BindingFlags]::NonPublic -bor [System.Reflection.BindingFlags]::Static);
        [Microsoft.SharePoint.Administration.SPApp]$spApp = $method.Invoke($null, @($stream, $web, 2, $false, $null, $null));
        $appInstanceId = $spApp.CreateAppInstance($web);
        Write-Host -ForegroundColor Yellow "AppInstanceID: $appInstanceId";
        $appInstance = [Microsoft.SharePoint.Administration.SPAppCatalog]::GetAppInstance($web, $appInstanceId);
        $appInstance.Install();
    }
}

SharePoint 2013 Build Numbers and PowerShell

If you’re in the business of maintaining SharePoint 2013 on-premises, you’ve undoubtedly come across Todd Klindt’s blog post with ongoing table of build numbers and corresponding CU names: http://www.toddklindt.com/sp2013builds.

Knowing the current patch version of your farm is pretty straight forward. You can look up the farm build number in Central Administration -> Manage Servers in Farm, and grab the number at the top of the page. Cross reference this number with the table in Todd’s blog post and you have the CU version installed in your farm.

I wanted to go a step further and write a PowerShell script that pulls the build number and looks up the details from Todd’s blog post automagically. Here it is:

[CmdletBinding()]Param();

$global:srcWebPage = "http://www.toddklindt.com/sp2013builds"; # Thanks Todd.

if ((Get-PSSnapin -Name "Microsoft.SharePoint.PowerShell" -ErrorAction SilentlyContinue) -eq $null) {
    Add-PSSnapin "Microsoft.SharePoint.PowerShell";
}

try {
    $farm = Get-SPFarm;
    $buildVersion = $farm.BuildVersion;
    $buildVersionString = $buildVersion.ToString();
    $site = Invoke-WebRequest -UseBasicParsing -Uri $global:srcWebPage;
    $pattern = "\<td.*\>.+(" + $buildVersionString.Replace(".", "\.") + ").*\</td\>\s*\<td.*\>(.+)\</td\>\s*\<td.*\>(.+)\</td\>";
    $pattern += '\s*\<td.*\>.*\<a.+href="(.+)".*\>(.+)\</a\>\</td\>';
    $pattern += '\s*\<td.*\>.*\<a.+href="(.+)".*\>(.+)\</a\>\</td\>';
    Write-Verbose $pattern;
    $m = [Regex]::Match($site.RawContent, $pattern, [System.Text.RegularExpressions.RegexOptions]::Multiline);
    if (!$m.Success) { throw "Could not find build number $buildVersionString in $global:srcWebPage"; }
    Write-Host -ForegroundColor white -NoNewline "Current Build Number: ";
    Write-Host -ForegroundColor yellow $buildVersionString;
    Write-Host -ForegroundColor white -NoNewline "Current Patch/CU: ";
    Write-Host -ForegroundColor yellow $m.Groups[2].Value;
    Write-Host -ForegroundColor white -NoNewline "KB of Current Patch/CU: ";
    Write-Host -ForegroundColor yellow $m.Groups[5].Value;
    Write-Host -ForegroundColor white -NoNewline "Download of Current Patch/CU: ";
    Write-Host -ForegroundColor yellow $m.Groups[6].Value;
    Write-Host
    $index = $m.Index + $m.Length;
    $pattern = "\<td.*\>.+([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+).*\</td\>\s*\<td.*\>(.+)\</td\>\s*\<td.*\>(.+)\</td\>";
    $pattern += '\s*\<td.*\>.*\<a.+href="(.+)".*\>(.+)\</a\>\</td\>';
    $pattern += '\s*\<td.*\>.*\<a.+href="(.+)".*\>(.+)\</a\>\</td\>';
    $m = [Regex]::Match($site.RawContent.Substring($index), $pattern, [System.Text.RegularExpressions.RegexOptions]::Multiline);
    if ($m.Success) {
        Write-Host -ForegroundColor white -NoNewline "Next Build Number: ";
        Write-Host -ForegroundColor green $m.Groups[1].Value;
        Write-Host -ForegroundColor white -NoNewline "Next Patch/CU: ";
        Write-Host -ForegroundColor green $m.Groups[2].Value;
        Write-Host -ForegroundColor white -NoNewline "KB of Next Patch/CU: ";
        Write-Host -ForegroundColor green $m.Groups[5].Value;
        Write-Host -ForegroundColor white -NoNewline "Download of Next Patch/CU: ";
        Write-Host -ForegroundColor green $m.Groups[6].Value;
    }

} catch {
    Write-Host -ForegroundColor Red $_.Exception;
}

PowerShell Register Provider-Hosted Add-in/App

My current client uses provider-hosted add-ins with SharePoint 2013 on-premises. We have a centralized server infrastructure – provider-host – for the add-ins, where we deploy the add-in/app logic and then deploy the APP files to different SharePoint 2013 environments.

Why? The add-ins we’ve developed use CSOM to effect changes in the environment they’re deployed (SharePoint). We have one team developing the provider-hosted add-ins, and another team testing the add-ins within their development environments. This post is not about lifecycle deployment of SharePoint Provider-Hosted Add-ins – besides, we have integration, staging, test, and production hosts for this purpose – but about a nifty PowerShell script to reuse APP files across environments.

So, the scenario goes like this…

We have an integration farm with the provider-hosted add-ins deployed (and working). Developers download these add-ins from the integration SharePoint farm app catalog and save the APP files locally. They then upload these APP files into the app catalog of their local development SharePoint farm. Each development farm has a registered Security Token Issuer, using the same issuer ID as the integration farm. The development farms also have a trusted root certificate for the High-Trust between the provider-host and SharePoint, also the same as integration. The remaining step is to ensure that each add-in deployed to the development farm has the same client/app ID as that registered in the integration farm.

The typical process to register a shared add-in would be to crack open the APP file (just a zip file), look in the manifest.xml file and pull the client ID, and then call https://site/_layouts/15/appregnew.aspx. However, I wanted a script that avoided all that nonsense, and here it is:

[CmdletBinding()]Param(
    [Parameter(Mandatory=$true)][string]$appPath,
    [Parameter(Mandatory=$true)][string]$webUrl
);

if ((Get-PSSnapin -Name "Microsoft.SharePoint.PowerShell" -ErrorAction SilentlyContinue) -eq $null) {
    Add-PSSnapin "Microsoft.SharePoint.PowerShell";
}

$zipStream = $null;
$streamReader = $null;
try {
    Write-Verbose "Looking for AppManifest in APP Zip";
    [System.Reflection.Assembly]::LoadWithPartialName('System.IO.Compression') | Out-Null;
    if (![System.IO.File]::Exists($appPath)) { throw "$appPath does not exist"; }
    $zipBytes = [System.IO.File]::ReadAllBytes($appPath);
    $zipStream = New-Object System.IO.Memorystream;
    $zipStream.Write($zipBytes, 0, $zipBytes.Length);
    $zipArchive = New-Object System.IO.Compression.ZipArchive($zipStream);
    $zipEntry = $zipArchive.GetEntry("AppManifest.xml");
    $streamReader = New-Object System.IO.StreamReader($zipEntry.Open());
    $manifest = New-Object System.Xml.XmlDocument;
    $manifest.LoadXml($streamReader.ReadToEnd());
    
    Write-Verbose "Looking for ClientID";
    $ns = New-Object System.Xml.XmlNamespaceManager($manifest.NameTable);
    $ns.AddNamespace("x", "http://schemas.microsoft.com/sharepoint/2012/app/manifest");
    $node = $manifest.SelectSingleNode("/x:App/x:AppPrincipal/x:RemoteWebApplication", $ns);
    $clientId = $node.Attributes["ClientId"].Value;
    $node = $manifest.SelectSingleNode("/x:App/x:Properties/x:Title", $ns);
    $appTitle = $node.InnerText;
    Write-Verbose "Found app with title $appTitle and clientID $clientId";
    
    Write-Verbose "Registering App ClientId with SharePoint";
    $web = Get-SPWeb $webUrl;
    $realm = Get-SPAuthenticationRealm -ServiceContext $web.Site;
    $fullAppId = $clientId + '@' + $realm;
    Register-SPAppPrincipal -DisplayName $appTitle -NameIdentifier $fullAppId -Site $web;

} catch {
    Write-Host -ForegroundColor Red $_.Exception;
} finally {
    if ($streamReader -ne $null) { $streamReader.Close(); }
    if ($zipStream -ne $null) { $zipStream.Close(); }
}

The script takes a full path to the APP file and a web URL to register the add-in. As you can see from the code, the script replicates the manual steps of unzipping the APP (in memory), pulls out the client ID and calls Register-SPAppPrincipal to register the add-in.

SharePoint Add-In Governance

I’ve been working with a client that recently asked me about Governance of SharePoint Provider-Hosted Add-ins in their on-premises SharePoint 2013. Essentially, they wanted to take control over how site owners installed add-ins from the corporate SharePoint App Catalog. The App Catalog allows administrators to toggle whether site owners can install add-ins or request permission to install them, but my client was looking for a more granular solution on a site-by-site basis.

My thought process went in the direction of detecting installation of an add-in and finding a way to intercept the installation process. This led me to the App Installed event that SharePoint supports for provider-hosted add-ins. Imagine a scenario where every add-in in the catalog fired an App Installed event that then checked the add-in against a central database/list etc., to determine whether the site owner could complete the installation. Seems simple enough. However, what if I then said administrators could upload provider-hosted add-ins to the catalog that might not have App Installed event code present? Perhaps a third-party add-in or add-in developed by another person outside the governance circle.

I wondered if it’d be possible to inject logic into existing add-ins after they’d been deployed to the app catalog. Turns out this isn’t as hard as it sounds. For those of you thinking I’m about to go with a code hack and a suggestion that could compromise compiled code, just hang in a moment longer.

Provider-Hosted Add-ins consist of an APP file in the app catalog, which redirects execution to an endpoint on another sever. The APP file is really just a glorified ZIP file with AppManifest.xml and other supporting files contained. Provider-Hosted add-ins that support the App Installed event (and Upgrading and Uninstalling events) include an endpoint reference to a remote event receiver web service in the AppManifest.xml. My theory was that I could download the APP file, unzip it, make the change to the AppManifest.xml file, zip it again, and upload it back to the catalog. The new modified APP file would contain a remote event receiver location of my choosing. So, I started work…

First to illustrate what I mean, here’s a snapshot of a test Provider-Hosted Add-in:

<?xml version="1.0" encoding="utf-8" ?>
<!--Created:cb85b80c-f585-40ff-8bfc-12ff4d0e34a9-->
<App xmlns="http://schemas.microsoft.com/sharepoint/2012/app/manifest"      Name="HelloWorld"      ProductID="{316a9436-6358-4626-a007-c31fafe306a2}"      Version="1.0.0.0"      SharePointMinVersion="15.0.0.0" >
  <Properties>
    <Title>Hello World</Title>
    <StartPage>~remoteAppUrl/Pages/Default.aspx?{StandardTokens}</StartPage>
    <InstalledEventEndpoint>~remoteAppUrl/Services/AppEventReceiver.svc</InstalledEventEndpoint>
    <UninstallingEventEndpoint>~remoteAppUrl/Services/AppEventReceiver.svc</UninstallingEventEndpoint>
    <UpgradedEventEndpoint>~remoteAppUrl/Services/AppEventReceiver.svc</UpgradedEventEndpoint>
  </Properties>

  <AppPrincipal>
    <RemoteWebApplication ClientId="*" />
  </AppPrincipal>
  <AppPermissionRequests AllowAppOnlyPolicy="false">
  </AppPermissionRequests>
</App>

Next, I needed some plumbing that would register event receivers on the App Catalog to manipulate add-ins uploaded to the catalog. In keeping with good SharePoint 2013 development, I created a Provider-Hosted Add-in for this purpose. I called this PH add-in the App Catalog Updater, ACU for short.

Because the ACU needs to make changes to the App Catalog, this add-in requires tenant control and the ability to run under the app-only context (as opposed to app and user credentials). This prevents this add-in from ever going into the Marketplace, but for my purpose this didn’t matter.

The following is the AppManifest.xml for the ACU:

<?xml version="1.0" encoding="utf-8" ?>
<!--Created:cb85b80c-f585-40ff-8bfc-12ff4d0e34a9-->
<App xmlns="http://schemas.microsoft.com/sharepoint/2012/app/manifest"      Name="SPAppsUpdateAppCat"      ProductID="{316a9436-6358-4626-a007-c31fafe306a2}"      Version="1.0.0.0"      SharePointMinVersion="15.0.0.0" >
  <Properties>
    <Title>SPApps.UpdateAppCat</Title>
    <StartPage>~remoteAppUrl/Pages/Default.aspx?{StandardTokens}</StartPage>
    <InstalledEventEndpoint>~remoteAppUrl/Services/AppEventReceiver.svc</InstalledEventEndpoint>
    <UninstallingEventEndpoint>~remoteAppUrl/Services/AppEventReceiver.svc</UninstallingEventEndpoint>
    <UpgradedEventEndpoint>~remoteAppUrl/Services/AppEventReceiver.svc</UpgradedEventEndpoint>
  </Properties>

  <AppPrincipal>
    <RemoteWebApplication ClientId="*" />
  </AppPrincipal>
  <AppPermissionRequests AllowAppOnlyPolicy="true">
    <AppPermissionRequest Scope="http://sharepoint/content/tenant" Right="FullControl" />
  </AppPermissionRequests>
</App>

As mentioned, the ACU registers remote events on the App Catalog, which I achieve, using the following code:

using System;
using Microsoft.SharePoint.Client.EventReceivers;

namespace SPApps.UpdateAppCatWeb.Services
{
    public class AppEventReceiver : IRemoteEventService
    {
        ///

<summary>
        /// Handles app events that occur after the app is installed or upgraded, or when app is being uninstalled.
        /// </summary>


        /// <param name="properties">Holds information about the app event.</param>
        /// <returns>Holds information returned from the app event.</returns>
        public SPRemoteEventResult ProcessEvent(SPRemoteEventProperties properties)
        {
            var result = new SPRemoteEventResult();
            Logger.Logger.LogInfo("ProcessEvent called for AppEventReceiver", () =>
            {
                using (var clientContext = TokenHelper.CreateAppEventClientContext(properties, false))
                {
                    switch (properties.EventType)
                    {
                        case SPRemoteEventType.AppInstalled:
                            // Remove any old RER first.
                            AppHelper.UnregisterRemoteEvents(clientContext);
                            // Install a RER for the App Catalog.
                            AppHelper.RegisterRemoteEvents(clientContext);
                            // Iterate existing apps and process them.
                            AppHelper.ProcessAppList(clientContext);
                            break;
                        case SPRemoteEventType.AppUninstalling:
                            // Remove RER from the App Catalog.
                            AppHelper.UnregisterRemoteEvents(clientContext);
                            break;
                    }
                }
            });
            return result;
        }

        ///

<summary>
        /// This method is a required placeholder, but is not used by app events.
        /// </summary>


        /// <param name="properties">Unused.</param>
        public void ProcessOneWayEvent(SPRemoteEventProperties properties)
        {
            throw new NotImplementedException();
        }

    }
}

The above code resides in the App Installed event receiver code for the ACU. This code executes when the ACU is installed the first time, and then calls a handy App Helper class to register the event receivers on the App Catalog. In addition, the code looks for existing add-ins in the catalog for processing, again using the App Helper.

Let’s take a look at the App Helper class:

using Microsoft.SharePoint.Client;
using System;
using System.Xml;
using System.Collections.Generic;
using System.Diagnostics;
using System.Globalization;
using System.IO.Compression;
using System.ServiceModel;

namespace SPApps.UpdateAppCatWeb
{
    static class AppHelper
    {
        private const string LISTNAME = "Apps for SharePoint";
        private const string RERNAME = "Apps_Remote_Event_Receiver";

        public static void RegisterRemoteEvents(ClientContext clientContext)
        {
            if (null == clientContext) throw new ArgumentNullException("clientContext");
            try
            {
                // Get the Apps List.
                Logger.Logger.LogInfo("Registering remote events", () =>
                {
                    var appCat = clientContext.Web.Lists.GetByTitle(LISTNAME);
                    clientContext.Load(clientContext.Web);
                    clientContext.ExecuteQuery();
                    // Get the operation context and remote event service URL.
                    var remoteUrl = GetServiceUrl("ListEventReceiver.svc");
                    // Add RER for Item Added.
                    if (!IsRemoteEventRegistered(clientContext, EventReceiverType.ItemAdded))
                    {
                        appCat.EventReceivers.Add(new EventReceiverDefinitionCreationInformation
                        {
                            EventType = EventReceiverType.ItemAdded,
                            ReceiverName = RERNAME,
                            ReceiverUrl = remoteUrl,
                            SequenceNumber = 10000,
                            Synchronization = EventReceiverSynchronization.Synchronous
                        });
                        clientContext.ExecuteQuery();
                    }
                    // Add RER for Item Updated
                    if (IsRemoteEventRegistered(clientContext, EventReceiverType.ItemUpdated)) return;
                    appCat.EventReceivers.Add(new EventReceiverDefinitionCreationInformation
                    {
                        EventType = EventReceiverType.ItemUpdated,
                        ReceiverName = RERNAME,
                        ReceiverUrl = remoteUrl,
                        SequenceNumber = 10001
                    });
                    clientContext.ExecuteQuery();
                });
            }
            catch (Exception ex)
            {
                Debug.WriteLine(ex.ToString());
                Logger.Logger.LogError(ex.ToString());
            }
        }

        public static bool IsRemoteEventRegistered(ClientContext clientContext, EventReceiverType type)
        {
            var result = false;
            if (null == clientContext) throw new ArgumentNullException("clientContext");
            try
            {
                // Get the list
                Logger.Logger.LogInfo("Checking if remote events registered", () =>
                {
                    var srcList = clientContext.Web.Lists.GetByTitle(LISTNAME);
                    clientContext.Load(clientContext.Web);
                    clientContext.ExecuteQuery();
                    // Iterate all event receivers.
                    clientContext.Load(srcList.EventReceivers);
                    clientContext.ExecuteQuery();
                    foreach (var er in srcList.EventReceivers)
                        if (0 == string.Compare(er.ReceiverName, RERNAME, true, CultureInfo.CurrentCulture) && er.EventType == type)
                        {
                            result = true;
                            break;
                        }
                });
                return result;
            }
            catch (Exception ex)
            {
                Debug.WriteLine(ex.ToString());
                Logger.Logger.LogError(ex.ToString());
            }
            return false;
        }

        public static void UnregisterRemoteEvents(ClientContext clientContext)
        {
            if (null == clientContext) throw new ArgumentNullException("clientContext");
            try
            {
                Logger.Logger.LogInfo("Unregistering remote events", () =>
                {
                    // Get the App Catalog.
                    var appCat = clientContext.Web.Lists.GetByTitle(LISTNAME);
                    clientContext.Load(clientContext.Web);
                    clientContext.ExecuteQuery();
                    // Remove all event receivers.
                    clientContext.Load(appCat.EventReceivers);
                    clientContext.ExecuteQuery();
                    var toDelete = new List<EventReceiverDefinition>();
                    // ReSharper disable once LoopCanBeConvertedToQuery
                    foreach (var er in appCat.EventReceivers)
                    {
                        if (er.ReceiverName == RERNAME) toDelete.Add(er);
                    }
                    foreach (var er in toDelete)
                    {
                        er.DeleteObject();
                        clientContext.ExecuteQuery();
                    }
                });
            }
            catch (Exception ex)
            {
                Debug.WriteLine(ex.ToString());
                Logger.Logger.LogError(ex.ToString());
            }
        }

        internal static void ProcessAppList(ClientContext clientContext)
        {
            if (null == clientContext) throw new ArgumentNullException("clientContext");
            try
            {
                Logger.Logger.LogInfo("Processing app catalog", () =>
                {
                    // Get the App Catalog and App List Item.
                    var appCat = clientContext.Web.Lists.GetByTitle(LISTNAME);
                    clientContext.Load(clientContext.Web);
                    clientContext.Load(appCat);
                    var query = CamlQuery.CreateAllItemsQuery();
                    var items = appCat.GetItems(query);
                    clientContext.Load(items);
                    clientContext.ExecuteQuery();
                    foreach (var item in items)
                        ProcessAppListItem(clientContext, item);
                });
            }
            catch (Exception ex)
            {
                Debug.WriteLine(ex.ToString());
                Logger.Logger.LogError(ex.ToString());
            }
        }

        internal static void ProcessAppListItem(ClientContext clientContext, int itemID)
        {
            if (null == clientContext) throw new ArgumentNullException("clientContext");
            if (itemID <= 0) throw new ArgumentOutOfRangeException("itemID");             try {                 // Get the App Catalog and App List Item.                 var appCat = clientContext.Web.Lists.GetByTitle(LISTNAME);                 clientContext.Load(clientContext.Web);                 clientContext.Load(appCat);                 var item = appCat.GetItemById(itemID);                 clientContext.Load(item);                 clientContext.ExecuteQuery();                 ProcessAppListItem(clientContext, item);             }             catch (Exception ex)             {                 Debug.WriteLine(ex.ToString());                 Logger.Logger.LogError(ex.ToString());             }         }         internal static void ProcessAppListItem(ClientContext clientContext, ListItem item)         {             if (null == clientContext) throw new ArgumentNullException("clientContext");             if (null == item) throw new ArgumentNullException("item");             try             {                 Logger.Logger.LogInfo("Processing list item with ID {0}", () => {
                    clientContext.Load(item.File);
                    var stream = item.File.OpenBinaryStream();
                    clientContext.ExecuteQuery();
                    var fileInfo = new FileSaveBinaryInformation();
                    fileInfo.ContentStream = new System.IO.MemoryStream();
                    // Load the app manifest file.
                    ProcessManifest(stream.Value, fileInfo.ContentStream, (manifest, ns) => {
                        // Load the properties.
                        var propNode = manifest.SelectSingleNode("x:App/x:Properties", ns);
                        // Look for the endpoints.
                        var installedNode = propNode.SelectSingleNode("x:InstalledEventEndpoint", ns);
                        var upgradedNode = propNode.SelectSingleNode("x:UpgradedEventEndpoint", ns);
                        var uninstalledNode = propNode.SelectSingleNode("x:UninstallingEventEndpoint", ns);
                        if (null == installedNode)
                        {
                            installedNode = manifest.CreateElement("InstalledEventEndpoint", manifest.DocumentElement.NamespaceURI);
                            propNode.AppendChild(installedNode);
                        }
                        if (null == upgradedNode)
                        {
                            upgradedNode = manifest.CreateElement("UpgradedEventEndpoint", manifest.DocumentElement.NamespaceURI);
                            propNode.AppendChild(upgradedNode);
                        }
                        if (null == uninstalledNode)
                        {
                            uninstalledNode = manifest.CreateElement("UninstallingEventEndpoint", manifest.DocumentElement.NamespaceURI);
                            propNode.AppendChild(uninstalledNode);
                        }
                        // NOTE: We're replacing the app installing and upgrading events so we can manage app lifecycle.
                        // If the deployed originally used these events, we've overridden them.
                        installedNode.InnerText = GetServiceUrl("AppMgmtReceiver.svc");
                        upgradedNode.InnerText = GetServiceUrl("AppMgmtReceiver.svc");
                        uninstalledNode.InnerText = GetServiceUrl("AppMgmtReceiver.svc");
                    });
                    // Save the manifest back to SharePoint.
                    fileInfo.ContentStream.Seek(0, System.IO.SeekOrigin.Begin);
                    item.File.SaveBinary(fileInfo);
                    clientContext.Load(item.File);
                    clientContext.ExecuteQuery();
                }, item.Id);
            }
            catch (Exception ex)
            {
                Debug.WriteLine(ex.ToString());
                Logger.Logger.LogError(ex.ToString());
            }
        }

        private static void ProcessManifest(System.IO.Stream inStream, System.IO.Stream outStream, Action<XmlDocument, XmlNamespaceManager> manifestDel)
        {
            if (null == inStream) throw new ArgumentNullException("inStream");
            if (null == outStream) throw new ArgumentNullException("outStream");
            if (null == manifestDel) throw new ArgumentNullException("manifestDel");
            using (var memory = new System.IO.MemoryStream())
            {
                var buffer = new byte[1024 * 64];
                int nread = 0, total = 0;
                while ((nread = inStream.Read(buffer, 0, buffer.Length)) > 0)
                {
                    memory.Write(buffer, 0, nread);
                    total += nread;
                }
                memory.Seek(0, System.IO.SeekOrigin.Begin);
                // Open the app manifest.
                using (var zipArchive = new ZipArchive(memory, ZipArchiveMode.Update, true))
                {
                    var entry = zipArchive.GetEntry("AppManifest.xml");
                    if (null == entry) throw new Exception("Could not find AppManifest.xml in the app archive");
                    var manifest = new XmlDocument();
                    using (var sr = new System.IO.StreamReader(entry.Open()))
                    {
                        manifest.LoadXml(sr.ReadToEnd());
                        sr.Close();
                    }
                    var ns = new XmlNamespaceManager(manifest.NameTable);
                    ns.AddNamespace("x", "http://schemas.microsoft.com/sharepoint/2012/app/manifest");
                    // Call the delegate.
                    manifestDel(manifest, ns);
                    // Write back to the archive.
                    using (var sw = new System.IO.StreamWriter(entry.Open()))
                    {
                        sw.Write(manifest.OuterXml);
                        sw.Close();
                    }
                }
                // Memory stream now contains the updated archive
                memory.Seek(0, System.IO.SeekOrigin.Begin);
                // Write result to output stream.
                buffer = new byte[1024 * 64];
                nread = 0; total = 0;
                while ((nread = memory.Read(buffer, 0, buffer.Length)) > 0)
                {
                    outStream.Write(buffer, 0, nread);
                    total += nread;
                }
            }
        }

        private static string GetServiceUrl(string serviceEndpoint)
        {
            if (string.IsNullOrEmpty(serviceEndpoint)) throw new ArgumentNullException("serviceEndpoint");
            if (null == OperationContext.Current) throw new Exception("Could not get service URL from the operational context.");
            var url = OperationContext.Current.Channel.LocalAddress.Uri.AbsoluteUri;
            var opContext = url.Substring(0, url.LastIndexOf("/", StringComparison.Ordinal));
            return string.Format("{0}/{1}", opContext, serviceEndpoint);
        }
    }
}

This class, both registers event receivers on the App Catalog as well as provide logic for the injection of App Installed events for new and existing add-ins. The ProcessListItem method is the most interesting (in my opinion). This method downloads the APP file from the catalog into a memory stream, unzips the contents with the .NET compression API, makes the changes to the AppManifest.xml file, and then zips the file and re-uploads the APP to the catalog.

The location of the injected endpoint is another remote event receiver on the ACU. The governance code should reside somewhere central, and hosted in the ACU seemed like as good a place as any.

In the above code, we can trace the call from the ACU ProcessEvent method through to the code that updates the existing add-ins in the catalog. For new add-ins added to the catalog later, we also need a list remote event receiver, which the ACU registers and calls an endpoint ListEventReceiver.svc. The following is the code-behind for this service (again hosted in the ACU):

using System;
using System.Diagnostics;
using Microsoft.SharePoint.Client.EventReceivers;
using SPApps.UpdateAppCatWeb;

namespace SPApps.SubSiteCreateWeb.Services
{
    public class ListEventReceiver : IRemoteEventService
    {
        public void ProcessOneWayEvent(SPRemoteEventProperties properties)
        {
            throw new NotImplementedException();
        }

        public SPRemoteEventResult ProcessEvent(SPRemoteEventProperties properties)
        {
            var result = new SPRemoteEventResult();
            Logger.Logger.LogInfo("ProcessEvent called on ListEventReceiver", () =>
            {
                if (null == properties) throw new ArgumentNullException("properties");
                try
                {
                    switch (properties.EventType)
                    {
                        case SPRemoteEventType.ItemAdded:
                            using (var clientContext = TokenHelper.CreateRemoteEventReceiverClientContext(properties))
                                AppHelper.ProcessAppListItem(clientContext, properties.ItemEventProperties.ListItemId);
                            break;
                    }

                }
                catch (Exception ex)
                {
                    Logger.Logger.LogError(ex.ToString());
                    Debug.WriteLine(ex.ToString());
                }
            });
            return result;
        }
    }
}

Going back to the App Helper, take note of the code that registers the List Remote Event Receiver. I register the App Installed event as Synchronous. This is important, because this event is fired when uploading an APP to the catalog. SharePoint provides a dialog box to enter metadata about the add-in as part of the upload process. Leaving the event as asynchronous (default) causes a conflict error when clicking the save button on the dialog.

Lastly, let’s look at the code that is called from the injected RER endpoint – the code that allows or denies installation of the add-in:

using System;
using Microsoft.SharePoint.Client.EventReceivers;

namespace SPApps.UpdateAppCatWeb.Services
{
    public class AppMgmtReceiver : IRemoteEventService
    {
        /// <summary>
        /// Handles app events that occur after the app is installed or upgraded, or when app is being uninstalled.
        /// </summary>
        /// <param name="properties">Holds information about the app event.</param>
        /// <returns>Holds information returned from the app event.</returns>
        public SPRemoteEventResult ProcessEvent(SPRemoteEventProperties properties)
        {
            var result = new SPRemoteEventResult();
            Logger.Logger.LogInfo("ProcessEvent called for AppMgmtReceiver", () =>
            {
                using (var clientContext = TokenHelper.CreateAppEventClientContext(properties, false))
                {
                    switch (properties.EventType)
                    {
                        case SPRemoteEventType.AppInstalled:
                            result.Status = SPRemoteEventServiceStatus.CancelWithError;
                            result.ErrorMessage = "You are not allowed to install this app!";
                            break;
                        case SPRemoteEventType.AppUpgraded:
                            break;
                    }
                }
            });
            return result;
        }

        /// <summary>
        /// This method is a required placeholder, but is not used by app events.
        /// </summary>
        /// <param name="properties">Unused.</param>
        public void ProcessOneWayEvent(SPRemoteEventProperties properties)
        {
            var result = new SPRemoteEventResult();
            Logger.Logger.LogInfo("ProcessEvent called for AppMgmtReceiver", () =>
            {
                using (var clientContext = TokenHelper.CreateAppEventClientContext(properties, false))
                {
                    switch (properties.EventType)
                    {
                        case SPRemoteEventType.AppInstalled:
                            break;
                        case SPRemoteEventType.AppUpgraded:
                            break;
                    }
                }
            });
        }

    }
}

The above code is very simplistic and just denies installation of any add-in. However, this could easily be adapted (and will be for my customer) to check a central repository for before determining that an add-in is allowed or denied installation.

So, there we have it. With a little bit of trickery to the App Catalog it’s possible to implement rudimentary add-in governance. The complete project is available on github: https://github.com/robgarrett/SharePoint-Apps.

ALL CODE PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND. I MAKE NO WARRANTIES, EXPRESS OR IMPLIED, THAT THEY ARE FREE OF ERROR, OR ARE CONSISTENT WITH ANY PARTICULAR STANDARD OF MERCHANTABILITY, OR THAT IT WILL MEET YOUR REQUIREMENTS FOR ANY PARTICULAR APPLICATION. I DISCLAIM ALL LIABILITY FOR DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES RESULTING FROM YOUR USE OF THE INCLUDED CODE. ALL CODE PROVIDED OR LINKED IS FREE FOR DISTRIBUTION AND IS NOT CONSIDERED PROTECTED INTELLECTUAL PROPERTY OF MINE NOR MICROSOFT CORPORATION.