Wednesday, September 14, 2016

SharePoint Automation: unified Powershell script to copy a page or a document in SharePoint Online and "on-premises"

This simple Powershell script automates copying a page with all web parts on it in the scope of the same sub-site. It can also be used for copying regular files situated inside or outside document libraries as well as just for deleting redundant pages and files (through the setting $onlyRenoveDestinationFile = $true).

This script can be useful, for example, when you create an Enterprise Search Center and want to add more pages to your search navigation. The ordinary routine process would include exporting all web-parts from an existing page, creating new pages and deploying the exported web-parts to them. However, this script provides more elegant and lightweight alternative for similar operations.

The main script

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
param ( 
  $webUrl = "news",                      # Site collection relative URL of the sub-site that contains source file to copy (use $null or "" for the root site).
  $sourceFile = "home.aspx",             # Name or site relative URL of the source file to copy (documents/file1.docx, set $usePublishingPageUrl=$false).
  $destinationFile = "newpage.aspx",     # Name or site relative URL of the destination file to copy the source one to (documents/file2.docx, set $usePublishingPageUrl=$false).
  $usePublishingPageUrl = $true,         # A flag that instructs to search for the source and the destination file URLs in the pages library of the sub-site.
  $overwrite = $true,                    # A flag that instructs to overwrite the destination file in the case it already exists.
  $publish = $true,                      # A flag that instructs to publish major version of the destination file after copying (concerns SharePoint Online).
  $approve = $true,                      # A flag that instructs to approve the destination file after copying (concerns SharePoint Online).
  $onlyRenoveDestinationFile = $false    # A flag that instructs only to remove the destination file without copying the source.
)
 
# Step 1. Connect to SharePoint environment amd get a client context.
write-host
write-host "Connecting to SharePoint environment..." -NoNewLine
. $PSScriptRoot\__LoadContext.ps1
if( $context -eq $null ) {
  write-host "Connection failed. Further execution is impossible."
  return
}
write-host "Done"

# Step 2. Open a sub-site, if required.
$serviceName = $null
$sourceUrl = $null
$destinationUrl = $null
$targetWeb = $null
if( ![string]::IsNullOrEmpty($webUrl) ) {
  $targetWeb = $context.Site.OpenWeb($webUrl)
  $context.Load($targetWeb)
  $context.ExecuteQuery()
} else {
  $targetWeb = $context.Site.RootWeb
  $serviceName = $context.Site.RootWeb.ServerRelativeUrl
}
$serviceName = $targetWeb.ServerRelativeUrl
if( $usePublishingPageUrl ) {
  $context.Load($targetWeb.Lists)
  $context.ExecuteQuery()
  $targetWeb.Lists | ? {$_.BaseTemplate -eq 850} | % {
    $context.Load($_.RootFolder)
    $context.ExecuteQuery()
    if( $sourceUrl -eq $null ) {
      $sourceUrl = [string]::Format("{0}/{1}", $_.RootFolder.Name, $sourceFile)
    }
    if( $destinationUrl -eq $null ) {
      $destinationUrl = [string]::Format("{0}/{1}", $_.RootFolder.Name, $destinationFile)
    }
  }
} else {
  $sourceUrl = $sourceFile
  $destinationUrl = $destinationFile
}
write-host
$message = "Processing the URL " + $targetWeb.Url
write-host $message
write-host

# Step 3. Delete the destination file if it already exists.
if( $onlyRenoveDestinationFile ) {
  $requestUrl = $targetWeb.Url + "/_vti_bin/_vti_aut/author.dll"
  $request = CreateWebRequest $requestUrl
  $request.Method = "POST"
  $request.ContentType = "application/x-www-form-urlencoded"
  $request.Headers["X-Vermeer-Content-Type"] = "application/x-www-form-urlencoded"

  $requestData = new-object "System.Collections.Generic.Dictionary[[string],[string]]"
  $requestData.Add("method", "remove documents:" + $context.ServerVersion)
  $requestData.Add("service_name", $serviceName)
  $requestData.Add("url_list", "[" + $destinationUrl + "]")

  $errorMessage = "(?i)<ul>\s+?<li>status=\d+.+<li>msg=([^<]+)<"
  $successMessage = "(?i)<body>\s+?<p>method=[^<]+\s+?<p>message=([^<]+)<.+<li>document_name=([^<]+)<"

  $content = ExecuteWebRequest $request $requestData ([ref]$errorMessage) ([ref]$successMessage)
  if( $errorMessage.Length -gt 0 ) {
    write-host $errorMessage -ForegroundColor Yellow
  }
  if( $successMessage.Length -gt 0 ) {
    write-host $successMessage -ForegroundColor Green
  }

  return
}

# Step 4. Copy the source page with all web parts to a new one.
$put_options = "createdir,migrationsemantics"
if( $overwrite ) {
  $put_options = "overwrite," + $put_options
}

$requestData = new-object "System.Collections.Generic.Dictionary[[string],[string]]"
$requestData.Add("method", "move document:" + $context.ServerVersion)
$requestData.Add("service_name", $serviceName)
$requestData.Add("oldUrl", $sourceUrl)
$requestData.Add("newUrl", $destinationUrl)
$requestData.Add("url_list", "[]")
$requestData.Add("rename_option", "findbacklinks")
$requestData.Add("put_option", $put_options)
$requestData.Add("docopy", "true")

$requestUrl = $targetWeb.Url + "/_vti_bin/_vti_aut/author.dll"
$request = CreateWebRequest $requestUrl
$request.Method = "POST"
$request.ContentType = "application/x-www-form-urlencoded"
$request.Headers["X-Vermeer-Content-Type"] = "application/x-www-form-urlencoded"

$errorMessage = "(?i)<ul>\s+?<li>status=\d+.+<li>msg=([^<]+)<.+</ul>\s+?<p>message=([^<]+)<"
$successMessage = "(?i)<body>\s+?<p>method=[^<]+\s+?<p>message=([^<]+)<"

$content = ExecuteWebRequest $request $requestData ([ref]$errorMessage) ([ref]$successMessage)
if( $errorMessage.Length -gt 0 ) {
  write-host $errorMessage -ForegroundColor Yellow
}
if( $successMessage.Length -gt 0 ) {
  write-host $successMessage -ForegroundColor Green
}

# Step 5. Publish and approve the destination file, if required.
if( !($errorMessage.Length -gt 0) ) {
  $fileUrl = $targetWeb.ServerRelativeUrl.TrimEnd('/') + '/' + $destinationUrl
  $file = $targetWeb.GetFileByServerRelativeUrl($fileUrl)
  $list = $file.ListItemAllFields.ParentList
  $context.Load($file)
  $context.Load($list)
  $context.ExecuteQuery()
  if( $? -eq $false ) {
    return $null
  }

  $enableMinorVersions = $list.EnableMinorVersions
  $enableModeration = $list.EnableModeration
  $processed = $false
  if( $publish -and $enableMinorVersions -and $file.MinorVersion -gt 0 ) {
    $file.Publish("")
    $context.ExecuteQuery()
    write-host " Published" -NoNewLine
    $processed = $true
  }
  if( $approve -and $enableModeration ) {
    if( $processed ) {
      write-host "," -NoNewLine
    }
    $file.Approve("")
    $context.ExecuteQuery()
    write-host " Approved" -NoNewLine
    $processed = $true
  }
  if( $processed ) {
    write-host
    write-host
  }
}

Limitations

The script shown above works in the scope of a single sub-site (web); it cannot copy pages or files between different sub-sites of site collections.

The current version of SharePoint Online does not seem to support copying file versions like it was supported in the past by SharePoint 2013 and 2010 "on-premises".

Context loader

The script below should have the name __LoadContext.ps1. It is loaded and used by the main script shown above. Also change values of the settings in the header of this script to comply with your SharePoint environment.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
<#
Unified automation script that loads a client context of SharePoint Online and "on-premises" (2013, 2016).
- This is a common script basic functions of which are loaded and executed from other scripts.
- The latest version: http://paulborisov.blogspot.fi/2016/09/sharepoint-automation-unified_14.html
#>
param(
  $siteCollectionUrl = "https://contoso.sharepoint.com/sites/sitecollection2016", # URL of either SharePoint Online or "on-premises" site collection
  $username = "paul.borisov@contoso.onmicrosoft.com",                             # Login name of either SharePoint Online account or "on-premises" Windows user in format DOMAIN\account
  #$siteCollectionUrl = "https://dev.contoso.local/sites/sitecollection2016",     # URL of either SharePoint Online or "on-premises" site collection
  #$username = "vanilla\paul.borisov",                                            # Login name of either SharePoint Online account or "on-premises" Windows user in format DOMAIN\account
  $password = "**********",                                                       # User's password
  $useLocalEnvironment = $false,    # Set to $true if you intend to use local SharePoint environment. This can be usefull in case of "on-premises" site collections (not in the SharePoint Online).
  $useDefaultCredentials = $false,  # Set to $true if you intend to use default network credentials instead of username and password
  $pathToDlls = "C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI", # Full path to the folder that contains SharePoint Client components (use /15/ for SP2013 and /16/ for SP2016)
  $logFile = $PSScriptRoot + "\" + (get-date).ToString("yyyy-MM-dd-HH-mm-") + "log.txt"
)
######################################################## FUNCTIONS ########################################################
function GetAuthenticationCookie() {
  $sharePointUri = New-Object System.Uri($context.Url)
  $authCookie = $context.Credentials.GetAuthenticationCookie($sharePointUri)
  if( $? -eq $false ) {
    return $null
  } else {
    $fedAuthString = $authCookie.TrimStart("SPOIDCRL=".ToCharArray())
    $cookieContainer = New-Object System.Net.CookieContainer
    $cookieContainer.Add($sharePointUri, (New-Object System.Net.Cookie("SPOIDCRL", $fedAuthString)))
    return $cookieContainer
  }
}
   
function GetClientContext() {
  $context = New-Object Microsoft.SharePoint.Client.ClientContext($siteCollectionUrl)
  if( $useDefaultNetworkCredentials ) {
    $context.Credentials = [System.Net.CredentialCache]::DefaultCredentials
  } elseif( $useLocalEnvironment ) {
    #$domain = $null
    #$login = $null
    #if( $username.Contains("@") ) {
    #  $domain = $username.Substring($username.IndexOf('@') + 1)
    #  $login = $username.Substring(0, $username.IndexOf('@'))
    #  $context.Credentials = (New-Object System.Net.NetworkCredential($login, $password, $domain))
    #} elseif( $username.Contains("\") ) {
    #  $domain = $username.Substring(0, $username.IndexOf('\'))
    #  $login = $username.Substring($username.IndexOf('\') + 1)
    #  $context.Credentials = (New-Object System.Net.NetworkCredential($login, $password, $domain))
    #} else {
    $context.Credentials = (New-Object System.Net.NetworkCredential($username, $password))
    #}
  } else {
    $securePassword = ConvertTo-SecureString $password -AsPlainText -Force
    $credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($username, $securePassword)
    $context.Credentials = $credentials
  }
  $context.Load($context.Site)
  $context.Load($context.Site.RootWeb)
   
  $context.ExecuteQuery()
  if( $? -eq $false ) {
    return $null
  } else {
    return $context
  }
}

function CreateWebRequest($requestUrl) {
  $cookieContainer = $null
  if( !$useLocalEnvironment -and !$useDefaultNetworkCredentials ) {
    $cookieContainer = GetAuthenticationCookie
    if( $cookieContainer -eq $null ) {
      return
    }
  }

  $request = $context.WebRequestExecutorFactory.CreateWebRequestExecutor($context, $requestUrl).WebRequest
  $request.Method = "GET"
  $request.Accept = "text/html, application/xhtml+xml, */*"
  if( $cookieContainer -ne $null )  {
    $request.CookieContainer = $cookieContainer
  } elseif ( $useDefaultNetworkCredentials ) {
    $request.UseDefaultCredentials = $true
  } elseif( $useLocalEnvironment ) {
    $request.Credentials = $context.Credentials
  }
  $request.UserAgent = "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"
  $request.Headers["Cache-Control"] = "no-cache"
  $request.Headers["Accept-Encoding"] = "gzip, deflate"
  return $request
}

function ExecuteWebRequest($request, $requestData, [ref]$errorMessage, [ref]$successMessage) {
  if( $requestData -ne $null -and $requestData.Keys -ne $null ) {
    # Format inputs as postback data string
    $strData = $null
    foreach( $inputKey in $requestData.Keys )  {
      if( -not([String]::IsNullOrEmpty($inputKey)) ) {
        $strData += [System.Web.HttpUtility]::UrlEncode($inputKey) `
          + "=" + [System.Web.HttpUtility]::UrlEncode($requestData[$inputKey]) + "&"
      }
    }
    $strData = $strData.TrimEnd('&')
    $requestData = [System.Text.Encoding]::UTF8.GetBytes($strData)
  } 
  $request.ContentLength = $requestData.Length
   
  # Add postback data to the request stream
  $stream = $request.GetRequestStream()
  try {
    $stream.Write($requestData, 0, $requestData.Length)
  } finally {
    $stream.Close()
    $stream.Dispose()
  }
     
  $response = $request.GetResponse()
  $stream = $response.GetResponseStream()
  try {
    if( -not([string]::IsNullOrEmpty($response.Headers["Content-Encoding"])) ) {
      if( $response.Headers["Content-Encoding"].ToLower().Contains("gzip") ) {
        $stream = New-Object System.IO.Compression.GZipStream($stream, 
          [System.IO.Compression.CompressionMode]::Decompress)
      } elseif ( $response.Headers["Content-Encoding"].ToLower().Contains("deflate") ) {
        $stream = New-Object System.IO.Compression.DeflateStream($stream, 
          [System.IO.Compression.CompressionMode]::Decompress)
      }
    }
         
    # Retrieve response content as string
    $reader = New-Object System.IO.StreamReader($stream)
    $content = $reader.ReadToEnd()
    #$content > $logFile
    if( $successMessage.Value -ne $null ) {
      $match = [regex]::Match($content, $successMessage.Value, "Singleline")
      $successMessage.Value = $null
      if( $match.Success ) {
        for( $i = 1; $i -lt $match.Groups.Count; $i++ ) {
          $tmp = $match.Groups[$i].Value
          if( $tmp.Length -gt 1 ) {
            $tmp = $tmp.Substring(0,1).ToUpper() + $tmp.Substring(1)
          }
          $successMessage.Value += $tmp
        }
      }     
    }
    $match = $null
    if( $errorMessage.Value -ne $null ) {
      $match = [regex]::Match($content, $errorMessage.Value, "Singleline")
    } else {
      $match = [regex]::Match($content, 
        '(?i)<div id="ms-error">.+<span id="ctl\d+_PlaceHolderMain_LabelMessage">(.[^<]+)<\/span>', "Singleline")
    }
    $errorMessage.Value = $null
    if( $match.Success ) {
      for( $i = 1; $i -lt $match.Groups.Count; $i++ ) {
        $tmp = $match.Groups[$i].Value
        if( $tmp.Length -gt 1 ) {
          $tmp = $tmp.Substring(0,1).ToUpper() + $tmp.Substring(1)
        }
        $errorMessage.Value += $tmp
      }
    }
    $reader.Close()
    $reader.Dispose()
  } finally {        
    $stream.Close()
    $stream.Dispose()
  }
}
####################################################### //FUNCTIONS #######################################################
   
####################################################### EXECUTION #########################################################
# Step 1. Ensure the client components loaded
[Reflection.Assembly]::LoadFile(($pathToDlls.TrimEnd('\') + "\Microsoft.SharePoint.Client.dll")) | out-null
[Reflection.Assembly]::LoadFile(($pathToDlls.TrimEnd('\') + "\Microsoft.SharePoint.Client.Runtime.dll")) | out-null
$context = GetClientContext

Known issues

If your connection attempt failed with the error "The remote server returned an error: (403) Forbidden." make sure you are connecting to the correct environment using valid credentials and correct configuration parameters.
  • If you connect to the SharePoint Online both configuration parameters $useLocalEnvironment and $useDefaultCredentials must be set to $false.
  • If you connect to an "on-premises" SharePoint environment the configuration setting $useLocalEnvironment must be set to $true while  $useDefaultCredentials can be $true or $false depending on the credentials you use to connect. If $useDefaultCredentials = $false is specified you should use valid $username and $password for your local environment.
  • Verify the credentials specified in the configuration parameters $username and $password are correct and correspond to the chosen environment.
  • Also verify a value for $pathToDlls. If you run the script on a machine that has a local SharePoint Farm installed this setting usually points a standard SP-hive. It can contain /15/ or /16/ depending on your local version of SharePoint.
    • I have had a challenge once when I have tested the script on a Virtual Machine with SharePoint 2013 installed that also had a folder of 16-hive with a stuff for SharePoint 2016 added by some tool. And I have tried connecting to SharePoint Online.
    • In that case the setting $pathToDlls = "C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI" have worked perfectly while $pathToDlls = "C:\Program Files\Common Files\microsoft shared\Web Server Extensions\16\ISAPI" have refused to work producing the error 403 (see above).
Posted by at
Labels: , , ,

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)