Wednesday, June 21, 2017

Avoid getting throttled or blocked in SharePoint Online

What is throttling?


SharePoint Online uses throttling to maintain optimal performance and reliability of the SharePoint Online service. Throttling limits the number of user actions or concurrent calls (by script or code) to prevent overuse of resources.
That said, it is extremely rare for a user to get throttled in SharePoint Online. The service is robust, and it is designed to handle very high volume. If you do get throttled, 99% of the time it is because of custom code. That doesn't mean that there aren't other ways to get throttled, just that they are less common. For example you spin up 10 machines and have a sync client going on all 10. On each sync 1TB of content. This would likely get you throttled.


What happens when you get throttled in SharePoint Online?

When a user exceeds usage limits, SharePoint Online throttles any further requests from that user account for a short period. All user actions are throttled while the throttle is in effect.
  • For requests that a user performs directly in the browser, SharePoint Online redirects you to the throttling information page, and the requests fail.
  • For all other requests, including CSOM or REST calls, SharePoint Online returns HTTP status code 429 ("Too many requests"), and the requests fail.
If the offending process continues to exceed usage limits, SharePoint Online might completely block the process; in this case, you may see HTTP status code 503 ("Service unavailable"), and we'll notify you of the block in the Office 365 Message Center. The error message is shown below:



503 Server unavailable message.

Common throttling scenarios in SharePoint Online


The most common causes of per-user throttling in SharePoint Online are client-side object model (CSOM) or Representational State Transfer (REST) code that performs too many actions too frequently.
  • Sporadic traffic
    Not a lot of traffic at any one time, but enough over time that you run in and out of throttling in an episodic way.
    • For example, after migrating files to SharePoint Online, you run a custom CSOM or REST script to update metadata on the files. The CSOM/REST script is updating a large number of files at a very high frequency, which triggers throttling. Similarly, an autocomplete UI widget using REST services, making too many calls to lists during each end user operation, may also cause throttling, depending on what other operations are consuming resources at the same time.
  • Overwhelming traffic
    A single process dramatically exceeds throttling limits, continually, over a long time period.
    • You used web services to build a tool to synchronize user profile properties. The tool updates user profile properties based on information from your line-of-business (LOB) human resources (HR) system. The tool makes calls at too high a frequency.
    • You're running a load-testing script on SharePoint Online and you get throttled. Load testing is not allowed on SharePoint Online.
    • You customized your team site on SharePoint Online, for example, by adding a status indicator on the Home page. This status indicator updates frequently, which causes the page to make too many calls to the SharePoint Online service - this triggered throttling.

Why can't you just tell me the exact throttling limits?


Setting and publishing exact throttling limits sounds very straightforward, but in fact, it's not the best way to go. We continually monitor resource usage on SharePoint Online. Depending on usage, we fine-tune thresholds so users can consume the maximum number of resources without degrading the reliability and performance of SharePoint Online. That's why it's so important for your CSOM or REST code to include incremental back off to handle throttling; this lets your code run as fast as possible on any given day, and it lets your code back off "just enough" if it hits throttling limits. The code samples later in this article show you how to use incremental back off.

Best practices to handle throttling


  • Reduce the number of operations per request
  • Reduce the frequency of calls
  • Use incremental back off to reduce the number and frequency of calls until no more throttling occurs
Incremental back off uses progressively longer waits between retries before trying again to run the code that was throttled. You can use the GitHub code samples, later in this article, written as extension methods, to add incremental back off to your code.
Backing off is the fastest way to handle being throttled because SharePoint Online continues to log resource usage while a user is being throttled. In other words, aggressive retries work against you because even though the calls fail, they still accrue against your usage limits. The faster you back off, the faster you'll stop exceeding usage limits. 

GitHub CSOM code samples: SharePoint Online Throttling

Before you run this code sample:
  • Open Program.cs and enter the following information in the Main method:
    • Your Office 365 Developer account credentials.
    • The URL of your Office 365 Developer Site.
    • The name of a test document library on your Office 365 Developer Site.
  • If you receive an error stating that the App.Config file is invalid, go to Solution Explorer, right click App.config, and choose Exclude From Project.
Core.Throttling runs as a console application using a user-only authorization policy, which means this code sample uses the permissions of the current user. In the Main method in Program.cs, a while loop repeatedly creates new folders in the test document library. A call is then made to ctx.ExecuteQueryWithExponentialRetry, which uses CSOM to perform the ExecuteQuery method. ExecuteQueryWithExponentialRetry is an extension method on the ClientContext object, and is defined in ClientContextExtension.cs.
If SharePoint Online throttles the ExecuteQuery statement, ExecuteQueryWithIncrementalRetry starts the incremental back off technique by:
  • Catching a WebException and checking the HttpWebResponse.StatusCode. If SharePoint Online throttled the ExecuteQuery statement, the HttpWebResponse.StatusCode is 429.
  • The current thread is suspended for the period specified in backoffInterval.
  • When the current thread resumes, the backoffInterval is doubled and the number of retries performed ( retryAttempts ) is incremented. By doubling backoffInterval your code suspends activity for a longer period of time before retrying the code that was throttled by SharePoint Online.
  • The process is repeated until either the ExecuteQuery statement is successful, or the number of allowed retries ( retryCount ) is exceeded.

CSOM Code sample: Incremental back off and retry (calls ExecuteQueryWithIncrementalRetry method, later in this article)

using (var ctx = new ClientContext(serverUrl))
{
//Provide account and pwd for connecting to the source
var passWord = new SecureString();
foreach (char c in password.ToCharArray()) passWord.AppendChar(c);
ctx.Credentials = new SharePointOnlineCredentials(login, passWord);
try
{
int number = 0;
// This loop will be executed 1000 times, which will cause throttling to occur
while (number < 1000)
{
// Try to create new folder based on Ticks to the given list as an example process
var folder = ctx.Site.RootWeb.GetFolderByServerRelativeUrl(listUrlName);
ctx.Load(folder);
folder = folder.Folders.Add(DateTime.Now.Ticks.ToString());
// Extension method for executing query with throttling checks
ctx.ExecuteQueryWithIncrementalRetry(5, 30000); //5 retries, with a base delay of 30 secs.
// Status indication for execution.
Console.WriteLine("CSOM request successful.");
// For loop handling.
number = number + 1;
}
}
catch (MaximumRetryAttemptedException mex)
{
// Exception handling for the Maximum Retry Attempted
Console.WriteLine(mex.Message);
}
}

CSOM Code sample: ExecuteQueryWithIncrementalRetry method

public static void ExecuteQueryWithIncrementalRetry(this ClientContext context, 
int retryCount, int delay)
        {
            int retryAttempts = 0;
            int backoffInterval = delay;
            if (retryCount <= 0)
                throw new ArgumentException("Provide a retry count greater than zero.");
           if (delay <= 0)
                throw new ArgumentException("Provide a delay greater than zero.");
           while (retryAttempts < retryCount)
            {
                try
                {
                    context.ExecuteQuery();
                    return;
                }
                catch (WebException wex)
                {
                    var response = wex.Response as HttpWebResponse;
                    if (response != null &amp;&amp; response.StatusCode == (HttpStatusCode)429)
                    {
                        Console.WriteLine(string.Format("CSOM request exceeded usage limits. 
Sleeping for {0}
 seconds before retrying.", backoffInterval));
                        //Add delay.
                        System.Threading.Thread.Sleep(backoffInterval);
                        //Add to retry count and increase delay.
                        retryAttempts++;
                        backoffInterval = backoffInterval * 2;
                    }
                    else
                    {
                        throw;
                    }
                }
            }
            throw new MaximumRetryAttemptedException(string.Format("Maximum retry attempts {0}, 
have been attempted.", retryCount));
       }

What should you do if you get blocked in SharePoint Online?

Blocking is the most extreme form of throttling. We rarely ever block a tenant,
unless we detect long-term, extremely excessive traffic that may threaten the overall 
health of the SharePoint Online service. We apply blocks to prevent excessive traffic from
 degrading the performance and reliability of SharePoint Online. A block - which is usually
 placed at the tenancy level - prevents the offending process from running until you fix the problem. 

If we block your subscription, you must take action to modify the offending processes before
 the block can be removed.

If we block your subscription, you'll see HTTP status code 503, and we'll notify you of the
 block in the Office 365 Message Center. The message describes what caused the block, provides
 guidance on how to resolve the offending issue, and tells you who to contact to get the block removed.

Wednesday, June 7, 2017

PowerShell to automate turning on two-step verification in azure

To change the state using Azure AD PowerShell, you can use the following. You can change $st.State to equal one of the following states:
  • Enabled
  • Enforced
  • Disabled
Note: It's not recommended to move users directly from the Disable state to the Enforced state. Non-browser-based apps will stop working because the user has not gone through MFA registration and obtained an app password. If you have non-browser-based apps and require app passwords, we recommend that you go from a Disabled state to Enabled. This allows users to register and obtain their app passwords. After that, you can move them to Enforced.

PowerShell would be an option for bulk enabling users. Currently there is no bulk enable feature in the Azure portal and you need to select each user individually. This can be quite a task if you have many users. By creating a PowerShell script using the following, you can loop through a list of users and enable them

$st = New-Object -TypeName Microsoft.Online.Administration.StrongAuthenticationRequirement
    $st.RelyingParty = "*"
    $st.State = "Enabled"
    $sta = @($st)
    Set-MsolUser -UserPrincipalName jo@sharepoint.com -StrongAuthenticationRequirements $sta


Below is an example

$users = "jo@sharepoint.com","jol@sharepoint.com"
foreach ($user in $users)
{
    $st = New-Object -TypeName Microsoft.Online.Administration.StrongAuthenticationRequirement
    $st.RelyingParty = "*"
    $st.State = "Enabled"
    $sta = @($st)
    Set-MsolUser -UserPrincipalName $user -StrongAuthenticationRequirements $sta
}

Azure Multi-Factor Authentication in the cloud

This post walks through how to get started using Azure Multi-Factor Authentication in the cloud (office 365)

The following provides information on how to enable users using the Azure Classic Portal



Enable Azure Multi-Factor Authentication

As long as your users have licenses that include Azure Multi-Factor Authentication, there's nothing that you need to do to turn on Azure MFA. You can start requiring two-step verification on an individual user basis. The licenses that enable Azure MFA are:
  • Azure Multi-Factor Authentication
  • Azure Active Directory Premium
  • Enterprise Mobility + Security
If you don't have one of these three licenses, or you don't have enough licenses to cover all of your users, that's ok too. You just have to do an extra step and Create a Multi-Factor Auth Provider in your directory.

Turn on two-step verification for users

To start requiring two-start verification on for a user, change the user's state from disabled to enabled. User States are as below

User accounts in Azure Multi-Factor Authentication have the following three distinct states:
StateDescriptionNon-browser apps affected
DisabledThe default state for a new user not enrolled Azure Multi-Factor Authentication (MFA).No
EnabledThe user has been enrolled in Azure MFA, but has not registered. They will be prompted to register the next time they sign in.No. They continue to work until the registration process is completed.
EnforcedThe user has been enrolled and has completed the registration process for Azure MFA.Yes. Apps require app passwords.
Use the following procedure to enable MFA for your users.

To turn on multi-factor authentication

  1. Sign in to the Azure classic portal as an administrator.
  2. On the left, click Active Directory.
  3. Under Directory, select the directory for the user you wish to enable. 
  4. Click Users at the top

  5. At the bottom of the page, click Manage Multi-Factor Auth. A new browser tab opens.
  6. Find the user that you wish to enable for two-step verification. You may need to change the view at the top. Ensure that the status is disabled. 
  7. Place a check in the box next to their name and click Enable
  8. Click enable multi-factor auth
  9. You can now notice that the user's state has changed from disabled to enabled
Once you enabled, inform the user via email as the next time they try to sign in, they will be asked to enrol their account for two-step verification (MFA). Once they start using two-step verification, they would also need to set up app passwords to avoid getting locked out of non-browser apps.


Monday, June 5, 2017

Create Azure Multi-Factor Auth Provider

Two-step verification is available by default for global administrators who have Azure Active Directory, and Office 365 users

An Azure Multi-Factor Authentication Provider is used to take advantage of features provided by the full version of Azure MFA. It is for users who do not have licenses through Azure MFA, Azure AD Premium, or EMS. Azure MFA, Azure AD Premium, and EMS include the full version of Azure MFA by default. If you have licenses, then you do not need an Azure Multi-Factor Authentication Provider.


Create a Multi-Factor Authentication Provider

Use the following steps to create an Azure Multi-Factor Auth Provider.
  1. Sign in to the Azure classic portal as an administrator.
  2. On the left, select Active Directory.
  3. On the Active Directory page, at the top, select Multi-Factor Authentication Providers
  4. Click New at the bottom
  5. Select Multi-Factor Auth Provider under App Services
  6. Select Quick Create and provide below details for MFA
    1. Name – The name of the Multi-Factor Auth Provider.
    2. Usage Model – Choose one of two options:
      • Per Authentication – purchasing model that charges per authentication. Typically used for scenarios that use Azure Multi-Factor Authentication in a consumer-facing application.
      • Per Enabled User – purchasing model that charges per enabled user. Typically used for employee access to applications such as Office 365. Choose this option if you have some users that are already licensed for Azure MFA.
    3. Directory – The Azure Active Directory tenant that the Multi-Factor Authentication Provider is associated with. Be aware of the following:
      • You do not need an Azure AD directory to create a Multi-Factor Auth Provider. Leave the box blank if you are only planning to use the Azure Multi-Factor Authentication Server or SDK.
      • The Multi-Factor Auth Provider must be associated with an Azure AD directory to take advantage of the advanced features.
      • Azure AD Connect, AAD Sync, or DirSync are only a requirement if you are synchronizing your on-premises Active Directory environment with an Azure AD directory. If you only use an Azure AD directory that is not synchronized, then this is not required. 
  7. Once you click create, the Multi-Factor Authentication Provider is created and you should see a message stating: Successfully created Multi-Factor Authentication Provider. Click Ok

Sunday, June 4, 2017

Face API Using Microsoft Cognitive Services

The Face API which is a part of the Microsoft Cognitive Services helps you to identify and detect faces. It is also used to find similar faces, verify images to see if it’s of same persons .In this blog post, I’ll just use the detect service which detects faces and shows the gender , age, emotions and other data of the face.

Prerequisites: Create the Face API Service in Azure


As all Microsoft cognitive services, you can also create the face API service in Azure via the portal. It is part of the “Cognitive Services APIs”, so just search for it and create it. 



Select the Face API as the type of the cognitive service and check the pricing options:



Now, we need to note down the API key and the API endpoint url. Navigate to the service in the Azure portal and you will see the endpoint url in the overview. It’s currently only available in west-us, so the endpoint url will be: https://westus.api.cognitive.microsoft.com/face/v1.0
The keys can be found in “Keys” – just copy one of them and you can use it later in the application:



Using the Face API with C#.Net

The face API can be accessed via C# with a simple HttpClient or with the NuGet package Microsoft.ProjectOxford.Face. My first sample will use the HttpClient just to show how it works. It also returns by sure all data that is currently available. The NuGet package is not fully up to date, so it for example does not contain the emotions.

Access Face API with C# and HttpClient

In the following sample, I’ll just send an image to the face API and show the JSON output in the console. If you want to work with the data, then you can use Newtonsoft.Json with JObject.Parse, or as already stated, the NuGet package which is described later in this post


using System;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Threading.Tasks;
namespace MyAzureCognitiveService.Face
{
    class Program
    {
        private static string APIKEY = "[APIKEY]";
        static void Main(string[] args)
        {
            Console.WriteLine("Welcome to the Azure Cognitive Services - Face API");
            Console.WriteLine("Please enter image url:");
            string path = Console.ReadLine();
             
            Task.Run(async () =>
            {
                var image = System.IO.File.ReadAllBytes(path);
                var output = await DetectFaces(image);
                Console.WriteLine(output);
            }).Wait();
             
            Console.WriteLine("Press key to exit!");
            Console.ReadKey();
        }
        public static async Task<string> DetectFaces(byte[] image)
        {
            var client = new HttpClient();
            client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", APIKEY);
            string requestParams = "returnFaceId=true&returnFaceLandmarks=true&returnFaceAttributes=age,
gender,headPose,smile,facialHair,glasses,emotion";
            string uri = "https://westus.api.cognitive.microsoft.com/face/v1.0/detect?" + requestParams;
            using (var content = new ByteArrayContent(image))
            {
                content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
                var response = await client.PostAsync(uri, content);
                var jsonText = await response.Content.ReadAsStringAsync();
                return jsonText;
            }
        }
    }
}


For testing, I have used the below image and output given below.



output.json

[{
        "faceId": "c41cd9de-76c8-4f10-b6f5-d01bb08ec616",
        "faceRectangle": {
            "top": 332,
            "left": 709,
            "width": 48,
            "height": 48
        },
        "faceLandmarks": {
            "pupilLeft": {
                "x": 723.6,
                "y": 344.7
            },
            "pupilRight": {
                "x": 744.2,
                "y": 346.3
            },
            "noseTip": {
                "x": 732.8,
                "y": 357.6
            },
            "mouthLeft": {
                "x": 720.7,
                "y": 365.6
            },
            "mouthRight": {
                "x": 743.7,
                "y": 367.1
            },
            "eyebrowLeftOuter": {
                "x": 715.8,
                "y": 341.8
            },
            "eyebrowLeftInner": {
                "x": 728.3,
                "y": 341.2
            },
            "eyeLeftOuter": {
                "x": 720.4,
                "y": 345.1
            },
            "eyeLeftTop": {
                "x": 723.3,
                "y": 344.5
            },
            "eyeLeftBottom": {
                "x": 723.3,
                "y": 345.8
            },
            "eyeLeftInner": {
                "x": 726.3,
                "y": 345.5
            },
            "eyebrowRightInner": {
                "x": 738.2,
                "y": 342.2
            },
            "eyebrowRightOuter": {
                "x": 752.0,
                "y": 342.8
            },
            "eyeRightInner": {
                "x": 740.5,
                "y": 346.3
            },
            "eyeRightTop": {
                "x": 743.6,
                "y": 345.7
            },
            "eyeRightBottom": {
                "x": 743.3,
                "y": 347.1
            },
            "eyeRightOuter": {
                "x": 746.4,
                "y": 347.0
            },
            "noseRootLeft": {
                "x": 730.5,
                "y": 346.3
            },
            "noseRootRight": {
                "x": 736.4,
                "y": 346.5
            },
            "noseLeftAlarTop": {
                "x": 728.3,
                "y": 353.3
            },
            "noseRightAlarTop": {
                "x": 738.3,
                "y": 353.7
            },
            "noseLeftAlarOutTip": {
                "x": 726.2,
                "y": 356.6
            },
            "noseRightAlarOutTip": {
                "x": 739.8,
                "y": 357.7
            },
            "upperLipTop": {
                "x": 733.0,
                "y": 365.1
            },
            "upperLipBottom": {
                "x": 732.7,
                "y": 366.4
            },
            "underLipTop": {
                "x": 731.7,
                "y": 370.6
            },
            "underLipBottom": {
                "x": 731.4,
                "y": 373.1
            }
        },
        "faceAttributes": {
            "smile": 1.0,
            "headPose": {
                "pitch": 0.0,
                "roll": 3.2,
                "yaw": -0.5
            },
            "gender": "male",
            "age": 33.6,
            "facialHair": {
                "moustache": 0.0,
                "beard": 0.2,
                "sideburns": 0.2
            },
            "glasses": "ReadingGlasses",
            "emotion": {
                "anger": 0.0,
                "contempt": 0.0,
                "disgust": 0.0,
                "fear": 0.0,
                "happiness": 1.0,
                "neutral": 0.0,
                "sadness": 0.0,
                "surprise": 0.0
            }
        }
    }
]



It seems I look like a 33-year-old man and that my face on the image is the pure happiness (100%). All other emotions are non-existent (0%).
Access Face API with C# and the NuGet package
As already mentioned, there is the NuGet package Microsoft.ProjectOxford.Face which makes it very easy to access the face API. Unfortunately, it does not wrap all properties (emotions), but there are already some commits in the GitHub project (https://github.com/Microsoft/Cognitive-Face-Windows) that will fix that.
Detect faces
This is nearly the same sample as above, but this time I’ll use the NuGet package. As already mentioned, the package does currently not contain the emotions, that’s why I’ll just show the smile factor


using Microsoft.ProjectOxford.Face;
using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Threading.Tasks;
namespace MyAzureCognitiveService.Face
{
    class Program
    {
        private static string APIKEY = "[APIKEY]";
        static void Main(string[] args)
        {
            Console.WriteLine("Welcome to the Azure Cognitive Services - Face API");
            Console.WriteLine("Please enter image url:");
            string path = Console.ReadLine();
            Task.Run(async () =>
            {
                var faces = await DetectFaces(path);
                foreach(var face in faces)
                {
                    Console.WriteLine($"{face.FaceAttributes.Gender},
{face.FaceAttributes.Age}: Smile: {face.FaceAttributes.Smile}");
                }
            }).Wait();
            Console.WriteLine("Press key to exit!");
            Console.ReadKey();
        }
        public static async Task<Microsoft.ProjectOxford.Face.Contract.Face[]> DetectFaces(string path)
        {
            var client = new FaceServiceClient(APIKEY);
            using (System.IO.Stream stream = System.IO.File.OpenRead(path))
            {
                var data = await client.DetectAsync(stream, true, true, new List<FaceAttributeType>()
                {
                    FaceAttributeType.Age,
                    FaceAttributeType.Gender,
                    FaceAttributeType.Glasses,
                    FaceAttributeType.Smile
                });
                return data;
            }
        }
    }
}


In this post, I used the detect service, but the face API using cognitive service has much more functionality.