.Net, Azure and occasionally gamedev

Managed identity and Azure Functions

2019/03/09

I've already blogged about using Managed identity (MSI) with .Net Core.

There are many more services that support MSI (e.g. Event Hub, Logic Apps, ...).

It is also possible to use Managed Identity in Azure Functions to retrieve secrets from a keyvault during function calls.

Unfortunately, not all function calls can really benefit from it due to the nature of Azure Functions.

Since function calls are isolated the MSI authentication and secret retrieval has to occur on every single function call.

This is in contrast to a classic webapp where you pay the MSI authentication cost once at the startup.

Measuring startup cost

Since we pay the startup cost on every function call it is important to understand just how much of a performance hit it is.

I've built a few functions already and measured the time impact the MSI authentication has:

I've built timer triggered functions (daily/hourly) and request triggered functions.

For my functions the MSI token retrieval takes one second on average if the previous token is no longer valid (which is the case if you run your function on a timer once per hour).

And while the token retrieval can be as fast as a few milliseconds (if you run the same function multiple times in a row), you can't just rely on the token cache because it only works if all requests hit the same node where your function is running on.

If requests hit a different node (e.g. during scaleout or even just load balancing) you will pay the full token retrieval cost and there is no way of knowing when the scalout/load balancing happens.

Add to that another second until all secrets are loaded from the keyvault and you are looking at 2 second (average) startup cost before any of your own code runs.

If your code connects to Table storage to read/write/query some entries, a function call usually completes in 4-7 seconds with the full MSI integration.

However couple that with potential cold starts during scale out and the function call ends up taking a long time (10s+) which is no longer viable if you want to use function as part of a user request cycle (e.g. downloading file).

Use cases

As shown above the startup cost of MSI can causes functions to take a long time to run.

You won't be able to build a serverless system with real-time user requests if you need to pull secrets from a keyvault on every request and intend to use MSI to securely access them.

However, for background jobs and processing this setup is ideal.

I personally don't care that my daily blob storage cleanup function has a 2 second startup cost - on the contrary, I'll gladly accept it for the benefit of securely storing all my secrets in a keyvault and not having to hardcode them in code or Azure App Settings.

Code

The setup code is identical to .Net Core applications (if you run C# functions) and I simply wrapped it into its own helper method so I can call it from every function I build.

For local development it will pull the keyvault name from local.appsettings.json whereas for the deployed version I simply opted to store the keyvault name in an environment variable (that can be set in the Function App Settings in the portal).

private static IConfiguration LoadConfig(ExecutionContext context)
{
    var builder = new ConfigurationBuilder()
        .SetBasePath(context.FunctionAppDirectory)
        .AddJsonFile("local.settings.json", optional: true)
        .AddEnvironmentVariables();
    var tmpConfig = builder.Build();

    var tokenProvider = new AzureServiceTokenProvider();
    var kvClient = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(tokenProvider.KeyVaultTokenCallback));
    builder.AddAzureKeyVault($"https://{tmpConfig["KeyVaultName"]}.vault.azure.net", kvClient, new DefaultKeyVaultSecretManager());

    return builder.Build();
}

All you need to install is Microsoft.Extensions.Configuration.AzureKeyVault from nuget.

If you are using Powershell you will have to authenticate by manually fetching the MSI token.

The docs have detailed explanation of how it works and while the documentation is written for VMs the same steps apply to Azure Functions as well (after all your functions run on some VM in the cloud) and here is even a handy gist with working Powershell code to retrieve a secret with MSI tokens.

Conclusion

While MSI has a performance impact it offers a convenient way of storing secrets in a keyvault and is usable both in the cloud and for local development.

This helps with keeping your code clean of secrets and making development easy (no Service Principal setup/shared known secret required).

Overall you will have to decide for each function if the startup impact is acceptable to you.

tagged as Azure, Keyvault, Functions, Managed Identity, MSI