.Net, Azure and occasionally gamedev

VSTS switch agent queue for all builds

2017/11/25

In Visual Studio Team Services you get 240 free build minutes per month by default.

You can use them on any of the provided hosted agents (build agents with either Linux or VS2017 and a few more capabilities installed).

It just so happens that I ran out of build minutes this month as I was testing new android builds along with a few projects that had staging/production environments and it ate up all my build minutes.

Instead I am using the hosted agents as well as a super fast F4 VM in azure that I only spin up when I do a lot of builds as it is expensive.

It has always annoyed me that I had to manually switch the 1-2 affected projects between hosted agents and my own agent whenever I switched between them but I always just manually changed them in the web ui (~10 clicks).

So now that I'm out of build minutes nothing is building and I would have to switch all build and release definitions in all projects manually to my VM and then back at again to the hosted queue at the end of the month.

Or I could just automate it via a script:

Or so I thought.

Unfortunately the release definition API is only in preview right now and doesn't allow changing queues.

I was able to get it to work for all build definitions, but still had to manually switch the release due to the api not supporting it (yet).

I will update this post when the api is updated to include release definition patching. Edit: It finally happended, check out my updated post for details.

# the target queue name, all builds will use this queue
# default queue names are "Hosted", "Hosted VS2017" and "Hosted Linux"
$newQueueName = "Hosted VS2017"

# needs to support manage build + manange queue + manage releases
$token = "Enter-your-token-here"
# vsts subdomain
$account = "Your-VSTS-name"

$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f "",$token)))

$projectApiUrl = "https://{0}.visualstudio.com/_apis/projects?api-version=3.0" -f $account
$projectResponse = Invoke-RestMethod -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} -Uri $projectApiUrl -Method Get

ForEach($p in $projectResponse.value) {
    $project = $p.name
    # each project has its own list of queues; same agent might have different ids between projects
    $buildAgentQueueUrl = "https://{0}.visualstudio.com/{1}/_apis/distributedtask/queues?api-version=4.0-preview" -f $account, $project
    $queuesResponse = Invoke-RestMethod -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} -Uri $buildAgentQueueUrl -Method Get
    
    "Working on: {0}" -f $project
    # "  Queues by name: "
    # $queuesResponse.value | foreach {
    #     "    {0} (id: {1})" -f $_.name, $_.id
    # }
    # get the id of the agent we want to use
    $newQueue = $queuesResponse.value | Where-Object {
        $_.name -eq $newQueueName | Select-Object -First 1
    }
    if ($newQueue) {
        # now update all build definitions
        $buildDefinitionsUrl = "https://{0}.visualstudio.com/{1}/_apis/build/definitions?api-version=3.0" -f $account, $project
        $buildResponse = Invoke-RestMethod -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} -Uri $buildDefinitionsUrl -Method Get
    
        "  Builds:"
        if ($buildResponse.count -eq 0) {
            "    No builds"
        }
        $buildResponse.value | foreach {
            # "    Build definition: '{0}' (id: {1})" -f $_.name, $_.id
            if ($_.queue.name -eq $newQueueName) {
                "    Build '{0}' already uses '{1}'" -f $_.name, $newQueueName
            }
            else {
                "    Patching '{0}' to use '{1}'" -f $_.name, $newQueueName
                # Download the build definition
                $updateUrl="$($_.url)?api-version=3.0"
                $definition = Invoke-RestMethod -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} -Uri $updateUrl -Method Get
                $definition.queue.name = $newQueue.name
                $definition.queue.id = $newQueue.id
                $definition.queue.pool = $newQueue.pool
                $json = $definition | ConvertTo-Json -Depth 20
                # now upload the changed definition
                $response = Invoke-RestMethod -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} -Uri $updateUrl -ContentType "application/json" -Method Put -Body $json
                "      Patched '{0}' to usue '{1}" -f $_.name, $response.queue.name
            }
        }
        # now update all release definitions

        # note the use of "vsrm" in the url (without it, it won't work).
        # all the "sample urls" in the doc use the ".vsrm." subdomain, but there is no mention of it elsewhere
        # https://www.visualstudio.com/en-us/docs/integrate/api/rm/definitions
        # https://stackoverflow.com/a/41585777
        $releaseDefinitionsUrl = "https://{0}.vsrm.visualstudio.com/{1}/_apis/release/definitions?api-version=4.0-preview" -f $account, $project
        $releaseResponse = Invoke-RestMethod -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} -Uri $releaseDefinitionsUrl -Method Get
        "  Release definitions:"
        if ($releaseResponse.count -eq 0) {
            "    No release definitions"
        }
        $releaseResponse.value | foreach {
            "    Release definition: '{0}' (id: {1})" -f $_.name, $_.id
            # Download the release definition
            # it is currently in preview only
            $updateUrl="$($_.url)?api-version=4.0-preview"
            $definition = Invoke-RestMethod -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} -Uri $updateUrl -Method Get
            
            # TODO: preview api doesn't even provide agent queue parameter yet
            "    TODO: release definition is in preview and doesn't yet provide a way to change agents. For now you'll have to manually switch them"
            
            $json = $definition | ConvertTo-Json -Depth 20
            # now upload the changed definition
            # Invoke-RestMethod -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} -Uri $updateUrl -ContentType "application/json" -Method Put -Body $json
        }
    }
    else {
        "  Could not find queue '{0}' in project '{1}'" -f $newQueueName, $project
    }
}

The script will:

Note: for now release definitions are not updated due to a lack of api surface

See here for my updated post.

tagged as Visual Studio Team Services, Build, VSTS and Azure DevOps