.Net, Azure and occasionally gamedev

Backing up VSTS repositories

2017/08/21

Each VSTS instance provides the ability to freely host a multitude of git repositories in various so called "projects".

Of course VSTS itself could always go offline or lose data at any point (a small but yet possible risk, it already happened at Gitlab).

To be prepared for such a disaster I like to have multiple backups of my repositories. All my public code is already mirrored between my gitlab and a VSTS project I named "Gitlab".

This owever still leaves all my private code vulnerable as I didn't clone it to Gitlab.

Most git repositories I actively work on are technically backed up because I have them "git cloned" to my laptop/desktop but this isn't the case for older stuff (WP7 apps) or prototypes/samples I built long ago and I don't want to rely on it either.

In order to back them up, I decided to write a small script that would do a full git clone of each repository and store it locally (specifically I store all of them in an encrypted VeraCrypt container and then backup the container file) to my various backup locations (one locally, one on OneDrive and one off-site).

That way I not only have a backup incase VSTS is unreachable or loses my data but thanks to git working on a filesystem level I can even use the backup as a "server" to fetch/pull from.

This already came in handy last week in italy when I had virtually no internet available (shitty wifi + only edge connection on mobile) but needed access to an older prototype.

I was able to simply mount the VeraCrypt container (which I stored locally) and had immediate access to all my git repositories. I was able to do "git clone G:\my-old-prototype" and all the code was available instantly.

Next up all I had to do was to pointed Visual Studio's nuget package manager to "C:\Users%username%.nuget\packages" (nuget keeps all packages you downloaded at some point in this directory) and thus I was able to "restore" Nuget packages from said cache without needing an internet connection. (The cache is a wooping 6GB on my laptop with ~600 distinct packages being cached in various versions each).

As such I was able to clone an old prototype and get it running without any internet connectivity.

Here's the full script I used to backup my VSTS repositories:

# The token must have at least code read access rights
$token = "Enter-your-token-here"
# Your VSTS account name (e.g. "marcstan" if your domain is "marcstan.visualstudio.com"
$account = "Enter-your-VSTS-name-here"

$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f "",$token)))

$api1 = "https://{0}.visualstudio.com/_apis/projects?api-version=3.0" -f $account
$projectResponse = Invoke-RestMethod -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} -Uri $api1 -Method Get
$projectResponse.value | foreach {
    # Projects; each may have multiple git repos
    $project = $_.name
    $api2 = "https://{0}.visualstudio.com/{1}/_apis/git/repositories?api-version=3.0" -f $account, $project
    $repoResponse = Invoke-RestMethod -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} -Uri $api2 -Method Get
    $repoResponse.value | foreach {
        $repo = $_.name
        $repoUrl = "https://{0}.visualstudio.com/{1}/_git/{2}" -f $account, $project, $repo
        $target = "{0}\{1}\{2}" -f $PSScriptRoot, $project, $repo
        "Now working with {0}" -f $target
        if (Test-Path -Path $target)
        {
            "Now updating existing repo: {0}\{1}" -f $project, $repo
            cd $target
            # fixes non-errors being displayed as errors as per https://stackoverflow.com/a/16334189
            &cmd /c "`"git fetch --all 2>&1`""
            cd $PSScriptRoot
        }
        else
        {
            $clone = "git clone {0} {1} --bare" -f $repoUrl, $target
            "Now cloning new repo: {0}\{1}" -f $project, $repo
            # fixes non-errors being displayed as errors as per https://stackoverflow.com/a/16334189
            &cmd /c "`"$clone 2>&1`""
        }
    }
}

The script will:

In order to create the token all you have to do, is go to https://yourdomainname.visualstudio.com/_details/security/tokens click add and select at least "code (read)", set an expiration date and then copy the token.

The script is very barebone (copies everything blindly) and will fail if you edited (--force) your git history (which you shouldn't do anyway).

For me, this is sufficient as a backup mechanism, as I just have it run bi-weekly to download all repos and if I ever edit my git history I can just manually delete the affected directories (which causes them to be redownloaded fully).

tagged as Visual Studio Team Services, Git, VSTS and Azure DevOps