Updating distributed RaspberryPI’s with automatic code updates

I’m working on a project that has multiple RaspberryPI’s, distributed in multiple locations, but all running the same code (Django Channels with Python background worker scripts). I wanted to be able to deploy any updates to the main code to all the PI’s with as minimal manual interaction as possible. I couldn’t find an obvious way to do this so thought I’d summarise what I’ve set-up in the hope I may get some feedback or it would help someone else.

Basic overview of solution

The main code base is stored in a Git repository in BitBucket.

I have a development PI set-up and when I’m ready I push code changes from it to the main branch of the BitBucket repo.

Each deployed PI has an updateWorker Python script running that periodically downloads the remote code from the BitBucket repository using the git fetch command. It then checks for changes using the git status command and finally updates the code if there are changes using git reset.

Running Git from Python

To run the Git commands from my Python script I’m using a Python Package called sh. It functions as a subprocess replacement that allows you to call any program as if it were a function. I plan to write another post about using it but the basics are intuitive. For example to do a git status call on the local directory and save the response in a variable called statusCheck:

import sh
from sh import git

statusCheck = git("status")

(See this tutorial for more details on how the login to BitBucket is handled)

updateWorker Script

import sh
from sh import git
import time
import os, sys

aggregated = ""

def CheckForUpdate(workingDir):
    print("Fetching most recent code from source..." + workingDir)

    # Fetch most up to date version of code.
    p = git("--git-dir=" + workingDir + ".git/", "--work-tree=" + workingDir, "fetch", "origin", "master", _out=ProcessFetch, _out_bufsize=0, _tty_in=True)               
    
    print("Fetch complete.")
    time.sleep(2)
    print("Checking status for " + workingDir + "...")

    statusCheck = git("--git-dir=" + workingDir + ".git/", "--work-tree=" + workingDir, "status")

    if "Your branch is up-to-date" in statusCheck:
        print("Status check passes.")
        print("Code up to date.")
        return False
    else:
        print("Code update available.")
        return True

def ProcessFetch(char, stdin):
    global aggregated

    sys.stdout.flush()
    aggregated += char
    if aggregated.endswith("Password for 'https://yourrepo@bitbucket.org':"):
        print(mainLogger, "Entering password...", True)
        stdin.put("yourpassword\n")

if __name__ == "__main__":
    checkTimeSec = 60
    gitDir = "/var/testupdate/"
   
    while True:
        print("*********** Checking for code update **************")                                                     
    
        if CheckForUpdate(gitDir):
            print("Resetting code...")
            resetCheck = git("--git-dir=" + gitDir + ".git/", "--work-tree=" + gitDir, "reset", "--hard", "origin/master")
            print(str(resetCheck)) 
                                                                                                                                                                
        print("Check complete. Waiting for " + str(checkTimeSec) + "seconds until next check...", True)
        time.sleep(checkTimeSec)

updateWorker uses the –git-dir and –work-tree options to define where the local git repo is for the code.

The script parses the response from the git status command to determine if there are updates. If an update is available we will see:

On branch master

Your branch is behind ‘origin/master’ by 2 commits, and can be fast-forwarded.
(use “git pull” to update your local branch)
nothing to commit, working directory clean

On my PIs the updateWorker then kills all the running scripts which are automatically restarted by another background process, this should mean the scripts are all running up to date versions. So far it’s working well but not sure if I’m doing something crazy wrong!

Exercise is goooooood

So today was meant to be the “blueist” day of the year and to be honest when I woke up this morning I didn’t feel keen to get out of bed, let alone get working. Even after our evening meal I was still feeling a bit meh and could happily have sat on the sofa watching crap on Netflix all night. But I didn’t, I dragged myself to boxercise and put in a shift! It was super tough but I felt so good after, exercise really is a natural high. That and the sweetest most lovely people on Undateables makes the world seem a better place!

Zcash

Zcash (ZEC) is another form of crytocurrency with its main differentiation being that it provides methods to make private transactions. Personally I can’t really think of when I’d need to use this feature but from a speculation point of view I’m interested because I’ve seen how much Bitcoin and Ethereum went up in value and I can see something similar happening with Zcash.

So I’m tempted to try and buy in early and see if the value rises. But I haven’t bitten the bullet yet and I’m not entirely sure why.

I guess one reason is it’s not super easy to buy, it’s not hard but it would still take a bit more effort than going to coinbase to buy BTC or ETH and that seems to stop me.

Also whenever I look online for guides on how to buyZEC there are usually guides on how to mine it too which looks like a “free” way to get some (ignoring the cost of time and electricity) and of course I’d rather get it for free!

Another reason is I’m afraid to loose money, I wouldn’t be buying a lot but I think I’m naturally risk averse when it comes to parting with cold hard cash.

Current price at this time is $43.91, will I miss out…

Blockchains & Digital Currency

Warning, this is a bit of a brain spew!

Digital currency such as Bitcoin and Ethereum and the blockchains they are built on are exciting and interesting technologies. I’ve been following various blogs about them for some time and there are often comparrisons to the early days of the internet. If that’s the case I want to be involved! But I’m not sure of the best way. And is it going to live up to the hype?

There appear to be two obvious ways to get into it. Firstly develop using the technologies or secondly – speculate on the associated currencies/tokens. I don’t know if I have the creativity or the time to dive into the development side of things although I do find it interesting, especially Ethereum. Speculating on the currency is easy enough to do, just purchase some, but it’s really just guess work, think I need to come up with some kind of strategy.

Git calls on another directory

Today I learned that it’s possible to call git commands on a directory/repository other than the one you are currently in. For example to call git status on a repo in /var/app just type:

git --git-dir=/var/app/.git/ --work-tree=/var/app status

I used this in a Python script I’ve been writing at work that automatically checks for updates in a repo and updates the code if there are any. More on that another time.