Progress

Well that’s been a fun and productive couple of months!

ConsenSys Academy 2018

I’m now officially a ConsenSys certified dApp Developer 👊! (Certificate apparently on its way)

The ConsenSys Developer course was definitely worthwhile. I covered a lot of Blockchain theory while following the course lectures and taking the quizzes. The real learning and fun came from the final project where I actually had to build something.

Colony Hackathon Winner

I also managed to bag myself a prize in the Colony Hackathon for my decentralised issue reporting app. I got the Creativity Honorable Mention which was pretty cool and I used my winnings to buy a Devcon IV ticket ✈️ 🤘!!

The Learnings

I came across a few things that I wanted to do while I was #BUIDLING but couldn’t easily find the info on so I’ve been keeping a kind of cheat sheet. Hopefully it might help someone else out there.

https://github.com/johngrantuk/dAppCheatSheet/blob/master/README.md

The Future Is Bright

The last few months I’ve confirmed to myself that the Blockchain/Ethereum world is something I want to be involved in. There’s so many different, exciting areas to investigate further, now I just have to chose one and dive further down the rabbit hole!

Django – Custom Migrations

The Problem

I have a Django app running on Heroku. Recently I had to change one of the models to add a new ForeignKey. The app in production has data stored in the database that I want to keep and this existing data can be used to define the new ForeignKey field.

After a bit of searching I discovered I can run a custom migration to manipulate the data in the database. Before now I never really understood what migrations were really doing. As usual the Django docs are really good and for this particular case I found this How to Create Django Data Migrations blog post useful – Vitor Freitas stuff is always good.

The Solution

First I added the new field to the model:

from django.db import models

class modelToChange(models.Model):
   manualAz = models.TextField(default="0")
   manualEl = models.TextField(default="0")
   manualPol = models.TextField(default="0")
   newField = models.ForeignKey(FkModel, null=True)

Next I ran the makemigrations command. The makemigrations command is responsible for creating new migrations based on the changes you have made to your models (kind of obvious really).

$python manage.py makemigrations

This creates a new migration file in the app/migrations folder which looks like:

# -*- coding: utf-8 -*-
# Generated by Django 1.11.7 on 2018-02-08 11:57
from __future__ import unicode_literals

from django.db import migrations, models
import django.db.models.deletion

class Migration(migrations.Migration):

dependencies = [
 ('myApp', '0015_auto_20180201_0936'),
 ]

operations = [
 migrations.AddField(
   model_name='modelToChange',
   name='newField',
   field=models.ForeignKey(null=True,         on_delete=django.db.models.deletion.CASCADE, to='myApp.FkModel'),
 ),
 ]

This adds a new field, ‘newField’, to the existing model, ‘modelToChange’. The field is a ForeignKey field link to another model class called FkModel.

Normally I’d run the migrate command next. Migrate – is responsible for applying and unapplying migrations. Basically, it updates the database. In this case I want to add some custom code to the migration to update the new field using existing data.

After the additions the migration file looks like this:

# -*- coding: utf-8 -*-
# Generated by Django 1.11.7 on 2018-02-08 11:57
from __future__ import unicode_literals

from django.db import migrations, models
import django.db.models.deletion


def addCustom(apps, schema_editor):

  ExistingRecords = apps.get_model('myApp', 'modelToChange')
  ForeignKeyRecords = apps.get_model('myApp', 'FkModel')

  for message in ExistingRecords.objects.all():
    fkRecord =ForeignKeyRecords.objects.filter(id=message.id)
    
    if len(fkRecord) == 0:
      fkRecord =ForeignKeyRecords(id="unknown:" + message.id)
      fkRecord.save()
    else:
      fkRecord =fkRecord[0]

   message.fkRecord =fkRecord
   message.save()

class Migration(migrations.Migration):

  dependencies = [
    ('myApp', '0015_auto_20180201_0936'),
  ]

  operations = [
    migrations.AddField(
    model_name='modelToChange',
    name='newField',
    field=models.ForeignKey(null=True,     on_delete=django.db.models.deletion.CASCADE, to='myApp.FkModel'),
    ),

    migrations.RunPython(addCustom),
  ]

Notice the addition of migrations.RunPython(addCustom) to the Migration class and the new function, addCustom.

The last step is just to run the migrate command:

$python manage.py migrate

When the migrate command is run the addCustom function will be called. The addCustom function iterates through existing records in the database and adds the foreign key object to the existing data.

Heroku Application

It’s also possible to run custom migrations on a Heroku application. Migration files will be pushed to the app along with new code. Once this is deployed the following command can be run:

heroku run python manage.py migrate.

You can put your app in maintenance first with:

heroku maintenance:on

Django And Heroku Postgres Databases

I’m using Heroku to run one of my Django applications. The application has a Heroku Postgres Add-on for storing data. I’d like to use this ‘live’ data when I’m working on the application on my local development set-up.

I can see two options – connect directly to the live database or retrieve a copy to use locally. Using the live database directly could lead to issues if I make a mistake so I’m going with the local copy.

PostgreSQL Mac Set-up

Initially my local app was using SQLite so the first thing I need to do is get my development Mac set up for PostgreSQL. As detailed in the Heroku docs I used the Postgres.app package.

To confirm it was running ok:

$ which psql
 /Applications/Postgres.app/Contents/Versions/latest/bin/psql

And verified:

$ psql -h localhost
 psql (10.1)
 Type "help" for help.

Making A Local Database Copy

The pg:pull command can be used to pull remote data from a Heroku Postgres database to a database on the local machine. I also want to use a user and password so the command I used is:

PGUSER=postgres PGPASSWORD=password heroku pg:pull DATABASE_URL bcmlocaldb

A few things to note:

This is run from the main application directory on my local machine.
PGUSER and PGPASSWORD set the authentication credentials for the local db.
My Django app has the Database URL stored under the DATABASE_URL environment variable. The URL can be viewed with the $ heroku config command or on the Heroku dashboard if you want to use it directly.
bcmlocaldb is the name of the new local db.

Once the command has successfully run I could then see the db in the PostgresSQL dashboard on my local machine.

Django Settings To Use Local DB

Now I have a local copy of the db I just need to change the local apps settings to use this instead of the old Sqlite. I’m using the DJ-Database-URL utility to configure my environment variable so in my local .env file I changed:

DATABASE_URL=sqlite:///db.sqlite3

to:

DATABASE_URL=postgres://postgres:password@localhost:5432/bcmlocaldb

Now when I run the local application its using the new database!

Heroku has a lot of nice documentation and the Postgres info can be found here.

Alternative – Connect to live Heroku Postgres Database

To get the applications Postgres URI I just run the $ heroku config command, you will see something along the lines of:

$ heroku config

DATABASE_URL: postgres://your_username:the_password@host.amazonaws.com:5432/database

Now you just replace the local DATABASE_URL .env variable with the info above.

There’s some good documentation going into more detail about connecting from outside Heroku here.

DApp Learnings – Storing & Iterating a Collection

I’ve been working on a rock, paper, scissors Ethereum DApp using Solidity, Web3 and the Truffle framework. I hit a few difficulties trying to replicate functionality that would normally be trivial in a non blockchain world so I thought I’d share what I learned.

My first thoughts for the DApp was to display a list of existing games that people had created. Normally if I were doing something like this in Django I’d create a game model and save any new games in the database. To display a list of existing games on the front end I’d query the db and iterate over the returned collection. (I realise storage is expensive when using the Ethereum blockchain but I thought trying to replicate this functionality would make sense and would be a good place to start.)

Solidity

Structures

While investigating the various data types that could be used I found the Typing and Your Contracts Storage page from Ethereum useful. I settled on using a struct, a grouping of variables, stored under one reference.

struct Game {
   string name;
   uint move;
   bool isFinished;
   address ownerAddress;
   uint stake;
   uint index;
}

That handles one game but I want to store all games. I attempted to do this in a number of different ways but settled on mapping using the games index as the key. Every time a new game is added the index is incremented so I also use gameCount to keep count of the total games.

mapping (uint => Game) games;
uint gameCount;

struct Game {
        string name;
        uint move;
        bool isFinished;
        address ownerAddress;
        uint stake;
        uint index;
    }

To add a new game I used this function:

function StartGame(uint moveId, string gameName) payable {
      require(moveId >= 0 && moveId <= 2);
      games[gameCount].name = gameName;
      games[gameCount].move = moveId;
      games[gameCount].isFinished = false;
      games[gameCount].ownerAddress = msg.sender;
      games[gameCount].stake = msg.value;
      games[gameCount].index = gameCount;
      gameCount++;
}

I also added a function that returns the total number of games:

function GetGamesLength() public returns (uint){
   return gameCount;
}

Returning A Structure

Next I want to be able to get information about a game using it’s index. In Solidity a structure can only be returned by a function from an internal call so for the front end to get the data I had to find another way. I went with the suggestion here — return the fields of the struct as separate return variables.

function GetGame(uint Index) public returns (string, bool, address, uint, uint) {
    return (games[Index].name, games[Index].isFinished, games[Index].ownerAddress, games[Index].stake, games[Index].index);
}

Front End

On the front end I use Web3 to iterate over each game and display it. To begin I call the GetGamesLength() function. As we saw previously this gives the total number of games. Then I can iterate the index from 0->NoGames to get the data for each game using the GetGame(uint Index) function.

When my page first loads it calls:

getGames: function() {
    var rpsInstance;
    App.contracts.RpsFirst.deployed().then(function(instance) {
      rpsInstance = instance;
      return rpsInstance.GetGamesLength.call();
    }).then(function(gameCount) {
      App.getAllGames(web3.toDecimal(gameCount), rpsInstance);
    }).catch(function(err) {
      console.log(err.message);
    });
  },

Web3 – Promises, Promises & more Promises…

The getAllGames function calls GetGame(uint Index) for each game. To do this I created a sequence of promises using the method described here:

getAllGames: function(NoGames, Instance){
   
   var sequence = Promise.resolve()

   for (var i=0; i < NoGames; i++){(function(){  
         var capturedindex = i
         sequence = sequence.then(function(){
            return Instance.GetGame.call(capturedindex);
         }).then(function(Game){
            console.log(Game + ' fetched!'
            // Do something with game data. 
            console.log(Game[0]); // Name
            console.log(Game[1]); // isFinished
         }).catch(function(err){
            console.log('Error loading ' + err)
         })
      }())
   }
}

Conclusion

Looking back at this now it’s all pretty easy looking but it took me a while to get there! I’m still not even sure if it’s the best way to do it. Any advice would be awesome and if it helps someone even better.

Python & Redis PUB/SUB

I recently had an issue where I have three Python scripts that need to be started on my RaspberryPI when it boots. Script 1 is identifying what serial devices are attached to the PI, saving the information to a database. Scripts 2 and 3 need to use this information, so have to wait until script 1 has completed.

I wasn’t sure of the best way to do this but I thought of a method that lets me experiment with Redis PUB/SUB which I’ve wanted an excuse to use for a while!

My Scripts

The redis-py Python Client is well documented and really easy to use. I created a function, RedisCheck(), that Scripts 2 & 3 call when they start. The function subscribes to a Redis channel called ‘startScripts’. It then goes into a loop until it receives a ‘START’ message on that channel. Once this is received the main scripts can continue with their primary jobs.

Script1 is very simple, the ‘START’ message is PUBLISHED via the ‘startScripts’ channel once the main work is done.

The following code should demonstrate how easy the code was to write and it works really well.

RedisCheck() SUB for Scripts 2 & 3

Script 1 PUB START

Publishing Via The Command Line

Another nice thing I found was how easy it was to PUBLISH the ‘START’ message using the redis-cli. All I do is run:

# redis-cli

> PUBLISH startScripts START

This is really useful if I’m debugging and so easy to do. Overall I really like Redis.

Python Map Plotting Using Cartopy

Recently I’ve been using Python and Cartopy to plot some Latitude/Longitude data on a map. Initially it took some time to figure out how to get it to work so I thought I’d share my code incase it was useful.

According to the Cartopy intro it is

“a Python package designed to make drawing maps for data analysis and visualisation as easy as possible.”

I’m not sure how active the project is and I found the documentation a bit lacking but once I was up and running it was pretty easy to use and I think the results look pretty good.

Plotting My Data

I have a csv file with various data timestamped and saved on each line. For this case I was interested in the lat/lng location, signal strength (for an antenna) and also a satellite number. An example of one line of data is:

2017–07–10 22:31:59:203,Processing UpdatePacket: [‘:’, ‘1’, ‘0’, ‘0’, ‘1’, ‘0’, ‘0’, ‘1.63’, ‘17.15’, ‘246.57’, ‘114.11’, ‘57.008263’, ‘-5.827861’, ‘310.00’, ‘1’, ‘NAN’, ‘0’, ‘2’, ‘0’, ‘c\n’]

and from that the information I require is:

lat/lng position: 57.008263,-5.827861
signal strength: 1.63
satellite number: 310.00

Initially for each lat/lng position I wanted to plot the point on a map and colour the marker at that point to show which satellite number it was. Also if the signal strength was -100 the marker colour should be shown as red. An example taken from some of the data is shown below.

The following Gist shows the Python script I used:

Script Details

Most of the script is actually concerned with reading the file and parsing the relevant data. The main plotting functionality is in the section:

ax = plt.axes(projection=ccrs.Mercator()) # Map projection
ax.coastlines(resolution=’10m’)           # Adds coastline to map at highest resolution

plt.scatter(lngArr, latArr, s=area, c=satLngArr, alpha=0.5, transform=ccrs.Geodetic())                # Plot
plt.show()

The projection sets the coordinate system and is selected from the Cartopy projection list (there’s a lot to pick from and I chose the one I thought looked the best).

Next a coastline is added to the projection. As I was focusing on a small section of Scottish coastline I went with the 10m resolution which is the highest but lower resolutions can be selected as detailed in the documentation.

Finally a scatter plot is created. The data has been parsed into equal sized lists of longitude and latitude points.

The ‘s’ parameter defines the size of the marker at each point, in this case all set to 1pt radii.

The ‘c’ parameter defines the colour of the marker, in this case blue for satellite 310, green for 60, yellow for 302, black for any other satellite and red if signal strength is -100.

Finally the transform=ccrs.Geodetic() sets the lat/lng coordinate system as defined here.

Scaling Marker Size

It’s also possible to adjust the radius of the marker at each point. To scale it relative to the signal strength (I removed the -100 strengths):

area = np.pi * (strengthNpArray)**2

Which gives:

DApp – My First Solidity Contract

As explained in my previous post my first smart contract is based on the “Withdrawal from Contracts” example from the Solidity Common Patterns documentation. In this post I review some of the Solidity code I learned.

Smart Contract Intro

The example in the documentation was inspired by the King of the Ether game and the rules of the contract are:

To play a user submits a value of X Ether paid from his address
If X > historicHighest:
user is king
old king can collect previous payments
historicHighest = X

The contract itself is:

pragma solidity ^0.4.2;

contract WithdrawalContract {
    address public richest;
    uint public mostSent;
    mapping (address => uint) pendingWithdrawals;

    function WithdrawalContract() payable {
        richest = msg.sender;
        mostSent = msg.value;
    }

    event BecameRichestEvent(address sender, uint amount);
    event FailedRichestEvent(address sender, uint amount);
    event Collected(uint amount);

    function BecomeRichest() payable returns (bool) {

        if (msg.value > mostSent) {
            pendingWithdrawals[richest] += msg.value;
            richest = msg.sender;
            mostSent = msg.value;
            BecameRichestEvent(msg.sender, msg.value);
            return true;
        } else {
            FailedRichestEvent(msg.sender, msg.value);
            return false;
        }
 }

function withdraw() {
    uint amount = pendingWithdrawals[msg.sender];
    // Remember to zero the pending refund before
    // sending to prevent re-entrancy attacks
    pendingWithdrawals[msg.sender] = 0;
    msg.sender.transfer(amount);
    Collected(amount);
 }
}

Solidity Specifics

I found this Introduction to Smart Contracts documentation a good source of information.

Addresses – address public richest;

Type address — holds a 20byte Ethereum address. Here it’s used to save the address of the user who is currently the richest.

public allows other contracts to access the variable.

An address has various member functions as detailed here. One of these is address.transfer(uint256 amount) which sends the given amount of Wei to the address. In this case transfer is used in the Withdraw function to safely send Ether to a valid users address.

`Mappings - mapping (address => uint) pendingWithdrawals;`

The next line, mapping (address => uint) public balances; also creates a public state variable, but it is a more complex datatype.

Mappings can be seen as hash tables. pendinWithdrawals maps addresses to unsigned integers. For example:

pendingWithdrawals[0xc7975434577b0d57248e1de03fcc6f27cd6cc742] = 7;

Maps the value 7 to address 0xc7975434577b0d57248e1de03fcc6f27cd6cc742 and can be called using:

x = pendingWithdrawals[0xc7975434577b0d57248e1de03fcc6f27cd6cc742]; 

x = 7

Transactions & Constructors – function WithdrawalContract() payable

This is a constructor – a function that is run when the contract is first created.

The payable modifier means the function is a transaction. A transaction is the transfer of data between two accounts (good explanation here). A transaction contains some standard data e.g. sender address or the amount of wei being transferred. We can access the transaction data using the global msg variable.

In this case msg is used to set the initial richest address to msg.sender which is the address of the person deploying the contract. It also saves the mostSent as the msg.value which is the number of wei sent during the deployment.

Events – event BecameRichestEvent(address sender, uint amount);

The front end of the application, in my case Web3.js, listens for Solidity events being fired so it can update the interface. For example the BecameRichest event fires when a new account becomes the richest.

The event is fired by the line:

BecameRichestEvent(msg.sender, msg.value);

The listener will have access to the sender and value data passed to the call.

In the next post I plan to detail some of the front end functionality, including how to listen for events.

Ethereum Work – Beginners Sending & Receiving Eth

Intro

I’ve been following the Status project for some time and recently they announced their virtual hackathon. I’d been planning on diving into the Ethereum/Decentralised App development world and thought this would be an ideal time to start and I pretty quickly saw how powerful and relatively simple this can be.

As a complete beginner to working with the blockchain I found it better to begin by forgetting about the Status part and even the Dapp part and just see how to interact with a blockchain. Basically you can send some Eth between accounts using one function – that’s impressive.

Setup

I figured you need to get set-up with three things to get going:

1. A blockchain

Normally this would be the Ethereum blockchain but whenever you run a smart contract on Ethereum you must pay for the computation using gas. Testnets simulate the Ethereum network on you local machine and allow you to experiment for free. I installed testrpc.

2. web3.js

web3.js is a Javascript library which runs in a browser, connects to a local blockchain node and allows you to make calls on the blockchain. It does this using remote procedure calls (RPC) which basically means passing messages in JSON format.

3. Truffle

Truffle is a development environment for Ethereum that makes development a lot easier. For this exercise I’m just using the Truffle console to interact with web3.js manually. It’s way more powerful than that, which I hope show during some more posts in the future.

The Action

Start the testnet

To begin the testrpc testnet needs to be started by entering the following command:

testrpc -p 8546

This just means a simulated Eth client is listening on my laptops IP address on port 8546.

When it starts gives a list of test wallets that are loaded with 100Eth (unfortunately not real Eth, just Eth for the simulated environment). These are handy for experimenting with. My output showed:

Available Accounts
 ==================
 (0) 0xc7975434577b0d57248e1de03fcc6f27cd6cc742
 (1) 0x74dcfef837aff77d7c43d44d1d69b1eef5d708a6
 (2) 0xac63a31fd24d692997e577c02524f81eb223cd1d
 (3) 0x86fd495d744d13d43d00aa700dc9517a92eb9eae
 (4) .......

Run web3 commands

Truffle has a console tool that can be used to run web3 commands. To start it just type (keep testrpc running in separate window):

truffle console

Then try:

web3.eth.accounts

you should see a list of the same accounts you saw when you started testrpc.

Send send send

So now the really cool part. To transfer Eth between accounts all you need to enter is:

var sender = web3.eth.accounts[0];

Which shows the first address in your list, in my case: ‘0x72ddd46949fddd6add06d99a3f64357a18506470′

var receiver = web3.eth.accounts[1]

Which shows the second address in your list, in my case:

‘0x05fc7ea9e73061234d45381e8cc2e1995d4d02da’

To show the current balances in both accounts:

web3.eth.getBalance(sender)

web3.eth.getBalance(receiver)

These should both return,

{ [String: '100000000000000000000'] s: 1, e: 20, c: [ 1000000 ] }

which shows 100Eth in both accounts.

We’ll transfer 0.01Eth:

var amount = web3.toWei(0.01, 'ether')

And the actual transfer (this could have been run by itself right from the start but the above steps make the process a bit clearer):

web3.eth.sendTransaction({from:sender, to:receiver, value: amount})

This returns a transaction address like: ‘0x9e9963f4782b4248b2cd42d8af17e85d64d049e052da335e3517f4a8a69aa064’

To confirm the transfer we can check the balances again:

web3.eth.getBalance(sender) this time shows:
{ [String: ‘99989999999999979000’] s: 1, e: 19, c: [ 999899, 99999999979000 ] }
web3.eth.getBalance(receiver) this time shows:

{ [String: ‘100010000000000000000’] s: 1, e: 20, c: [ 1000100 ] }

So 0.01Eth has been transferred!

If you look in your testrpc window you should see the transaction information on the blockchain, something like:

0x9e9963f4782b4248b2cd42d8af17e85d64d049e052da335e3517f4a8a69aa064
 Gas usage: 0x5208
 Block Number: 0x01
 Block Time: Tue Jun 06 2017 19:34:54 GMT+0100 (BST)

So with as little as one command you can transfer value electronically, cryptographically proven and with minimal fee. From a pure tech geek point of view that’s awesome, no wonder people get excited about this stuff!

Antenna Arrays And Python – The Array (finally!)

As mentioned in my intro post an array antenna is a set of individual antennas connected to work as a single antenna. So far we’ve covered the individual antennas, i.e. the square patches, now it’s time to look at how they can be connected to work together.

Array Factor Fun

You can’t get far digging into arrays before you come across the Array Factor. It looks complicated but to me the easiest way to think of it is that it combines every elements position, radiating amplitude and radiating phase to give the overall array performance.

The Array Factor demonstrates that by altering an element, such as its position or phase, we can alter the arrays properties. For example the arrays beam could be steered to a desired position by altering the phase of each element.

The Array Factor is given by:

Where:

θ, φ = Direction from origin

N = number of elements

An= Amp of element

βn = Phase of element in rads

k0 = 2π/λ rads/m

And:

is the relative phase of incident wave at element n located at xn, yn, zn.

Array Factor in Python

The script below shows how easy it is to calculate the Array Factor in Python:

Array Radiation Pattern, Directivity & Gain

The above Array Factor equation is independent of each elements individual radiating pattern. The overall radiation pattern of an array is determined by the array factor combined with the radiation pattern of each element, Fn(θn, φn), giving:

The overall radiation pattern results in a certain directivity and thus gain linked through the efficiency as discussed previously.

Some Examples

To demonstrate the effects the individual element patterns have on the overall array performance we can investigate some examples using Python.

Isotropic antenna elements

In this case each element radiates equally in all directions so Fn(θn, φn) is the same for each θn, φn. The antenna radiation pattern is now just the Array Factor as described in the code above.

Patch antenna elements

Using elements described by the PatchFunction discussed previously:

PatchFunction(θ, φ, Freq, 10.7e-3, 10.47e-3, 3e-3, 2.5)

Python script for patch array:

Horn antenna elements

And finally using horn antenna elements represented by cos²⁸(θ) function.

Python script for horn array:

Next time we can see how to calculate an elements phase to steer the beam.

Antenna Arrays And Python – Patch Efficiency & Gain

Last post I dealt with antenna directivity, this post discusses antenna gain which is closely related.

Gain combines an antennas directivity and efficiency to describe how good it is at sending/receiving power in a direction. This is useful for things like link budget analysis which basically calculates if two antennas can communicate.

Antenna Gain, G, can be calculate from the directivity, D, and antenna efficiency, εr by:

The efficiency of an antenna describes how well the input power is radiated by the antenna (and due to reciprocity receive efficiency is the same as transmit efficiency):

A high efficiency antenna will radiate most of the input power while a low efficiency antenna will lose a lot of the input power before it is radiated due to things like dielectric loss, impedance mismatch, etc.

Patch Efficiency

The Python script at the bottom of the post is based on the ArrayCalc calc_patchr_eff.m file and defines a function, CalculatePatchEff, that calculate the efficiency of a rectangular patch based on the patch dimensions and materials. The comments provide some example material properties and the main function has some examples that demonstrate the effects of the material selection on the efficiency:

FR4 Patch, 14GHz, Efficiency = 47.27%

RO4350 Patch, 14GHz, Efficiency = 62.32%

Python Script