Note – Deploying A Contract Via A Delegate With Truffle Set-Up

This is quite random but I had to learn a few things so worth taking note.

I was recently working on a Truffle Dapp. I had to deploy one of my contracts in a roundabout way – basically one account signing it but another paying the gas (Metatransactions are basically the same). I’d never done this before. After that I still wanted my Truffle Dapp to be able to access the deployed contract but this required a bit of a tweak. So here’s how I did it.

Deploying Via A Delegate

This was done using a simple node script and the process looks like this:

  • Unlock your ‘sender’ account and ‘delegate’ account
  • Compile the smart contract to get the bytecode
    • In this case I used Truffle to compile and accessed the bytecode from the artificat file
  • Create a transaction object with the bytecode as the call data
  • Sender signs the transaction object
  • Delegate sends the signed transaction
  • Note the receipt so you have access to the deployed contract address

I’ve included the code I used for this below.

Using Truffle With The Deployed Contract

Because the contract was deployed using the script the Truffle artificats, etc don’t have the required information so my Dapp couldn’t interact. By making a few manual changes I managed to get it to work:

  • Make sure your contract has been compiled previously using Truffle it should have an artifact file.
  • Find your Truffle artificat file for the contract. It should be of form YouContract.json and is probably under a ‘contract’ folder in your Dapps project.
  • Find the “networks” section of the artificact.
  • Add a new network entry with the following info:
    • Network ID – should be the network that you deployed to.
    • Address – The address your contract was deployed to (from the receipt)
    • TransactionHash – I don’t think this is actually required but it was handy to record it.
    • My entry is shown in the gist below.
  • That’s it! Your Dapp should now work with the deployed contract.

Code Gist

https://gist.github.com/johngrantuk/453de92c28bfae6848ebe13e3e62c74f

Photo by David Travis on Unsplash

NuCypher & Proxy Re-encryption

Photo by Joshua Sortino on Unsplash

 

In April I entered (and won!) the NuCypher+CoinList hackathon. I didn’t actually know much about the NuCypher tech before I got started but once I had built my DApp it was clear this is really interesting stuff and it’s stuck with me ever since as something interesting to build on.

Proxy Re-encryption

The NuCypher solution will eventually provide a decentralised privacy infrastructure but during the hackathon I was mainly making use of a subset of the tech, Proxy Re-encryption.

Proxy re-encryption is a set of encryption algorithms that allow you to transform encrypted data. Specifically… it allows you to re-encrypt data — so you have data that’s encrypted under one set of keys, you can re-encrypt the data without de-encrypting it first, so that now it’s encrypted under a second, different set of keys —NuCypher co-founder MacLane Wilkison

So What?

To understand why this is pretty awesome imagine I have some encrypted data I want to share with Bob, what are the options to do this?

Crazy way – I just give me private encryption key to Bob (who I’m sharing the data with) who can use it to decrypt the data. But now Bob has my key and who knows where this ends up.

Inefficient way – I decrypt the encrypted data then rencrypt it using Bobs public key. This is more secure for sure but I have to do a lot more work. What if I have to do this many times? What if the encrypted data is stored and accessed over a network? Hows the information all being shared? Intensive!

How about the Proxy Re-encryption way:

With Proxy Re-encryption I encrypt the data once.

The encrypted data can be stored anywhere — Amazon, Dropbox, IPFS, etc. I only need to upload it once and provide access to the Proxy service (eventually this will be a NuCypher decentralised service)

The Proxy can rencrypt the data for anyone else I choose (provided I have their public key) efficiently and without ever having access to the decrypted data.

Bob decrypts the data using his own key and resources.

If the data I’m sharing is a stream, i.e. a Twitter feed, then I can enable/revoke decryption access whenever I want — i.e. I can stop someone seeing the data.

NuCypher will eventually provide a decentralised privacy infrastructure which will replace a centralized proxy with a decentralized network. A really good overview of the NuCypher solution is here.

Combine all this with decentralised smart contract as a source of access — very cool!

My DApp — thisfeedisalwaysforsale

My DApp was innspired by Simon de la Rouvieres This Artwork Is Always On Sale where he implements a Harberger Tax on the ownership of a digital artwork. In my app, instead of an artwork, access to a feed of data is always for sale. NuCypher is used to encrypt the data and only the current Patron can decrypt (using NuCypher) to get access. Anyone can buy this access from the current Patron for the sale price set when they took ownership. Whilst they hold ownership they pay a 5% fee to the feed owner. In the demo app the data is a Twitter like feed but the concept could be extended to have more than one Patron and could also be used for other kinds of feed data such as sensor data, camera/video feeds, music, etc.

I was super happy to get a mention in Token Economy as Stefanos favourite entry!

Thought Timeline

Photo by JESHOOTS.COM on Unsplash

I can’t believe it’s March already – it’s been a fast start to 2019!

Since my last post I haven’t come across any real technical inspiration for a new, in-depth technical article but I read this post by Fred Wilson yesterday and liked the idea of recording my thoughts as a timeline for the future.

It’s not that my interest in crypto has dissipated or that I haven’t been busy. Jan/Feb was busy with the day job and in my spare time I was focused on my entry to the 0x Hackathon. I did learn some of the details of 0x protocol but I was mainly using skills I picked up during my work on other Dapps. I didn’t feel like there was anything worth writing a detailed post about. It was fun though.

I’ve spent a lot of time digging into the world of Decentralised Finance, or DeFi. To be honest I’m still digesting the information but there seems to be a real fit with DeFi and Ethereum:

  • Programmable and transparent – organisations can unleash code on Ethereum that just runs and does what it’s meant to do. No one controls it and anyone can educated themselves about it.
  • Unregulated (So far)  – If you do educate yourself then you can access it, nobody can tell you you’re not suitable to use it if you want.
  • Efficiency – Cutting out middle men and rent takers, making things cheaper and opening up new opportunities.

It’s a whole new interesting world! I’ve been doing a lot of reading on the MakerDAO system figuring out how it works and what the opportunities are. I could probably write multiple posts about all that! Some of the areas I’ve found exciting so far:

  • Earning value on crypto assets – this is becoming the latest trend. There’s a lot of people HODLing a decent amount but at the moment it’s doing nothing. There seems to be more and more ways coming out to earn value on this.
    • Uniswap – A decentralised exchange where there is no token, no centralization, and no fees going to any of the founders. You can provide liquidity and 0.3% of all trade volume is distributed proportionally to all liquidity providers.
    • Borrowing/Lending – Dharma/Compound/MakerDAO all provide opportunities to lend assets for a fee.
  • Combining DeFi with Predicition Markets such as Augur:
  • Loaning yourself – A super interesting post about using MakerDAO and existing collateral to loan yourself $ to pay off a house at a lower interest rate than your mortgage. Kind of mind blowing BUT you do need collateral to start. Is this something that only the people who got in early can afford? SO INTERESTING!!!

Cool Crypto — Sending Value With A Link

One of the functions I found the most interesting was the ability to send some value, in this case xDai, to someone using a link (try out the app here). The functionality behind this method is pretty cool and makes use of a lot of web3/crypto fundamentals. I found it interesting to dig into it and thought it would be worth sharing.

To Begin

First of all, the Dapp isn’t really ‘sending’ the xDai, it’s more of a deposit/claim pattern with some cool cryptography used to make sure only the person with the correct information, provided by the sender via the link, can claim the value.

Secondly there are two parts — the web Dapp and the smart contract on the blockchain. The web Dapp is really just a nice way of interacting with the smart contract.

Step By Step

A step by step description helped me get it into my head. Please note that I’m not showing all the details of the code here, it’s more a high level description to show the concept.

Sending

Using the Dapp the ‘sender’ enters the amount they want to send from and hits send.

App Send Screen

Once send is hit the Dapp does a bit of work in the background to get the inputs required by the smart contract set-up.

The Dapp uses web3.js to hash some random data:

let randomHash = web3.utils.sha3("" + Math.random());

Now the Dapp uses web3.js to generate a random account which will have a private key and a public key:

let randomWallet = web3.eth.accounts.create();

The random hashed data is then signed using the random wallet private key (see below for more details on signing, etc):

let sig = web3.eth.accounts.sign(randomHash, randomWallet.privateKey);

The Dapp sends a transaction to the blockchain smart contract with the value equal to the amount that is being sent along with the signature and the hashed data:

Contract.send(randomHash, sig.signature), 140000, false, value ...
// This is just a pseudo code to give the gist, see the repo for the full code

The smart contract contains a mapping of Fund structures to bytes32 ID keys:

struct Fund {
    address sender;
    address signer;
    uint256 value;
    uint256 nonce;
    bool claimed;
}

mapping (bytes32 => Fund) public funds;

When the Dapp ‘sends’ the value the smart contract creates a new Fund structure with the signer field set to the public key of the random wallet created by the Dapp. This field is important as it is used as the check when a claim is made:

newFund = Fund({
    sender: msg.sender,
    signer: randomWallet.publicKey,
    value: msg.value,
    nonce: nonce,
    claimed: false
})

Now the newFund is mapped using the randomHash value as the key:

funds[randomHash] = newFund;

The xDai from the sender has now been sent to the smart contract and is ready to be claimed by anyone with the required information.

Finally the Dapp generates a link with the randomHash and random wallet private key as the link parameters:

Link Format: xDai.io/randomHash;privateKey

The link can then be copied and sent via WhatsApp, SMS, etc.

The Claim:

Here it’s probably worth noting the link is really just a nice way to share the important information required to claim the xDai. The Dapp also does the hard work of interacting with the blockchain smart contract.

When the link is visited the Dapp parses the randomHash and the privateKey from the link.

It then signs a message using the privateKey from the link:

let accountHash = web3.utils.sha3(claimAccount);
let sig = web3.eth.accounts.sign(accountHash, privateKey);

Now the smart contract claim function is called using the signature and the original data:

Contact.claim(accountHash, sig, randomHash, claimAccount)

The Solidity ecrecover function is used to get the public address from the signature (this is the magic, see the info below):

address signer = recoverSigner(accountHash, sig);

Finally, the smart contract checks the fund with key matching randomHash has the ‘signer’ equal to the address recovered from the signature. If it does then it can send the value to the claimers account:

if(funds[randomHash].signer == signer && funds[randomHash].claimed == false){    
    funds[randomHash].claimed = true;
    claimAccount.send(funds[randomHash].value);
}

Phew, that’s it! It feels like a lot going on but the basics is it’s a smart way for a user to store value in a smart contract that can only be claimed with the correct information without showing what that information is on the public blockchain.

The Cool Crytography

Signing, ecrecover, eh what?? There’s some things that are probably worth going into a bit more detail.

Wallets, Accounts, etc

An account generated with web3.eth.accounts.create() has it’s own a private key and public key. More info can be found in the docs and here. The private and public keys are linked through an algorithm that has signing and validation properties.

Signing & Validating

The following is a very brief summary of this helpful post.

Signing is the act of a user “signing” data that anyone can validate came from that user.

The signing function will take in a private key and the data. The output will be another string that is the signature.

To validate the signature is from the owner of the private key the signature, the original data and the public key is required.

A validator function is run that recovers the public key from the signed data.

The recovered public key is then compared to the original one and if both are the same the signature is valid.

ecrecover

In this case the Solidity ecrecover (Eliptic Curve Recover) function is used to recover the address associated with the public key. The recoverSigner function in the smart contract code shows an example of how this is done and this is a pretty decent explanation of what is going on.

I think that’s a pretty awesome example of crypto in action!

Ethereum — Scaling Is Not The Problem?

Warp Speed! Photo by chuttersnap on Unsplash

The reality is that State Channels are here now, and work great, today. We can ship commercial quality dapps that use them NOW — JezSan FunFair

The quote above was from a Reddit post highlighted in the November issue of Week In Ethereum. It confirmed something that I didn’t really appreciate until I went to Devcon IV — scaling isn’t the issue that is preventing main stream adoption of Ethereum. It’s the fact that the dApps out there don’t really offer any real improvement over the existing solutions for the main stream user.

As Jez says further down the post — State Channels are a useful and immediate form of scaling and they work today.

Both FunFair and SpankChain already have products that can scale and that in look and feel matches what’s already out there. But that’s the problem—to a normal user they are the same as what’s out there already, they’re not actually any better. In fact by the time you add in all the hassle of dealing with ‘crypto’ it’s actually worse.

It’s Just Not Good Enough

I actually experienced this first hand at the FunFair DevCon party. I took a non-crypto friend along and I explained the concept to him while we watched a demo on a big screen. He appreciated the look and quality of the demo but asked why should he chose this over any other?

Why should we use the dApp? — Photo by Ken Treloar on Unsplash

One of the FunFair developers actually tried to answer – ‘it’s provably fair’. My friend answered, quite fairly, with how can he, himself, verify that it’s fair? (I have to admit I’m fairly ok with code but I would find it hard and he has no experience) The FunFair guy admitted this is a problem.

Next suggestion was that he would always have control over his funds so there’s no issue with an operator delaying or withholding them. To my friend this just wasn’t a problem he had ever experienced so he didn’t really see it as a benefit.

I need what tokens to use it? Ok, lets just leave it there! Basically it came down to the fact that for my friend there was no real improvement and actually the whole thing was a bit of a pain in the ass to use.

It was a an eye opener for me as I’ve always been a big fan of FunFair and I really like the tech — but I guess I’m not a mainstream customer, in fact I’m not actually a customer at all! (I still reckon FunFair will crack it though 😄) When I think about in general the stuff I have used isn’t very user friendly and it’s normally just my enthusiasm for the tech that pushes me to use it.

BUIDL To The Rescue

So what to do? Well at Devcon there was a lot of focus on design and user experience so obviously smarter people realised this issue a good while before me! And smart people are experimenting and building. Universal Logins are super cool. Week In Ethereum also linked to some details about the work of Austin Griffith at Gitcoin labs. I think this is the kind of thing that’s going to drives things forward and there’s a lot of room to get involved.

Check out some of the stuff below, dive in and experiment.

My Devcon IV

The Quick & Not So Nasty Summary

It’s going to happen. Ethereum is going to scale, PoS is going to come, UI/UX is going to improve and more people will use DApps. Amazing minds are working on all these things. The researchers are super inspiring. The community is strong, dedicated and buidling away. The ship is being steered in the right direction. The combination of all these things feels a bit special and it might just be the start of a new future.

I need to contribute.

My Personal Highlights

ENS & Universal Logins — Nick Johnson, Alex Van De Sande

These kind of things are really going to drive adoption. Both Nick Johnson and Alex Van De Sandé were quality speakers.

I should look at using ENS in a DApp to gain some experience and the technical details are interesting to look into more, might be something I could contribute to?

Universal Logins, meta transactions, etc are just cool and the demo is awesome. I wonder what work is being done on the next stage? i.e. what’s the Universal Login equivalent for setting up your own wallet properly, or installing MetaMask?

Material world — Vinay Gupta

This was interesting because Mattereum seems to have solved the problem of how Ethereum tech can legally control real world assets. It still looks bloody complex but worth following for sure.

Making Sense Of Layer 2 — Josh Stark, L4

Explained state channels, plasma and other layer 2 technologies in a really good way.

One feeling I got from the conference was that some of these technologies, i.e. state channels seem usable already but don’t seem to be widely adopted to solve so called scaling issues. Is it because they’re hard to implement or is there some other reason?

Philip Daian — Smart Contract Security

Ended up at this one by accident and it was really interesting. Philip is a researcher and presented lots of interesting data. He was refreshingly original, realistic and non-biased. I’d like to watch his talk again and will definitely follow more.

CryptoEconomics at Scale — Karl Floersch

This guy was great. Energetic, enthusiastic, funny and explained things well. Very inspiring and encouraged everyone to get involved because there’s loads to do.

Andy Milenius — MakerDao, A new hope for financial reform

Did not expect much from this but it was excellent, really inspiring. These guys have a grand vision and they seem to be very clear and focused .Don’t know if I’ve ever fully appreciated the real power of stable coins but this definitely made me more aware.

Turbo geth — Alexey Akhunov

This guy was awesome! Possibly the best I saw. I only caught the end of the talk as didn’t plan to go but glad I randomly ended up there. He was so knowledgeable a fantastic presenter and it was super inspiring.

Contributing To Ethereum & OpenSource — Danny Ryan

One of the reasons I wanted to go to Devcon was because I know I want to be involved in the Ethereum ecosystem, I’ve been dabbling in the tech but I haven’t really found the best way for me to move forward so this presentation was perfect. Danny presented loads of ways to get involved and he believes it’s there are great opportunities for everyone. Some things to consider:

  • Follow conversations on https://ethresear.ch/
  • Build interfaces for research projects
  • Build prototypes for researchers
  • Get involved in meta transaction work
  • or ENS
  • Hack away and build my own applications, gain experience, skills and see if I identify any specific pain points.

So, what to do, what to do 🤔

The Experience In General

I really, really enjoyed it! It was fun, inspiring, exciting and super interesting.

Prague is awesome. A beautiful city that offers a lot of fun. We even got the weather! The Congress Centre was kind of spectacular and the food was amazing.

Prague Congress Centre

You can have a lot of fun in Prague, partying on the 24th floor of the Corinthian was FUN and Halloween is celebrated in style!

Plenty of Fun

Although there was so much good work going on at the conference there’s still this kind of out-there, wacky, punky vibe too. The keynote sing song was amusing and surprisingly The chill out zone was cool and useful to rest the head after a late one the night before!

The amount of swag was crazy, I can’t say no to a free t-shirt!

Hand Luggage Struggles

There’s so many smart, enthusiastic and decent people working on this. I kind of knew about the smart and enthusiastic side but I was pleasantly surprised at how decent the people I met were — friendly, approachable, cooperative and mostly non-egotistical. Really different from attitudes I’ve experienced elsewhere.

I need to dive in even more and start contributing.

Also-prepare for Devcon V!

Ethereum — Vyper Development Using Truffle

Why Vyper?

Vyper is a contract-oriented, pythonic programming language that targets the Ethereum Virtual Machine (EVM)

Vyper is a relatively new language that has been written with a focus on security, simplicity and audibility. It’s written in a Pythonic way which appeals to me and as a more secure alternative to Solidity I think it has a lot of potential. I plan on writing more about working with Vyper in the future.

Truffle — Too Much Of A Sweet Tooth?

I’ve recently finished working on a hackathon project and completed the 2018 ConsenSys Academy and during that time, for better or worse, I’ve become pretty accustomed to using the Truffle development environment for writing code, testing and deploying— it just makes life easier.

So, in an ideal world I’d like to use Truffle for working with Vyper. After a bit of investigation I found this ERC721 Vyper implementation by Maurelian who did the work to make it Truffle compatible. I thought it might be useful to document the build process for use in other projects.

How To — Vyper Development Using Truffle

Install Vyper

The first step is to make sure Vyper is installed locally. If this has been done before you can skip — you can check by running the $ vyper -h command. There are various ways to install, including using PIP, the docs are here. I’m using a Mac and did the following:

Set up virtual environment:

$ virtualenv -p python3.6 --no-site-packages ~/vyper-venv

Remeber to activate the environmet:

$ source ~/vyper-venv/bin/activate

Then in my working dir:

$ git clone https://github.com/ethereum/vyper.git
$ cd vyper
$ make
$ make test

Install Truper

Next I installed Truper, a tool written by Maurelian to compile Vyper contracts to Truffle compatible artifacts. It uses Vyper which is why we installed it previously. (See the next section for details of what it’s doing). To install run:

$ npm i -g truper

Compiling, Testing, Deploying

From your project dir (you can clone the ERC-721 project for a quick test).

Run ganache test network:

$ ganache-cli

Compile any Solidity contracts as usual using:

$ truffle compile

Compile Vyper contracts using the command:

$ truper
* this must be called from the project dir and you must have the virtual environment you built Vyper in running.

Truffle tests can be written and run the usual way, i.e.:

Use artifacts in test files:
const NFToken = artifacts.require('NFToken.vyper');
Run tests using:
$ truffle test

Truffle migrations also work the usual way. For example I used the following migration file to deploy to ganache:

2_deploy_contracts.js
const NFToken = artifacts.require('NFToken.vyper');
const TokenReceiverMockVyper = artifacts.require('NFTokenReceiverTestMock.vyper');
module.exports = function(deployer) {
  deployer.deploy(NFToken, [], []);
  deployer.deploy(TokenReceiverMockVyper);
};
$ truffle migrate

What’s Going On

Truper uses Vyper which is why we installed it in the first step. If we look at https://github.com/maurelian/truper/blob/master/index.js we can see Truper is creating Truffle artifact files for each Vyper contract and writing them to the ./build/contracts folder of the project.

Truffle Artifact Files

These *.json files contain descriptions of their respective smart contracts. The description includes:

  • Contract name
  • Contract ABI (Application Binary Interface — a list of all the functions in the smart contracts along with their parameters and return values). Created by Truper using: $ vyper -f json file.vy
  • Contract bytecode (compiled contract data). Created by Truper using: $ vyper -f bytecode file.vy
  • Contract deployed bytecode (the latest version of the bytecode which was deployed to the blockchain). Created by Truper using: $ vyper -f bytecode_runtime file.vy
  • The compiler version with which the contract was last compiled. (Doesn’t appear to get added until deployed.)
  • A list of networks onto which the contract has been deployed and the address of the contract on each of those networks. (Doesn’t appear to get added until deployed.)

Maurelian describes it as a hacky stop-gap but it works so thank you!

Progress

Well that’s been a fun and productive couple of months!

ConsenSys Academy 2018

I’m now officially a ConsenSys certified dApp Developer 👊! (Certificate apparently on its way)

The ConsenSys Developer course was definitely worthwhile. I covered a lot of Blockchain theory while following the course lectures and taking the quizzes. The real learning and fun came from the final project where I actually had to build something.

ConSensys Academy Final Project

My final project was a bounty DApp that allows anyone to upload a picture of an item they want identified along with an associated bounty in Eth for the best answer. I got a lot of experience using the various parts of the Web3 technology stack. I used Truffle for development/testing, IPFS for storing the pictures and data (was cool to use this, very powerful idea), uPort for identity, OpenZeppelin libraries (which are really useful) an upgradeable design pattern, deployment to Rinkeby and lots of practice securing and testing smart contracts.

Colony Hackathon Winner

I also managed to bag myself a prize in the Colony Hackathon for my decentralised issue reporting app. I got the Creativity Honorable Mention which was pretty cool and I used my winnings to buy a Devcon IV ticket ✈️ 🤘!!

The Learnings

I came across a few things that I wanted to do while I was #BUIDLING but couldn’t easily find the info on so I’ve been keeping a kind of cheat sheet. Hopefully it might help someone else out there.

https://github.com/johngrantuk/dAppCheatSheet/blob/master/README.md

The Future Is Bright

The last few months I’ve confirmed to myself that the Blockchain/Ethereum world is something I want to be involved in. There’s so many different, exciting areas to investigate further, now I just have to chose one and dive further down the rabbit hole!

Blockchain – The Adjacent Infrastructure

The Adjacent Infrastructure Movement

Chris Robisons Ethreal summary post really reinforces the feeling I get about Blockchain. He mentions how Joseph Lubin described the tech as an Adjacent Infrastructure. This really rings true to me. He goes on to say that from the ground up a new parallel society is being built and people can opt into it.  Some nice examples are given:

  • Crypto – adjacent financial industry infrastructure
  • Ujo – adjacent music industry infrastructure
  • Hoard – adjacent virtual entertainment infrastructure

It’s my first real experience of a “community”. This community is growing. There are people genuinely enthusiastic about the technology and they are willing to contribute and collaborate to drive things forward. As more and more adjacent infrastructure is built more people are going to opt in. At the same time the building blocks of the Adjacent Infrastructure are still being built. There’s loads of opportunity to get involved and help make this happen. Personally I’m looking forward to getting stuck in to the Consensys Academy in June/July and here are some of the other things that have enthused me lately!

Ideo, Designing for Blockchain: Three Ways to Get Started

Really good article centered around three Blockchain challenges that can be worked on now:

  1. Design secure and user-friendly ways to store private keys.
  2. Design better ways to help users make a decision about transaction costs, or find ways to abstract it away all together.
  3. The challenge: Design ways to display blockchain addresses in a more readable or recognizable format.

The challenges are real. People who are well versed, immersed and interested in the technology sometimes forget how difficult it can be to actually just use it. Solving these challenges would go a long way to getting more adoption and real life use. The article also provides a bit of inspiration to get started and it’s definitely thought provoking.

WalletConnect/Balance

A project I’ve enjoyed reading about lately is WalletConnect from the team at Balance. Their blog posts are great and the code is interesting and well documented. Richard Burton appears to be really open to collaboration with a genuine desire to drive the technology forward.

The WalletConnect concept is an example of something built to solve the usability challenges described in the Ideo article. The solution feels natural to me and I can definitely imagine using it. Balance themselves are in the process of developing a mobile wallet and I think this is something to look forward to.

ETHPrize

I read the introduction post to ETHPrize and it sounds like a great idea. Bounties as a driving force for development seem to be taking off.

The list of key points is really interesting. It shows how early stage this technology and the tools around it is. There’s so much opportunity to get involved and really make a difference.

I found it really weird to be inspired to do something about documentation! Not something I naturally gravitate towards but its something I know can actually make a massive difference and from experience it’s an issue in places. Seems like a great way to get involved and do valuable work for projects.

 

Django – Custom Migrations

The Problem

I have a Django app running on Heroku. Recently I had to change one of the models to add a new ForeignKey. The app in production has data stored in the database that I want to keep and this existing data can be used to define the new ForeignKey field.

After a bit of searching I discovered I can run a custom migration to manipulate the data in the database. Before now I never really understood what migrations were really doing. As usual the Django docs are really good and for this particular case I found this How to Create Django Data Migrations blog post useful – Vitor Freitas stuff is always good.

The Solution

First I added the new field to the model:
from django.db import models

class modelToChange(models.Model):
   manualAz = models.TextField(default="0")
   manualEl = models.TextField(default="0")
   manualPol = models.TextField(default="0")
   newField = models.ForeignKey(FkModel, null=True)
Next I ran the makemigrations command. The makemigrations command is responsible for creating new migrations based on the changes you have made to your models (kind of obvious really).
$python manage.py makemigrations
This creates a new migration file in the app/migrations folder which looks like:
# -*- coding: utf-8 -*-
# Generated by Django 1.11.7 on 2018-02-08 11:57
from __future__ import unicode_literals

from django.db import migrations, models
import django.db.models.deletion

class Migration(migrations.Migration):

dependencies = [
 ('myApp', '0015_auto_20180201_0936'),
 ]

operations = [
 migrations.AddField(
   model_name='modelToChange',
   name='newField',
   field=models.ForeignKey(null=True,         on_delete=django.db.models.deletion.CASCADE, to='myApp.FkModel'),
 ),
 ]

This adds a new field, ‘newField’, to the existing model, ‘modelToChange’. The field is a ForeignKey field link to another model class called FkModel.

Normally I’d run the migrate command next. Migrate – is responsible for applying and unapplying migrations.  Basically, it updates the database. In this case I want to add some custom code to the migration to update the new field using existing data.

After the additions the migration file looks like this:

# -*- coding: utf-8 -*-
# Generated by Django 1.11.7 on 2018-02-08 11:57
from __future__ import unicode_literals

from django.db import migrations, models
import django.db.models.deletion


def addCustom(apps, schema_editor):

  ExistingRecords = apps.get_model('myApp', 'modelToChange')
  ForeignKeyRecords = apps.get_model('myApp', 'FkModel')

  for message in ExistingRecords.objects.all():
    fkRecord =ForeignKeyRecords.objects.filter(id=message.id)
    
    if len(fkRecord) == 0:
      fkRecord =ForeignKeyRecords(id="unknown:" + message.id)
      fkRecord.save()
    else:
      fkRecord =fkRecord[0]

   message.fkRecord =fkRecord
   message.save()

class Migration(migrations.Migration):

  dependencies = [
    ('myApp', '0015_auto_20180201_0936'),
  ]

  operations = [
    migrations.AddField(
    model_name='modelToChange',
    name='newField',
    field=models.ForeignKey(null=True,     on_delete=django.db.models.deletion.CASCADE, to='myApp.FkModel'),
    ),

    migrations.RunPython(addCustom),
  ]

Notice the addition of migrations.RunPython(addCustom) to the Migration class and the new function, addCustom.

The last step is just to run the migrate command:

$python manage.py migrate

When the migrate command is run the addCustom function will be called. The addCustom function iterates through existing records in the database and adds the foreign key object to the existing data.

Heroku Application

It’s also possible to run custom migrations on a Heroku application. Migration files will be pushed to the app along with new code. Once this is deployed the following command can be run:

heroku run python manage.py migrate.

You can put your app in maintenance first with:

heroku maintenance:on