A Muddy DeFi World

Introduction

This is a write up for my entry to the EthGlobal Autonomous Worlds hackathon, the imaginatively titled: MUD Powered Balancer Swaps. (Github)

Unfortunately I have no game development skills so the idea was to see how MUD could be used with an existing DeFi protocol. In this case by creating a new Balancer Relayer that was integrated with MUD and a front end to show swap data.

From the MUD docs: “MUD is a framework for ambitious Ethereum applications. It compresses the complexity of building EVM apps with a tightly integrated software stack.” The stand out for me is:


No indexers or subgraphs needed, and your frontend is magically synchronized!

Link

Since my early days at Balancer the Subgraph has been one of the main pain points I’ve come across. I’ve always thought there’s a clear need/opportunity for a better way of doing things. When I first saw the DevCon videos showing how the MUD framework worked it reminded me of the early days of the Meteor framework which was seemed like magical frontend/backend sync technology when I first saw it. With MUD we also get the whole decentralised/composability aspect too. It really seems like this could be a challenger and the hackathons a perfect way to get some experience hacking on it!

Solution Overview

Balancer Relayers are contracts that allow users to make calls to the Balancer Vault on behalf of the users. They can use the sender’s ERC20 vault allowance, internal balance, and BPTs on their behalf. As I’ve written before, multiple actions such as exit/join pools, swaps, etc can be chained together, improving the UX.

It’s important to note that because the Relayers have permissions over user funds they have to be authorized by the protocol. This authorisation is handled by Balancer Gorvernance and you can see a past governance proposal and authorisation PR here and here.

The MUD Store is the onchain database that can be written and read from similar to a normal database. The MUD framework handles all the complexity and makes developing with the Store super smooth.

By developing a new MUD enabled Relayer we can use a well established, battle tested Balancer protocol (Balancer 80/20 pools in particular could be interesting as liquidity for gaming assets) combined with all the benefits the MUD framework offers.

The How

Mainnet Forking

By using a local node forked from mainnet we can use all the deployed Balancer info including the real pools, assets and governance setup. To build this into the dev setup based off the MUD template project I added a .env with a mainnet archive from Alchemy and edited the root package.json` node script like so:

"node": "anvil -b 1 --block-base-fee-per-gas 0 --chain-id 31337 --fork-block-number 17295542 -f $(. ./.env && echo $ALCHEMY_URL)"

Now when the pnpm dev command is run it spins up a forked version of mainnet (with a chainId of 31337 which makes everything else keep working) and all the associated MUD contracts used during the normal dev process will be deployed there for use.

Relayer With MUD

The most recent Balancer Relayer V5 code can be found here. In the Hackathon spirit I decided to develop a very simple (and unsafe) version (I initially tried replicating the Relayer/Library/Multicall approach used by Balancer but had issues with proxy permissions on the store that I didn’t have time to solve). It allows a user to execute a singleSwap. The complete code is shown below:

import { System } from "@latticexyz/world/src/System.sol";
import { Swap } from "../codegen/Tables.sol";
import { IVault } from "@balancer-labs/v2-interfaces/contracts/vault/IVault.sol";
import "@balancer-labs/v2-interfaces/contracts/standalone-utils/IBalancerRelayer.sol";

contract RelayerSystem is System {
  IVault private immutable _vault;

  constructor() {
    _vault = IVault(address(0xBA12222222228d8Ba445958a75a0704d566BF2C8));
  }

  function getVault() public view returns (IVault) {
    return _vault;
  }

  function swap(
        IVault.SingleSwap memory singleSwap,
        IVault.FundManagement calldata funds,
        uint256 limit,
        uint256 deadline,
        uint256 value
    ) external payable returns (uint256) {
        require(funds.sender == msg.sender || funds.sender == address(this), "Incorrect sender");
        uint256 result = getVault().swap{ value: value }(singleSwap, funds, limit, deadline);
        bytes32 key = bytes32(abi.encodePacked(block.number, msg.sender, gasleft()));
        Swap.set(key, address(singleSwap.assetIn), address(singleSwap.assetOut), singleSwap.amount, result);
        return result;
  }
}

I think the simplicity of the code snippet really demonstrates the ease of development using MUD. By simply inheriting from the MUD System I can read and write to the MUD Store. In this case I want to write the assetIn, assetOut, amount and result for the trade being executed into the Swap table in the store where it can be consumed by whoever (see the Front End section below to see how). I do this in:

Swap.set(key, address(singleSwap.assetIn), address(singleSwap.assetOut), singleSwap.amount, result);

To setup the Swap table all I have to do is edit the mud.config.ts file to look like:

export default mudConfig({
  tables: {
    Swap: {
      schema: {
        assetIn: "address",
        assetOut: "address",
        amount: "uint256",
        amountReturned: "uint256"
      }
    }
  },
});

The rest (including deployment, etc) is all taken care of by the framework 👏

Permissions

Before I can execute swaps, etc there is some housekeeping to take care of. Any Balancer Relayer must be granted permission via Governance before it can be used with the Vault. In practice this means that the Authoriser grantRoles(roles, relayer) `function must be called from a Governance address. By checking out previous governance actions we can see the DAO Multisig has previously been used to grant roles to relayers. Using hardhat_impersonateAccount on our fork we can send the transaction as if it was from the DAO and grant the required roles to our Relayer. In our case the World calls the Relayer by proxy so we grant the role to the world address (not safe in the real world :P).

async function grantRelayerRoles(account: string) {
    const rpcUrl = `http://127.0.0.1:8545`;
    const provider = new JsonRpcProvider(rpcUrl);
    // These are the join/exit/swap roles for Vault
    const roles = ["0x1282ab709b2b70070f829c46bc36f76b32ad4989fecb2fcb09a1b3ce00bbfc30", "0xc149e88b59429ded7f601ab52ecd62331cac006ae07c16543439ed138dcb8d34", "0x78ad1b68d148c070372f8643c4648efbb63c6a8a338f3c24714868e791367653", "0xeba777d811cd36c06d540d7ff2ed18ed042fd67bbf7c9afcf88c818c7ee6b498", "0x0014a06d322ff07fcc02b12f93eb77bb76e28cdee4fc0670b9dec98d24bbfec8", "0x7b8a1d293670124924a0f532213753b89db10bde737249d4540e9a03657d1aff"];
    // We impersonate the Balancer Governance Safe address as it is authorised to grant roles
    await provider.send('hardhat_impersonateAccount', [governanceSafeAddr]);
    const signer = provider.getSigner(governanceSafeAddr);

    const authoriser = new Contract(authoriserAddr, authoriserAbi, signer);

    const canPerformBefore = await authoriser.callStatic.canPerform(roles[0], account, balancerVaultAddr);

    // Grants the set roles for the account to perform on behalf of users
    const tx = await authoriser.grantRoles(roles, account);
    await tx.wait();
    const canPerformAfter = await authoriser.callStatic.canPerform(roles[0], account, balancerVaultAddr);
    console.log(canPerformBefore, canPerformAfter);
}

The World address is updated each time a change is made to contracts, etc so its useful to use a helper:

import worldsJson from "../../contracts/worlds.json";

export function getWorldAddress(): string {
    const worlds = worldsJson as Partial<Record<string, { address: string; blockNumber?: number }>>;
    const world = worlds['31337'];
    if(!world) throw Error('No World Address');
    return world.address;
}

The Relayer must also be approved by the user who is executing the swap. In this case I select a user account that I know already has some funds and approvals for Balancer Vault. That account must call setRelayerApproval(account, relayer, true) on the Balancer Vault.

async function approveRelayer(account: string, relayer: string) {
    const rpcUrl = `http://127.0.0.1:8545`;
    const provider = new JsonRpcProvider(rpcUrl);
    await provider.send('hardhat_impersonateAccount', [account]);
    const signer = provider.getSigner(account);
    const vault = new Contract(balancerVaultAddr, vaultAbi, signer);
    const tx = await vault.setRelayerApproval(account, relayer, true);
    await tx.wait();
    const relayerApproved = await vault.callStatic.hasApprovedRelayer(account, relayer);
    console.log(`relayerApproved: `, relayerApproved);
}

In packages/helpers/src/balancerAuth.ts there’s a helper script that can be run using pnpm auth which handles all this and it should be run each time a new World is deployed.

Front End

Disclaimer – my front-end UI is ugly and some of the code is hacky, but it works! The idea here was to just show a super simple UI that updates anytime a swap is made through our relayer.

To trigger a swap via the UI I’ve got a simple button wired up to a systemCall:

const worldSwap = async (poolId: string, assetIn: string, assetOut: string, amount: string) => {
    const rpcUrl = `http://127.0.0.1:8545`;
    const provider = new JsonRpcProvider(rpcUrl);
    // Impersonates testAccount which we know has balances for swapping
    await provider.send('hardhat_impersonateAccount', [testAccount]);
    const signer = provider.getSigner(testAccount);
    const singleSwap = {
        poolId,
        kind: '0',
        assetIn,
        assetOut,
        amount,
        userData: '0x'
    };
    const funds = {
        sender: testAccount,
        fromInternalBalance: false,
        recipient: testAccount,
        toInternalBalance: false
    }
    const limit = '0';
    const deadline = '999999999999999999';
    console.log(`Sending swap...`);
    const test = await worldContract.connect(signer).swap(singleSwap, funds, limit, deadline, '0');
    console.log(`Did it work?`)
  };

I took the approach of impersonating the test account that we previously setup the Relayer permission for to avoid the UX of approving, etc via the UI. We just submit the swap data via the worldContract which proxies the call to the Relayer.

To display the swap data from the Store I use the storeCache which is typed and reactive. A simplified snippet shows how:

import { useRows } from "@latticexyz/react";
import { useMUD } from "./MUDContext";

export const App = () => {
  const {
    systemCalls: { worldSwap },
    network: { storeCache },
  } = useMUD();

  const swaps = useRows(storeCache, { table: "Swap" });
  
  return (
    <>
      ...
      <div>Swaps:</div>
      <ul>
        {swaps.map(({ value }, increment) => (
          <li key={increment}>
            Amount In: {value.amount.toString()} Amount Out: {value.amountReturned.toString()}
          </li>
        ))}
      </ul>
    </>
  );
};

(One other hack I had to make to get it working. In packages/client/src/mud/getNetworkConfig.ts I had to update the initialBlockNumber to 17295542.)

To demonstrate the reactive nature I also added another helper script that can be used to execute a swap with a random amount (see: packages/helpers/src/worldSwap.ts). This can be run using pnpm swap and its awesome to see the UI update automatically. I also really like the MUD Dev Tools which shows the Store updating.

Composability

I think one of the most exciting and original aspects of Autonomous Worlds is the opportunities for composability. With the standardisation of data formats in the MUD Store experimentation is made easier. As an extremely basic implementation of this I thought it was cool to show how the swap data could be used in another non-defi related app like a game. In this case I implemented the famous Google Dino hopper game where a cactus is encountered whenever a swap is made. We can import the swap data as before and trigger a cactus whenever a new swap record is added. (See packages/client/src/dino for the implementation).

Although basic, hacky and ugly it demonstrates how an Autonomous World of composable games, defi and data can start to develop. The really cool thing is who knows how it takes shape! MUD is a super cool tool and I’m excited to see it develop.

Building an SDK v1.0.1-beta.13 – Typechain

Intro

The TypeChain project provides developers a tool to generate TypeScript typings for smart contracts they are interacting with. This gives all the usual benefits of Typing for example – flagging an error if you try to call a function on the smart contract that doesn’t exist.

In the SDK we were using the @balancer-labs/typechain package which aims to provide TypeChain bindings for the most commonly used Balancer contracts but we decided it would be better to remove this dependency and generate the bindings as needed. This enables us to remain up to date with new contracts (e.g. Relayers) without waiting for the package support.

Making The Changes

TypeChain is really pretty easy to use but we had to add a few additonal changes to the SDK.

ABIs
To generate the typed wrapper TypeChain uses the Smart Contract ABIs. These were added in src/lib/abi. These can be found in the balancer-v2-monorepo or even from etherscan if the contract is already deployed/verified.

Targets
TypeChain will generate appropriate code for a given web3 library. In the SDK we use ethers.js so we need to make sure the @typechain/ethers-v5 package is added to our dev dependencies. (See the other available targets here)

CLI Command
To actually generate the files we need to run the typechain command and specifify the correct target, path to ABIs, and out path. For example:

typechain --target ethers-v5 --out-dir src/contracts './src/lib/abi/Vault.json'

Will target ethers and use the Vault ABI to generate the bindings in the src/contracts dir. You can see the full CLI docs here.

Its recommended that the generated file are not commited to the codebase so we add src/contracts/ to .gitignore. And in package.json a helper is added to scripts:

"typechain:generate": "npx typechain --target ethers-v5 --out-dir src/contracts './src/lib/abi/Vault.json' './src/lib/abi/WeightedPoolFactory.json' './src/lib/abi/BalancerHelpers.json' './src/lib/abi/LidoRelayer.json' './src/lib/abi/WeightedPool.json'"

and the CI is updated to call this command post install.

Updating the code
The last change to make was removing the old package and replacing any references to it. This is almost a direct replacement and just requires updating to use the path from the new contracts path. E.g.:

// Old
import { BalancerHelpers__factory } from "@balancer-labs/typechain";
// New
import { BalancerHelpers__factory } from '@/contracts/factories/BalancerHelpers__factory';

// Example of use
this.balancerHelpers = BalancerHelpers__factory.connect(
      this.contractAddresses.balancerHelpers,
      provider
    );

Example Of The Benefits

During the updates one of the benefits was highlighted. A previous example was incorrectly calling the queryExit function on the BalancerHelpers contract. This is a function that although it is used like a view it is actually a special case that requires it to be used with an eth_call (see here for more info). This led to a Type warning when trying to access the response. After correctly updating to use a callStatic the response typing matched the expected.

// Incorrect version
const response = await contracts.balancerHelpers.queryExit(...);
expect(response.amountsIn)....
// Shows: Property 'amountsIn' does not exist on type 'ContractTransaction'.

// Correct version
const response = await contracts.balancerHelpers.callStatic.queryExit
expect(response.amountsIn)....
/*
Shows:
const response: [BigNumber, BigNumber[]] & {
    bptOut: BigNumber;
    amountsIn: BigNumber[];
}
*/

Photo by Kristian Strand on Unsplash

Building an SDK v0.1.30 – Swaps With Pool Joins & Exits

In the Balancer Smart Order Router (SOR) we try to find the best “path” to trade from one token to another. Until recently we only considered paths that consisted of swaps but the Relayer allows us to combine swaps with other actions like pool joins and exits and this opens up new paths to consider.

Pools, Actions and BPTs

Lets take a look at the humble 80/20 BAL/WETH weighted balancer pool and see some of the associated actions.

A token holder can join a Balancer pool by depositing tokens into it using the joinPool function on the vault. In return they receive a Balancer Pool Token (BPT) that represents their share in this pool. A user can join with a single token or a combination of tokens, as long as the tokens used already exist in the pool.

A BPT holder can exit the pool at anytime by providing the BPT back to the Vault using the exitPool function. And they can exit to one or a combination of the pool tokens.

In the Balancer veSystem users lock the BPT of the 80/20 BAL/WETH weighted balancer pool. This is cool because it ensures that even if a large portion of BAL tokens are locked, there is deep liquidity that can be used for swaps.

A swap against the 80/20 pool with a “normal” token swap would usually just involve swapping tokens that exist in the pool. e.g. swapping BAL to WETH. This can be achieved by calling the `Swap` function on the Balancer Vault.

We also have multihop swaps that chain together swaps across different pools, which in Balancers case is super efficient because of the Vault architeture. This can be achieved by calling the `batchSwap` function on the Vault.

BPT tokens are actually an ERC20 compatible token which means they have the same approve, transfer, balance functionality as any other ERC20. This means it can itself also be a token within another Balancer pool. This opens up a whole world of interesting use cases, like Boosted Pools. Another example is the auraBal stable pool.

Aura

There’s lots of detailed info in the veBal and Aura docs but as a quick summary:

veBAL (vote-escrow BAL) is a vesting and yield system based based on Curves veCRV system. Users lock the 80/20 BPT and gain voting power and protocol rewards.

Aura Finance is a protocol built on top of the Balancer system to provide maximum incentives to Balancer liquidity providers and BAL stakers.

auraBAL is tokenised veBAL and the stable pool consists of auraBal and the 80/20BPT. Now if a user wants to trade auraBal to Weth they can do a multihop swap like:

For larger trades this requires deep liquidity in the BPT/WETH pool, which in the Aura case hasn’t always been available. But there is another potential path, using a pool exit, that can make use of the deep liquidity locked in the 80/20 pool:

With the similar join path also being available:

Updating The Code

So we can see that adding support for these additional paths is definitely useful but it requires some changes to the existing code.

SOR Path Discovery

First we need to adapt the SOR so it considers join/exits as part of a viable path. An elegant and relatively easy to implement solution was suggested by Fernando. Some pools have pre-minted (or phantom) BPT which basically means the pool contains it’s own BPT in its tokens list. This means a swap can be used to trade to or from a pool token to join or exit, respectively. We can make the SOR consider non preminted pools in the same way by artificially adding the BPT to the pool token list.

        if (useBpts) {
            for (const pool of pools) {
                if (
                    pool.poolType === 'Weighted' ||
                    pool.poolType === 'Investment'
                ) {
                    const BptAsToken: SubgraphToken = {
                        address: pool.address,
                        balance: pool.totalShares,
                        decimals: 18,
                        priceRate: '1',
                        weight: '0',
                    };
                    pool.tokens.push(BptAsToken);
                    pool.tokensList.push(pool.address);
                }
            }
        }

We also have to make sure that each pool also has the relevant maths for BPT<>token swaps. Once these are added the SOR can create the relevant paths and will use the existing algorithm to determine the best price.

Call Construction

Paths containing only swaps can be submitted directly to the Vault batchSwap function. A combination of swaps with joins/exits can not – they have to be submitted via the Relayer multicall function. We wanted to try and keep the SOR focused on path finding so we added some helper functions to the SDK.

The first function `someJoinExit checks whether the paths returned from the SOR need to be submitted via the Vault (e.g. swaps only) or the Relayer (swaps and joins/exits). We can do this by checking if any of the hops involve a weighted pool with one of the tokens being the pool bpt. This works on the assumption that the weighted pools are not preminted.

// Use SOR to get swap information
const swapInfo = await sor.getSwaps(tokenIn, tokenOut, ...);
// Checks if path contains join/exit action
const useRelayer = someJoinExit(pools, swapInfo.swaps, swapInfo.tokenAddresses)

The second, buildRelayerCalls, formats the path data into a set of calls that can be submitted to the Relayer multicall function.

First it creates an action for each part of the path – swap, join or exit using getActions:

  // For each 'swap' create a swap/join/exit action
  const actions = getActions(
    swapInfo.tokenIn,
    swapInfo.tokenOut,
    swapInfo.swaps,
    swapInfo.tokenAddresses,
    slippage,
    pools,
    user,
    relayerAddress
  );

which use the isJoin and isExit functions:

// Finds if a swap returned by SOR is a join by checking if tokenOut === poolAddress
export function isJoin(swap: SwapV2, assets: string[]): boolean {  
  // token[join]bpt
  const tokenOut = assets[swap.assetOutIndex];
  const poolAddress = getPoolAddress(swap.poolId);
  return tokenOut.toLowerCase() === poolAddress.toLowerCase();
}

// Finds if a swap returned by SOR is an exit by checking if tokenIn === poolAddress
export function isExit(swap: SwapV2, assets: string[]): boolean {
  // bpt[exit]token
  const tokenIn = assets[swap.assetInIndex];
  const poolAddress = getPoolAddress(swap.poolId);
  return tokenIn.toLowerCase() === poolAddress.toLowerCase();
}

Then these actions are ordered and grouped. The first step is to categorize actions into a Join, Middle or Exit as this determines the order the actions can be done:

export function categorizeActions(actions: Actions[]): Actions[] {
  const enterActions: Actions[] = [];
  const exitActions: Actions[] = [];
  const middleActions: Actions[] = [];
  for (const a of actions) {
    if (a.type === ActionType.Exit || a.type === ActionType.Join) {
      // joins/exits with tokenIn can always be done first
      if (a.hasTokenIn) enterActions.push(a);
      // joins/exits with tokenOut (and not tokenIn) can always be done last
      else if (a.hasTokenOut) exitActions.push(a);
      else middleActions.push(a);
    }
    // All other actions will be chained inbetween
    else middleActions.push(a);
  }
  const allActions: Actions[] = [
    ...enterActions,
    ...middleActions,
    ...exitActions,
  ];
  return allActions;
}

The second step is to batch all sequential swaps together. This should minimise gas cost by making use of the batchSwap function. We use the batchSwapActions function to do this:

const orderedActions = batchSwapActions(categorizedActions, assets);

and it is essentially checking if subsequent swaps have the same source/destination – if they do then they can be batched together and the relevant assets and limits arrays are updated.

Each of the ordered actions are encoded to their relevant call data. And finally the Relayer multicall is encoded.

  const callData = balancerRelayerInterface.encodeFunctionData('multicall', [
    calls,
  ]);

And here’s a full example showing how the new functions can be used:

/**
* Example showing how to find a swap for a pair using SOR directly
* - Path only uses swaps: use queryBatchSwap on Vault to see result
* - Path use join/exit: Use SDK functions to build calls to submit tx via Relayer
*/
import dotenv from 'dotenv';
import { BigNumber, parseFixed } from '@ethersproject/bignumber';
import { Wallet } from '@ethersproject/wallet';
import { AddressZero } from '@ethersproject/constants';
import {
BalancerSDK,
Network,
SwapTypes,
someJoinExit,
buildRelayerCalls,
canUseJoinExit,
} from '../src/index';
import { ADDRESSES } from '../src/test/lib/constants';
dotenv.config();
async function getAndProcessSwaps(
balancer: BalancerSDK,
tokenIn: string,
tokenOut: string,
swapType: SwapTypes,
amount: BigNumber,
useJoinExitPaths: boolean
) {
const swapInfo = await balancer.swaps.sor.getSwaps(
tokenIn,
tokenOut,
swapType,
amount,
undefined,
useJoinExitPaths
);
if (swapInfo.returnAmount.isZero()) {
console.log('No Swap');
return;
}
// console.log(swapInfo.swaps);
// console.log(swapInfo.tokenAddresses);
console.log(`Return amount: `, swapInfo.returnAmount.toString());
const pools = balancer.swaps.sor.getPools();
// someJoinExit will check if swaps use joinExit paths which needs additional formatting
if (
useJoinExitPaths &&
someJoinExit(pools, swapInfo.swaps, swapInfo.tokenAddresses)
) {
console.log(`Swaps with join/exit paths. Must submit via Relayer.`);
const key: any = process.env.TRADER_KEY;
const wallet = new Wallet(key, balancer.sor.provider);
const slippage = '50'; // 50 bsp = 0.5%
try {
const relayerCallData = buildRelayerCalls(
swapInfo,
pools,
wallet.address,
balancer.contracts.relayerV3!.address,
balancer.networkConfig.addresses.tokens.wrappedNativeAsset,
slippage,
undefined
);
// Static calling Relayer doesn't return any useful values but will allow confirmation tx is ok
// relayerCallData.data can be used to simulate tx on Tenderly to see token balance change, etc
// console.log(wallet.address);
// console.log(await balancer.sor.provider.getBlockNumber());
// console.log(relayerCallData.data);
const result = await balancer.contracts.relayerV3
?.connect(wallet)
.callStatic.multicall(relayerCallData.rawCalls);
console.log(result);
} catch (err: any) {
// If error we can reprocess without join/exit paths
console.log(`Error Using Join/Exit Paths`, err.reason);
await getAndProcessSwaps(
balancer,
tokenIn!,
tokenOut!,
swapType,
amount,
false
);
}
} else {
console.log(`Swaps via Vault.`);
const userAddress = AddressZero;
const deadline = BigNumber.from(`${Math.ceil(Date.now() / 1000) + 60}`); // 60 seconds from now
const maxSlippage = 50; // 50 bsp = 0.5%
const transactionAttributes = balancer.swaps.buildSwap({
userAddress,
swapInfo,
kind: 0,
deadline,
maxSlippage,
});
const { attributes } = transactionAttributes;
try {
// Simulates a call to `batchSwap`, returning an array of Vault asset deltas.
const deltas = await balancer.contracts.vault.callStatic.queryBatchSwap(
swapType,
swapInfo.swaps,
swapInfo.tokenAddresses,
attributes.funds
);
console.log(deltas.toString());
} catch (err) {
console.log(err);
}
}
}
async function swapExample() {
const network = Network.MAINNET;
const rpcUrl = `https://mainnet.infura.io/v3/${process.env.INFURA}`;
const tokenIn = ADDRESSES[network].WETH.address;
const tokenOut = ADDRESSES[network].auraBal?.address;
const swapType = SwapTypes.SwapExactIn;
const amount = parseFixed('18', 18);
// Currently Relayer only suitable for ExactIn and non-eth swaps
const canUseJoinExitPaths = canUseJoinExit(swapType, tokenIn!, tokenOut!);
const balancer = new BalancerSDK({
network,
rpcUrl,
});
await balancer.swaps.sor.fetchPools();
await getAndProcessSwaps(
balancer,
tokenIn!,
tokenOut!,
swapType,
amount,
canUseJoinExitPaths
);
}
// yarn examples:run ./examples/swapSor.ts
swapExample();
view raw SwapExample.ts hosted with ❤ by GitHub

Photo by Jakob Owens on Unsplash

Building an SDK v0.1.24 – Balancer Relayers and Pool Migrations

What Is A Relayer?

A relayer is a contract that is authorized by the protocol and users to make calls to the Vault on behalf of the users. It can use the sender’s ERC20 vault allowance, internal balance and BPTs on their behalf. Multiple actions (such as exit/join pools, swaps, etc) can be chained together which improves the UX.

For security reasons a Relayer has to be authorised by the Balancer DAO before it can be used (see previous votes for V1 and V2) and even after authorisation each user would still be required to opt into the relayer by submitting an approval transaction or signing a message.

How It Works

Contracts

The Balancer Relayers are composed of two contracts, BalancerRelayer, which is the single point of entry via the multicall function and a library contract, such as the V3 VaultActions, which defines the allowed behaviour of the relayer, for example – VaultActions, LidoWrapping, GaugeActions.

Having the multicall single point of entry prevents reentrancy. The library contract cannot be called directly but the multicall can repeatedly delegatecall into the library code to perform a chain of actions.

Some psuedo code demonstrating how an authorisation, exitPool and swap can be chained and called via the multicall function:

const approval = buildApproval(signature); // setRelayerApproval call
const exitPoolCallData = buildExitPool(poolId, bptAmt); // exitPool call
const swapCallData = buildSwap(); // batchSwap call

const tx = await relayer.multicall([approval, exitPoolCallData, swapCallData]);

Approval

A user has to approve each Relayer before they can use it. To check if a Relayer is approved we can use hasApprovedRelayer on the Vault:

const isApprove = await vault.hasApprovedRelayer(userAddress, relayerAddress)

And we can grant (or revoke) approval for a given relayer by using setRelayerApproval:

const approvalTx = await vault.setRelayerApproval(userAddress, relayerAddress, isApprove);

A Relayer can also be approved by using the setRelayerApproval function from the BaseRelayerLibrary contract. Here a signed authorisation message from the user is passed as an input parameter. This allows an approval to be included at the start of a chain of actions so the user only needs to submit a single transaction creating a better UX.

Chained References

Output References allow the Relayer to store output values from once action which can then be read and used in another action. This allows us to chain together actions. For example we could exit a pool, save the exit amounts of each token to a reference and then do a batchSwap using the references as input amounts for each swap:

An OutputReference consists of an index and a key:

struct OutputReference {
  uint256 index;
  uint256 key;
}

Where the key is the slot the value will be stored at. Index indicates which output amount should be stored. For example if exitPool exits to 3 tokens, DAI (index 0), USDC (1), USDT (2), we would want to use index 0 to store DAI, 1 for USDC, etc.

Example Use Case – Pool Migration

Intro

Balancer aims for the best capital efficiency for LPs so it made sense to offer the option to migrate from the old “staBal3” pool consisting of DAI, USDC and USDT to a new “boosted” stable pool which is more capital efficient because it uses yield bearing assets.

To migrate between these pools would take multiple steps:

  1. unstake from staBal3 gauge → staBalBpt
  2. exitPool from staBal, staBalBpt → DAI, USDC, USDT
  3. join the bb-a-usd2 pool by using batchSwaps
    1. DAI → bbausd2Bpt
    2. USDC → bbausd2Bpt
    3. USDT → bbausd2Bpt
  4. stake bbausd2Bpt in gauge

This would be quite an ordeal for a user to do manually but the Relayer can be used to combine all these actions into a single transaction for the user.

Details

As this is a well defined one off action we decided to add this function to the SDK as a “Zap” under a Migrations module. The user can call the staBal3 function to get all the call data required to call the tx:

{ to, data } = migrations.stabal3(
  userAddress,
  staBal3Amount,
  minBbausd2Out,
  isStaked,
  authorisationSignature
);

Behind the scenes all the call data for each step is crafted and the encoded multicall data is returned:

calls = [
        this.buildSetRelayerApproval(authorisation),
        this.buildWithdraw(userAddress, staBal3Amount),
        this.buildExit(relayer, staBal3Amount),
        this.buildSwap(minBbausd2Out, relayer),
        this.buildDeposit(userAddress),
      ];

const callData = balancerRelayerInterface.encodeFunctionData('multicall', [
      calls,
    ]);

buildSetRelayerApproval allows the user to pass the approval signature if this is their first time using the relayer. This allows us to approve and execute the migration all in a single transaction.

buildWithdraw and buildDeposit handle the gauge actions. The initial call is to withdraw from the staBal gauge and the final call deposits the bbausd2 bpt into the new gauge. We withdraw directly to the Relayer address rather than the users. The gauges return the tokens to the caller, so sending them to the user costs more as we need to manually transfer them:

gauge.withdraw(amount);
// Gauge does not support withdrawing BPT to another address atomically.
// If intended recipient is not the relayer then forward the withdrawn BPT on to the recipient.
if (recipient != address(this)) {
    IERC20 bptToken = gauge.lp_token();
    bptToken.transfer(recipient, amount);
}

Skipping this has two benefits. Firstly it saves gas by avoiding an extra transfer. It also avoids approval issues as now the Relayer is just using its own funds. The final deposit uses the userAddress to send the staked tokens from the Relayer back to the user.

buildExit creates the exitPool call:

// Ask to store exit outputs for batchSwap of exit is used as input to swaps
    const outputReferences = [
      { index: assetOrder.indexOf('DAI'), key: EXIT_DAI },
      { index: assetOrder.indexOf('USDC'), key: EXIT_USDC },
      { index: assetOrder.indexOf('USDT'), key: EXIT_USDT },
    ];

    const callData = Relayer.constructExitCall({
      assets,
      minAmountsOut: ['0', '0', '0'],
      userData,
      toInternalBalance: true,
      poolId: this.addresses.staBal3.id,
      poolKind: 0, // This will always be 0 to match supported Relayer types
      sender,
      recipient: this.addresses.relayer,
      outputReferences,
      exitPoolRequest: {} as ExitPoolRequest,
    });

Output references are used to store the final amounts of each stable token received from the pool. We have precomputed the keys by using the Relayer.toChainedReference helper, like:

const EXIT_DAI = Relayer.toChainedReference('21');
const EXIT_USDC = Relayer.toChainedReference('22');
const EXIT_USDT = Relayer.toChainedReference('23');

These will be used later as inputs to the swaps.

Also of interest is the fact we set toInternalBalance to true. The Balancer V2 vault can accrue ERC20 token balances and keep track of them internally in order to allow extremely gas-efficient transfers and swaps. Exiting to internal balances before the swaps allows us to keep gas costs down.

Because we have previously exited into internal balances we also don’t have to worry about the users having previously approved the Relayer for the tokens:

if (fromInternalBalance) {
// We take as many tokens from Internal Balance as possible: any remaining amounts will be transferred.
uint256 deductedBalance = _decreaseInternalBalance(sender, token, amount, true);
// Because deductedBalance will be always the lesser of the current internal balance
// and the amount to decrease, it is safe to perform unchecked arithmetic.
amount -= deductedBalance;
}

if (amount > 0) {
token.safeTransferFrom(sender, address(this), amount);
}

so the amount will be 0 and the safeTransferFrom call will not be executed.

buildSwap – We can join bbausd2 using a swap thanks to the PhantomBpt concept so here we create a batchSwap call that swaps each stable token to the bbausdBpt and we use the output references from the exitPool call as the input amounts to the swap (which is great as we don’t need to precompute these).

const swaps: BatchSwapStep[] = [
    {
      poolId: this.addresses.linearDai2.id,
      assetInIndex: 1,    // DAI
      assetOutIndex: 2,   // bDAI
      amount: EXIT_DAI.toString(),
      userData: '0x',
    },
    {
      poolId: this.addresses.bbausd2.id,
      assetInIndex: 2,  // bDAI
      assetOutIndex: 0,  // bbausd2
      amount: '0',
      userData: '0x',
    }
    ...
    {
      poolId: this.addresses.linearUsdc2.id,
      assetInIndex: 3,  // USDC
      assetOutIndex: 4, // bUSDC
      amount: EXIT_USDC.toString(),
      userData: '0x',
    },
    ...

In the Relayer VaultActions contract we can see how the swap amounts are set to the value stored in the reference:

for (uint256 i = 0; i < swaps.length; ++i) {
	uint256 amount = swaps[i].amount;
  if (_isChainedReference(amount)) {
	  swaps[i].amount = _getChainedReferenceValue(amount); //e.g. EXIT_DAI
  }
}

And finally (😅) we use another output reference to store the total amount out of bbausd2:

const outputReferences = [{ index: 0, key: SWAP_RESULT_BBAUSD }];

This is used as an input to the final gauge deposit to make sure we stake all the BPT that we have received and that should conclude the migration! You can see this in action on a local fork (yay no real funds required!) by running the integration test here.

Conclusion

The Balancer Relayer is probably not that well known so hopefully this has given a good overview of some of its functionality and flexibility. There’s a lot of room for experimentation and improvement of UX for complex operations so its worth investigating!

Photo by Austrian National Library on Unsplash

Building an SDK 0.1.14 – Adding a Contracts module

Intro

The idea of adding this was to make accessing Balancer contracts easier for users. Normally you need to find and import ABIs and deal with deployment addresses, if we want to make it easy we should just remove that complexity.

Also we are trying to make the main SDK functions return the contract name and functions as part of the attributes returned. This means the user could then just call using something like:

const { contractName, functionName, attributes } = transactionAttributes;

sdk.contracts[contractName][functionName](attributes)

Typechain

Typechain is a package that provides TypeScript bindings for Ethereum contracts. This means functions are statically typed and there is also IDE support which makes things safer and easier to develop against.

Balancer has its own @balancer-labs/typechain package that exposes instances of the commononly used contracts. Adding this to the SDK means we can remove the need to import ABI jsons and we can now create instances of contracts by doing:

import {
    Vault__factory
} from '@balancer-labs/typechain';

Vault__factory.connect(
            this.networkConfig.addresses.contracts.vault,
            provider
        );

which will return a typed Vault contract.

Module

  • Uses BALANCER_NETWORK_CONFIG and config.network to find vault/lidoRelayer/multicall addresses.
  • Added contracts getter to SDK module:
constructor(
        public config: BalancerSdkConfig,
        public sor = new Sor(config),
        public subgraph = new Subgraph(config),
        public pools = new Pools(config),
        public balancerContracts = new Contracts(config, sor.provider)
    ) { ... }

get contracts(): ContractInstances {
        return this.balancerContracts.contracts;
    }

This can then be called like:

const vaultContract = balancer.contracts['vault'];

or:

const vaultContract = balancer.contracts.vault

which will provide typed function refs.

Tradeoffs

One interesting discussion is the trade off of using the Contracts module within other modules. As of now only the Swaps and Multicaller modules using contracts. Using the Contracts module means we either have to pass Contracts in constructor, which adds an extra step if someone want to use modules independently:

const contracts = new Contracts(config)
const swaps = new Swaps(config, contracts)

or we instantiate Contracts within the module – which ends up happening twice if we use the high level SDK function as it is also instantiated there. For now we have decided to use the Typechain factories to instantiate the contracts within the module and will revisit in future if needed.

Photo by Pablo Arroyo on Unsplash

Forking Brilliant

Houston We Have A Problem

I want to check that a transaction will work on mainnet for an account I don’t control. In this case it’s for a large LP in the Balancer staBal3 pool and I want to check they could successfully migrates their staBal3 to the new bb-a-USD using a Relayer multicall with the params created by the SDK.

This definitely isn’t the most elegant way of doing things but it works!

Whale Hunting

The first thing I need to do is to find a large staBal3 LP account and figure out their BPT balance. I can use the Balancer Subgraph to query account pool shares for the staBal3 pool. Query looks like:

query MyQuery {
  poolShares(where: {poolId: "0x06df3b2bbb68adc8b0e302443692037ed9f91b42000000000000000000000063"}, orderBy: balance, orderDirection: desc) {
    id
    balance
  }
}

I then manually worked my way down the list of addresses until I found one that was an EOA: https://etherscan.io/address/0x4086e3e1e99a563989a9390facff553a4f29b6ee and at the time of investiagation this had a BPT balance of: 10205792.037653741889318463 (a cool $10.28mil!).

Exit Stage Right

The SDK helper function (relayer.exitPoolAndBatchSwap) that creates the call data requires an input param of expectedAmountsOut which in this case represents the DAI/USDC/USDT amounts out when exiting the staBal3 pool. Because I don’t have the maths required for this exposed yet a quick way to get this is to see the output amounts using Withdraw in the UI. There’s a very nice tool that allows us to simulate this when we don’t have control of the account of interest: https://www.impersonator.xyz/

Now that I’ve got all the info required I can generate the call data by using the helper function. In this case we get an array of call data which represent an exitPool call on staBal3 followed by a batchSwap that swaps the stables received from the exit to the bb-a-USD BPT.

The Magic

Tenderly has lots of useful features including Transaction Simulations. To begin I tried simulating the multicall call on the Mainnet Relayer but the tx failed highlighting a new issue – the account hasn’t approved the Balancer Relayer. To get around this I can use a Tenderly Fork – “Forks allow you to chain simulation and test out complex scenarios with live on-chain data”. This is cool because I can now fork the chain make an approval on the relayer from the account which then allows me to succesfully simulate the multicall!

Photo by Joel Muniz on Unsplash

Building an SDK 0.1.0 – Improving SOR data sourcing

Intro

A big focus for Balancer Labs this year is to make it really easy to build on top of the protocol. To aid in that we’re putting together the `@balancer-labs/sdk npm package. As the current lead in this project I thought I’d try and document some of the work to help keep track of the changes, thought process and learning along the way. It’ll also be useful as a reminder of what’s going on!

SOR v2

Some background

We already have the Smart Order Router (@balancer-labs/sor), a package that devs can use to source the optimal routing for a swap using Balancer liquidity. It’s used in Balancers front-end and other projects like Copper and is a solver for Gnosis BGP. It’s also used in the Beethoven front-end (a Balancer friendly fork on Fantom, cool project and team and worth checking out).

The SOR is also used and exposed by the SDK. It’s core to making swaps accesible but is also used for joining/exiting Boosted Pools which uses PhantomBpt and swaps (a topic for another time I think!).

SOR Data

The diagram below shows some of the core parts of the SOR v2.

SOR v2

To choose the optimal routes for a swap the SOR needs information about the Balancer pools and the price of assets. And as we can see from the diagram the sourcing of this data is currently very tightly coupled to the SOR. Pools data is retrieved from the Subgraph and updated with on-chain balances using a multicall. And asset pricing is retrieved from CoinGecko.

Recently Beethoven experienced a pretty large growth spurt and found there were some major issues retrieving data from the Subgraph. They also correctly pointed out that CoinGecko doesn’t always have the asset pricing (especially on Fantom) and this information could be available from other sources.

After some discussions with Daniel (a very helpful dev from Beethoven) it was agreed that a good approach would be to refactor the SOR to create composability of data fetching so the user is able to have more control over where data is coming from. With this approach, the SOR doesn’t need to know anything about CoinGecko or the Subgraph and the data could now come from anywhere (database, cache, on chain, etc.), and as long as it implements the interface, the SOR will work properly.

Changes – SOR v3

I came back from Christmas break and Daniel had made all the changes – friendly forks for the win 💪! The interface changes are breaking but the improvements are worth it – SOR 3.0.0.

Config

The goal was to remove all the chain specific config from the SOR and pass it in as a constructor parameter. This helps to avoid non-scalable hard-coded values and encorages a single source of truth. It also gives more flexibility for the variables and makes the code easier to test.

There is now the SorConfig type:

export interface SorConfig {
    chainId: number;
    weth: string;
    vault: string;
    staBal3Pool?: { id: string; address: string };
    wethStaBal3?: { id: string; address: string };
    usdcConnectingPool?: { id: string; usdc: string };
}

Pool Data

The goal here is to allow for flexibility in defining where the pool data is fetched from. We define a generic PoolDataService that has a single function getPools, which serves as a generic interface for fetching pool data. This allows allow for any number of custom services to be used without having to change anything in the SOR or SDK.

export interface PoolDataService {
    getPools(): Promise<SubgraphPoolBase[]>;
}

Approaching it this way means all the Subgraph and on-chain/multicall fetching logic is removed from the SOR. These will be added to the Balancer SDK as stand-alone services. But as a simple example this is a PoolDataService that retrieves data from Subgraph:

export class SubgraphPoolDataService implements PoolDataService {
    constructor(
        private readonly chainId: number,
        private readonly subgraphUrl: string
    ) {}

    public async getPools(): Promise<SubgraphPoolBase[]> {
        const response = await fetch(this.subgraphUrl, {
            method: 'POST',
            headers: {
                Accept: 'application/json',
                'Content-Type': 'application/json',
            },
            body: JSON.stringify({ query: Query[this.chainId] }),
        });

        const { data } = await response.json();

        return data.pools ?? [];
    }
}

Asset Pricing

The goal here is to allow for flexibility in defining where token prices are fetched from. We define a generic TokenPriceService that has a single function getNativeAssetPriceInToken. Similar to the PoolDataService this offers flexibility in the service that can be used, i.e. CoingeckoTokenPriceService or SubgraphTokenPriceService.

export interface TokenPriceService {
    /**
     * This should return the price of the native asset (ETH) in the token defined by tokenAddress.
     * Example: BAL = $20 USD, ETH = $4,000 USD, then 1 ETH = 200 BAL. This function would return 200.
     * @param tokenAddress
     */
    getNativeAssetPriceInToken(tokenAddress: string): Promise<string>;
}

All the CoinGecko code is removed from the SOR (to be added to SDK). An example TokenPriceService using CoinGecko:

export class CoingeckoTokenPriceService implements TokenPriceService {
    constructor(private readonly chainId: number) {}

    public async getNativeAssetPriceInToken(
        tokenAddress: string
    ): Promise<string> {
        const ethPerToken = await this.getTokenPriceInNativeAsset(tokenAddress);

        // We get the price of token in terms of ETH
        // We want the price of 1 ETH in terms of the token base units
        return `${1 / parseFloat(ethPerToken)}`;
    }

    /**
     * @dev Assumes that the native asset has 18 decimals
     * @param tokenAddress - the address of the token contract
     * @returns the price of 1 ETH in terms of the token base units
     */
    async getTokenPriceInNativeAsset(tokenAddress: string): Promise<string> {
        const endpoint = `https://api.coingecko.com/api/v3/simple/token_price/${this.platformId}?contract_addresses=${tokenAddress}&vs_currencies=${this.nativeAssetId}`;

        const response = await fetch(endpoint, {
            headers: {
                Accept: 'application/json',
                'Content-Type': 'application/json',
            },
        });

        const data = await response.json();

        if (
            data[tokenAddress.toLowerCase()][this.nativeAssetId] === undefined
        ) {
            throw Error('No price returned from Coingecko');
        }

        return data[tokenAddress.toLowerCase()][this.nativeAssetId];
    }

    private get platformId(): string {
        switch (this.chainId) {
            case 1:
                return 'ethereum';
            case 42:
                return 'ethereum';
            case 137:
                return 'polygon-pos';
            case 42161:
                return 'arbitrum-one';
        }

        return '2';
    }

    private get nativeAssetId(): string {
        switch (this.chainId) {
            case 1:
                return 'eth';
            case 42:
                return 'eth';
            case 137:
                return '';
            case 42161:
                return 'eth';
        }

        return '';
    }
}

Final Outcome

After the changes the updated diagram shows how the SOR is more focused and less opinionated:

The plan for the Balancer front-end is to move away from using the SOR directly and use it via the SDK package. The SDK will have the data fetching functionality as serparate services (which can be used independetly for fetching pools, etc) and these will be passed to the SOR when the SDK is instantiated. BUT it’s also possible to use the SOR independendtly as shown in this swapExample.

This was a large and breaking change but with the continued issues with Subgraph and more teams using the SOR/SDK it was a neccessary upgrade. Many thanks to Daniel from the Beethoven team for pushing this through!