I want to check that a transaction will work on mainnet for an account I don’t control. In this case it’s for a large LP in the Balancer staBal3 pool and I want to check they could successfully migrates their staBal3 to the new bb-a-USD using a Relayer multicall with the params created by the SDK.
This definitely isn’t the most elegant way of doing things but it works!
Whale Hunting
The first thing I need to do is to find a large staBal3 LP account and figure out their BPT balance. I can use the Balancer Subgraph to query account pool shares for the staBal3 pool. Query looks like:
The SDK helper function (relayer.exitPoolAndBatchSwap) that creates the call data requires an input param of expectedAmountsOut which in this case represents the DAI/USDC/USDT amounts out when exiting the staBal3 pool. Because I don’t have the maths required for this exposed yet a quick way to get this is to see the output amounts using Withdraw in the UI. There’s a very nice tool that allows us to simulate this when we don’t have control of the account of interest: https://www.impersonator.xyz/
Now that I’ve got all the info required I can generate the call data by using the helper function. In this case we get an array of call data which represent an exitPool call on staBal3 followed by a batchSwap that swaps the stables received from the exit to the bb-a-USD BPT.
The Magic
Tenderly has lots of useful features including Transaction Simulations. To begin I tried simulating the multicall call on the Mainnet Relayer but the tx failed highlighting a new issue – the account hasn’t approved the Balancer Relayer. To get around this I can use a Tenderly Fork – “Forks allow you to chain simulation and test out complex scenarios with live on-chain data”. This is cool because I can now fork the chain make an approval on the relayer from the account which then allows me to succesfully simulate the multicall!
A big focus for Balancer Labs this year is to make it really easy to build on top of the protocol. To aid in that we’re putting together the `@balancer-labs/sdk npm package. As the current lead in this project I thought I’d try and document some of the work to help keep track of the changes, thought process and learning along the way. It’ll also be useful as a reminder of what’s going on!
SOR v2
Some background
We already have the Smart Order Router (@balancer-labs/sor), a package that devs can use to source the optimal routing for a swap using Balancer liquidity. It’s used in Balancers front-end and other projects like Copper and is a solver for Gnosis BGP. It’s also used in the Beethoven front-end (a Balancer friendly fork on Fantom, cool project and team and worth checking out).
The SOR is also used and exposed by the SDK. It’s core to making swaps accesible but is also used for joining/exiting Boosted Pools which uses PhantomBpt and swaps (a topic for another time I think!).
SOR Data
The diagram below shows some of the core parts of the SOR v2.
SOR v2
To choose the optimal routes for a swap the SOR needs information about the Balancer pools and the price of assets. And as we can see from the diagram the sourcing of this data is currently very tightly coupled to the SOR. Pools data is retrieved from the Subgraph and updated with on-chain balances using a multicall. And asset pricing is retrieved from CoinGecko.
Recently Beethoven experienced a pretty large growth spurt and found there were some major issues retrieving data from the Subgraph. They also correctly pointed out that CoinGecko doesn’t always have the asset pricing (especially on Fantom) and this information could be available from other sources.
After some discussions with Daniel (a very helpful dev from Beethoven) it was agreed that a good approach would be to refactor the SOR to create composability of data fetching so the user is able to have more control over where data is coming from. With this approach, the SOR doesn’t need to know anything about CoinGecko or the Subgraph and the data could now come from anywhere (database, cache, on chain, etc.), and as long as it implements the interface, the SOR will work properly.
Changes – SOR v3
I came back from Christmas break and Daniel had made all the changes – friendly forks for the win 💪! The interface changes are breaking but the improvements are worth it – SOR 3.0.0.
Config
The goal was to remove all the chain specific config from the SOR and pass it in as a constructor parameter. This helps to avoid non-scalable hard-coded values and encorages a single source of truth. It also gives more flexibility for the variables and makes the code easier to test.
The goal here is to allow for flexibility in defining where the pool data is fetched from. We define a generic PoolDataService that has a single function getPools, which serves as a generic interface for fetching pool data. This allows allow for any number of custom services to be used without having to change anything in the SOR or SDK.
Approaching it this way means all the Subgraph and on-chain/multicall fetching logic is removed from the SOR. These will be added to the Balancer SDK as stand-alone services. But as a simple example this is a PoolDataService that retrieves data from Subgraph:
The goal here is to allow for flexibility in defining where token prices are fetched from. We define a generic TokenPriceService that has a single function getNativeAssetPriceInToken. Similar to the PoolDataService this offers flexibility in the service that can be used, i.e. CoingeckoTokenPriceService or SubgraphTokenPriceService.
export interface TokenPriceService {
/**
* This should return the price of the native asset (ETH) in the token defined by tokenAddress.
* Example: BAL = $20 USD, ETH = $4,000 USD, then 1 ETH = 200 BAL. This function would return 200.
* @param tokenAddress
*/
getNativeAssetPriceInToken(tokenAddress: string): Promise<string>;
}
All the CoinGecko code is removed from the SOR (to be added to SDK). An example TokenPriceService using CoinGecko:
export class CoingeckoTokenPriceService implements TokenPriceService {
constructor(private readonly chainId: number) {}
public async getNativeAssetPriceInToken(
tokenAddress: string
): Promise<string> {
const ethPerToken = await this.getTokenPriceInNativeAsset(tokenAddress);
// We get the price of token in terms of ETH
// We want the price of 1 ETH in terms of the token base units
return `${1 / parseFloat(ethPerToken)}`;
}
/**
* @dev Assumes that the native asset has 18 decimals
* @param tokenAddress - the address of the token contract
* @returns the price of 1 ETH in terms of the token base units
*/
async getTokenPriceInNativeAsset(tokenAddress: string): Promise<string> {
const endpoint = `https://api.coingecko.com/api/v3/simple/token_price/${this.platformId}?contract_addresses=${tokenAddress}&vs_currencies=${this.nativeAssetId}`;
const response = await fetch(endpoint, {
headers: {
Accept: 'application/json',
'Content-Type': 'application/json',
},
});
const data = await response.json();
if (
data[tokenAddress.toLowerCase()][this.nativeAssetId] === undefined
) {
throw Error('No price returned from Coingecko');
}
return data[tokenAddress.toLowerCase()][this.nativeAssetId];
}
private get platformId(): string {
switch (this.chainId) {
case 1:
return 'ethereum';
case 42:
return 'ethereum';
case 137:
return 'polygon-pos';
case 42161:
return 'arbitrum-one';
}
return '2';
}
private get nativeAssetId(): string {
switch (this.chainId) {
case 1:
return 'eth';
case 42:
return 'eth';
case 137:
return '';
case 42161:
return 'eth';
}
return '';
}
}
Final Outcome
After the changes the updated diagram shows how the SOR is more focused and less opinionated:
The plan for the Balancer front-end is to move away from using the SOR directly and use it via the SDK package. The SDK will have the data fetching functionality as serparate services (which can be used independetly for fetching pools, etc) and these will be passed to the SOR when the SDK is instantiated. BUT it’s also possible to use the SOR independendtly as shown in this swapExample.
This was a large and breaking change but with the continued issues with Subgraph and more teams using the SOR/SDK it was a neccessary upgrade. Many thanks to Daniel from the Beethoven team for pushing this through!
₿ A from-scratch tour of Bitcoin in Python – create, digitally sign, and broadcast a Bitcoin transaction in pure Python, from scratch, and with zero dependencies.
Unbeatable Video Games – I like this and it echoes thoughts I had after watching Competitive Dog Grooming and I kind of wish I had something like that 💭
At Balancer we recently finished a pretty major release and it was probably the most stressful release I’ve been involved in with there so far. I wanted to write this as a kind of log of a cool thing I helped build but also as a reminder that its possible to get through the tough times and come succesfully out the other side!
The release was actually two pretty major things – first there was the launch of Balancers MetaStable pools, and second was a partnership with Lido. MetaStable pools allow pools of tokens with highly correlated prices, for example cDAI and DAI and Lido were making use of a MetaStable pool to launch an incentivised WETH/wstETH pool. We integrated Lido pretty deeply and the goal was to support trades in stETH by using a Relayer contract to wrap/unwrap automatically, this meant we also had to take into account the stETH/wstETH exchange rates for calcualtions and display. We also wanted to make use of the deep WETH/Stable liquidity Balancer has and allow stable <> Lido trades, this was a slight issue because it involved a two hop trade (i.e. USDC > DAI > WETH > wstETH) which the SOR doesn’t currently support so we had to add static routes for those.
There was a lot going on across all the different teams in Balancer with some fairly tight/strict deadlines. At the time (and actually still now) there’s no Product Manager at Balancer so there wasn’t really someone managing the project. Because of this it very much felt like cat herding and some of the communication could have been better. Also things were changing a lot quite close to the deadline which made things pretty stressful for someone like me who likes to try and get things done early! The last mistake was the deployments of everything happened over the weekend for a Monday launch which wasn’t ideal. These are all things that are being addressed now that we’ve had some retrospectives – Balancer always seem good at continuously trying to improve.
A lot of this probably won’t make a great deal of sense but some of the things that I touched during the work:
SOR Hardcoded paths
SOR MarketSP support
Swap amount via query
Stable pool issues due to invariant
Subgraph update and rollout
Kovan test deployment:
stETH + wstETH token on Kovan
Faucet for wstETH
Correct pools
Relayer
Relayer – authorisation and approval debug
Relayer join/exits being canned after I spent time on them,
Limits on front-end
Front-end using correct stETH/wstETH exchange
And there was a whole bunch of other stuff being worked on across other teams.
And at the end of it all we made it! The launch went ahead smoothly and on time. The Lido pools are already some of the largest in the Balancer Vault and there haven’t been any major issues so far. I certainly learned a lot and enjoyed working with some team members I haven’t been involved with before. And although it was really stressful and tough at times, the old cliche of breaking down something large into smaller pieces and just working through them is true. Phew!
Update:
Kristen, the Balancer COO, posted a tweet about Impostor Syndrome from Just Kan that I thought was pretty pertinent.
“You will always feel like you are drowning. Even when you are succeeding, it feels like you are drowning because you are constantly being forced to do something new that you haven’t had experience in. While figuring this stuff out on the fly, you’re going to feel like you’re failing and that you’re an impostor.”
😽 Example of a Flashbot bundle to rescure a cryptokitty from a compromised wallet.
💡 Another UMA post – they keep posting interesting use cases that I’m always interested to dive into but never have the time to do (I think this is maybe part of their problem). Keeping it logged here as a bookmark!
Really good advice (especially given the latest massive correction and volatility) from Ari Paul 🦉
I rarely have good ideas. To overcome this limitation, I think about one topic (like habits) for an unreasonable amount of time. Then, I revise, revise, revise until only the best stuff remains. It’s slow, but it works. You can either be a genius or you can be patient.
😎 And this seems like something useful you could do with a node – eth_call any contract with an arbitrary code to access storage values without corresponding public views.
🧠 This was cool – post by samczsun giving a guided walkthrough for swap, the hardest challenge in Paradigm CTF 2021.
🧮 I like what @adlrocha is doing with the Improving my math intuition series and the first part about Network Applications of Bloom Filters was really interesting.
🔌 Teardown of a PC power supply – always wanted to get a breakdown of a PSU and this is a really accessible one. Better than the majority of my Uni lectures!!
⛓️ Interesting Tweet from a project dev about writing his own code rather than use TheGraph. Covalent was mentioned as alternative – get events by topic hash or get events by contract address to fetch fully decoded log events for any contract.
The Margin Loan – Interesting tradfi example of using leverage and the benefits of doing it 🤔
And a DeFi version of above – FollowTheChain ⛓️ & RicBurton demonstrating how using a MakerDAO vault to live of DAI and avoid selling your Eth during the Bear market.
Maker Vaults wouldn’t work with Liquidators 🦾 and this is a nice intro.
Have a plan! Some good advice in this tweet. Particularly like – “But who really gives a shit about the “top” if you know exactly how much you need to retire and you get there well before the “top”? You just hit that number and you exit the game with a grin…Draw up your expenses… calculate how much you’ll have leftover after you pay taxes on some huge ass crypto gains. Then, assume about 5-7% a year in income thanks to some “safe” dividend stocks, maybe an annuity and some cefi stablecoin income…” 🦉
⚠️ Interesting tweet about BTC yield. Info about risks – > 50% flash crash risk >wBTC custodial risk > AAVE risks > Yearn/Curve risks, USDT risk too using 3pool. And how to maximise %, i.e. If using AAVE you borrow TUSD at 1.5% then swap to USDT. If using Compound you borrow DAI but receive COMP for overall around 0%.
🎨 Some, cool, colourful and reasonably priced artwork.
🎸 Long live Rock And Roll! Our ‘Lost’ Weekend With Van Halen – A pretty amusing story that’s also a bit shocking in the way it shows how times have changed.
🖼️ Some more cool art – this time Crypto Art (but a horrible website!)
😱 Big liquidation event on Compound because of DAI Oracle price. Pretty scary lesson learned here.
❓ Good questions for the dinner table from Polina Marinovas newsletter – What was the most formative event in your life that you believe shaped who you are today? and What is one belief you hold that you would be afraid to share in public?
🧰 Recommended by rabmarut as an easier to use alternative to Metamask – Frame.
🏈 Some of the Huddle Up posts are really good. “Equity Deals Only” about DeAndre Hopkins was worth a read.
Etherscan is a really useful website but fairly recently I discovered they also have a pretty handy Developer API. I used it to analyse gas a contract was using over a period of time and thought it might be useful to record.
Getting Started
First of all an API key is required and this can be created for free by logging in and going to: https://etherscan.io/myapikey.
I’m going to paste my code with comments which should be easy enough to follow. The API covers a whole lot of other end point such as Logs, Tokens, etc that I didn’t use but the docs are good and worth a scan.
I read about Ocean Farming ages ago and haven’t been able to shake the crazy idea that this would be a cool thing to do. 🐟
🎥 I was pretty proud of this 🦚 feels like I’ve done a circle and gone from hacker to judge, was also very cool to appear on the schedule next to the big names:
Look- here’s a table covered with red cloth. On it is a cage the size of a small fish aquarium. In the cage is a white rabbit with a pink nose and pink-rimmed eyes.
On its back, clearly marked in blue ink, is the numeral 8. […] The most interesting thing here isn’t even the carrot-munching rabbit in the cage, but the number on its back. Not a six, not a four, not nineteen-point-five. It’s an eight. This is what we’re looking at, and we all see it. I didn’t tell you. You didn’t ask me. I never opened my mouth and you never opened yours.
We’re not even in the same year together, let alone the same room… except we are together. We are close. We’re having a meeting of the minds. We’ve engaged in an act of telepathy. No mythy-mountain shit; real telepathy.
Stephen King – Books are portable magic. 🧙♂️
⚖️ Clearing out some old bookmarks and this Tweet and this AMA with Bankless are from back when it all got started with Balancer!
