Blogs & Articles: Wasabigeddon 🔗 2 years ago

“In the New Testament, Armageddon is the last battle between good and evil before the Day of Judgement.” What follows is the story of four cyber attacks amid Wasabi Wallet 2.0 launch. Illustrated by DALL-E (AI) with the help of Wiesław Šoltés (human.) I also covered the topic in a talk at Mallorca Blockchain days.

Bitcoin is the single most important revolution going on right now. It is the revolution of money. Money allowed us, for the first time in human history, to abstract away the fruit of our labor to its purest form: value. Then, we could preserve it and even exchange it for other goods and services. The more securely stored our worth is and the less friction is required to exchange it, the better the quality of the money. Therefore, it’s crucial for the progress of humanity to build increasingly improving money technologies. Cryptocurrencies bring digital scarcity to a world where digital scarcity is scarce. The notion that Bitcoin is the most valued cryptocurrency in existence means the free market is predicting it’ll be the next reserve currency in the world.

With the exceptions of stability, acceptability, portability and fungibility, Bitcoin fulfils the properties of good money. I regard stability and acceptability as meta properties. As the market capitalization of a currency grows, these properties get satisfied. Portability is also something well incentivized to be taken care of. Although the challenges of making Bitcoin cheap and fast are far from trivial, the brainpower⚡ directed towards achieving these goals is extraordinary. This leaves us with fungibility as the primary property to focus. The most important thing one can choose to work on is Bitcoin’s fungibility.

Fungibility contributes to the quality of money. It tells how interchangeable and indistinguishable individual units of a money are. In his 2019 paper, economist Alastair Berg states “That the fungibility of money reduces the costs of exchange is well explored (Brunner and Meltzer, 1971; see also Banerjee and Maskin, 1996, p.958; Menger, 1892)”.

Year -2, Month -5, Day -15. By now Wasabi Wallet has been out and running for two years. We have finally arrived at the point where we could call this system “stable.” It was possible to have privacy in Bitcoin with Wasabi. However, enabling private usage of Bitcoin does not mean Bitcoin is suddenly anonymous or fungible. In order to achieve that, users must adopt it and for that to happen we need to create a frictionless user experience, where privacy is the default. We started the Wasabi Research Club, where we reviewed the Bitcoin privacy research literature. We always invited the authors and sometimes they even came. After half a year we’ve moved onto our own research, called WabiSabi. In the following two years, we implemented it along with a complete UI and UX redesign.

Year 0, Month 0, Day -15. Voilà, Wasabi Wallet 2.0 will be launched two weeks from now, on June 15, 2022. It is the state-of-the-art Bitcoin fungibility solution. While other projects working on Bitcoin privacy are building new privacy features, Wasabi Wallet 2.0 is an attempt to abstract its comprehensive privacy tooling to the background, letting the users deal with other important things happening in their lives. This makes Wasabi Wallet 2.0, the missing piece of Bitcoin: it solves its fungibility, as far as English speaker, hot desktop wallet users are concerned.

Coinjoins are the heart of Wasabi Wallet. The single most important component of the software. We’ve just finished a week of successful coinjoin testing period on the mainnet, as well as half a year coinjoin testing period on the testnet. We’ve never had such a well tested release before, not even close. Everything had to go perfectly.

Year 0, Month 0, Day 0. Day Of The Release. Wasabi Wallet 2.0 was released with no big issues. I kept repeating: “smoothest release ever.” Others kept saying “the calm before the storm,” but I am not superstitious, so I disregarded it, of course. For the first three days, everything went as planned. We were patiently waiting for sufficient number of users to upgrade to start out with a decent coinjoin liquidity.

Year 0 , Month 0, Day 3. At the dawn of the third day we reached our target, but there was a problem: coinjoins were failing. We were afraid this might happen, so while we were waiting, we set our contingency plan in motion, which was to lower the required input count progressively.

Meanwhile, we investigated the situation, put our heads together to theorize why is this happening? The most plausible explanation came when we asked: why are we failing now and weren’t when we were testing on the mainnet or on the testnet. The most plausible explanation was that our tests were not representative of the Wasabi users. Developers running multiple instances with mostly fine Internet connection is different when you want to coordinate hundreds of users over an unreliable anonymity network. Nevertheless, this did not change our plan. It was still to lower the required input count until coinjoins can run properly. We continue improving the software so over time we’ll be able to coordinate larger and larger rounds. As we kept lowering the required input count, the first coinjoin happened with 72 inputs.

Year 0 , Month 0, Day 4. We’re under attack! Coinjoins were still barely happening. This made no sense. Why much larger coinjoins were working on the testnet then? Why was it working on the mainnet during our testing then? We debugged our systems and we’ve found unusual Tor network delays and failures. Then we tested on the testnet and, to our surprise, no coinjoins happening there either. The only thing different was the network. Is there something wrong with the Tor network? Yes. The memo came in: the Tor network is under DDoS attack.

During the following days, we fine-tuned the coordinator configuration and implemented a few low-risk improvements and deployed the coordinator. This made coinjoins get going, but it wasn’t sufficient. In parallel, the work on client side reliability fixes started. We were again on the path of getting things bootstrapped, but then…

Year 0, Month 0, Day 7. New attacks! On the seventh day we received two fresh attacks: a DDoS on our website and four of us got spam attacked on their their personal emails. Luckily the former was fended off so fast it didn’t even worth a mention in itself and the latter only caused us minor inconveniences, so we continued to work on our next v2.0.1 release set to the 10th day, the 25th of June, Monday.

Year 0, Month 0, Day 10. 2nd Release Day. We’ve improved our anti-fragility to a point where we could coordinate coinjoins with even under a Tor DDoS attack. We had three hours till the release, I finally had a little time to relax, so I decided to cleanup my email spam folder, where I stumbled upon something interesting: a ransom letter. The attacker was claiming the weekend DDoS was going to be just a warmup to give us time to think. The real attack will come on Monday, coinjoins will stop and all hell will break loose. What do we do now? To release or not to release? Luckily, the attacker also told us where he’s planning to attack, so we could review our defenses and found everything to be in place, but you never know with these kinds of things. In a volatile situation, anything can happen. We went ahead with the release and prepared for the final clash with the hacker.

Year 0, Month 0, Day 11. Day of Judgement

Keeping the watch through the night we observed something unnerving in the distance. First shot fired, and then another. We didn’t know where the cyber attacks were coming from, black hat hackers were DDoS-ing us all over the place. There was smoke and blood everywhere. Two servers down, one maintainer shaking on the floor from caffeine overdose… just kidding. Sorry to disappoint, but this was pretty anticlimactic. Keeping the watch through the night we observed some unusual activity, but we did not detect an attack with absolute certainty.

Year 0, Month 0, Day 14.

A few days after the release, right when I started writing this article, we set the required input count back to 150 as we originally planned.

Secrets

Transparency is an accountability tool, best applied when groups of individuals accumulate powers over other individuals, like governments, companies and other kinds of organizations. The open source ethos emphasizes accountability through transparency and that’s the mindset I’ve grown up in. In zkSNACKs and Wasabi Wallet everything was public, not only the source code but also our security related configuration options. Nothing illustrates better our extreme adherence to transparency is the fact that we started out by livestreaming every meeting we had.

Accountability should be applied for security, not at expense of it. Places where zero day vulnerabilities might be mined from are to be taken away from prying eyes. We’ve been aware of this learning for quite a while, but now, due to us becoming a much larger target than we were previously, we actively went and removed sensitive security configuration related public conversations.

Debugging Under Fire

Two and a half years ago, simultaneously to starting the development of WW2, I adopted a strict workout routine and I have not missed a single session ever since. I did it in the cold winter; I did it in humid, tropical weather and I did it when I got sick. There was even a time when I was working out in a capsule hotel, where I wasn’t even able to stand up. However, I skipped my workout during these two weeks. These were the most intense weeks of my life. We were debugging day and night, racing against the clock. Such a high pressure, high stress, high stakes situation is not unfamiliar to seasoned software developers. Everyone will encounter these at one time. Here I’ll leave three rules for future me on how to conduct myself in such situations.

1. Sleep Management: The last thing you want is impaired judgement. Get your sleep right!

2. Questions > Hypothesis: You want to make relevant judgements. Don’t jump right to the theories. One may realize relevance through asking questions and your hypothesis must be relevant. Skip the questions at your own peril, because you will find yourself spending valuable hours working on something that does not matter.

3. Recovery vs Understanding: A common and important judgement you’ll have to make. When something goes wrong, your priority is to recover the system, but before you’d put on your operator hat and press restart, for a brief moment put on your debugger hat and do some investigation while the bug is well and alive, otherwise you might miss the chance of ever finding it.

The Machine Is Running

I am proud of how the team got a grip on the situation. Special mention for the debut of Adam and Roland, because it was the first time they encountered a debugging under fire scenario with us. They’ve grown a lot in such a short time.

Wasabi Wallet 2.0 was born in the crossfire of a cyber war. It prevailed, and WW2 coinjoins are now thriving. During these two weeks I’ve learnt to appreciate the tremendous stability work that’s been done on WW1 coinjoins and it opened my eyes to the need for doing the same for WW2 coinjoins. The future is bright, we just have a lot of work to do.