Blogs & Articles: The Status of Proof of Reserve as of Year End 2022 🔗 1 year ago

A bright spot in an otherwise bad year

Image courtesy of BitMEX Research

It’s undeniable that momentum around cryptographic Proofs of Reserve is gathering. This is a genuine silver lining from the FTX debacle. We may well yet emerge from this crisis with a major step forward in exchange credibility. But there remains a huge amount of confusion around Proofs of Reserve, and some of the exchanges taking their first steps towards the procedure have plenty of work to do.

In the last two months, major exchanges Kraken, BitMEX, OKX, and Binance, among others, have all published attestations they call Proof of Reserve, although these don’t all grant the same assurances. Coinbase also published a blog post explaining their status as a public company with audited financials and quarterly disclosures.

Here I’ll answer some lingering questions around PoR and dig into some of these recent attestations, so users can better understand what these recent procedures actually mean. I also summarize PoR efforts to date and introduce a simple framework through which to judge their effectiveness.

Who has done a PoR recently?

Click for full size

I’ve compiled data on all the PoRs I could find in 2022 above. I have mainly included procedures which give clients the ability to verify their inclusion in the liability set (with the exception of Luno, which just published an attestation and no corresponding cryptographic proof). For the purposes of this exercise, I’m tallying up custodial, centralized exchanges, with liabilities outstanding to their end users. Other things commonly referred to as ‘proof of reserve’ are not included. (A brief sidenote for my Chainlink readers: when I talk about Proof of Reserve I’m mostly referring to the process whereby exchanges or custodial institutions compare user liabilities to assets held in reserve. I know there’s a spectrum of similar attestations that cover all sorts of scenarios, like comparing WBTC to BTC held in reserve. I’m not deliberately snubbing Chainlink, just that their PoR procedures generally do not pertain to exchanges proving reserves to clients.) For a broader look at historical PoR beyond the table see my website. A few notes on the table:

• Impressively, I found that these exchanges in the aggregate underwent Proofs of Reserve covering $33b worth of assets (or 4% of total crypto market cap). Of course, these attestations are of varying quality, but it’s still a good and growing level of coverage. This is not to say that the other tens or hundreds of billions of custodial cryptoassets are not safe — many are with credible institutions like Coinbase, Fidelity Digital Assets, Gemini, etc.
• Only BitMEX and Deribit are undertaking these procedures with a high frequency. This is doable because they don’t rely on auditors to oversee their process. Kraken has a slower cadence due to their usage of an auditor for oversight. For auditor-overseen PoRs, I don’t expect much faster than a monthly policy.
• Only BitMEX and Deribit allow third parties to verify liabilities for themselves. Most of the exchanges covered allow their clients to individual verify whether their liabilities are included in the set (thus creating a kind of ‘herd immunity’ assuming that some clients actually did the verification and found it proper), but mostly these exchanges didn’t publish the full liability set. However, as a third party, I much prefer the model where anyone, whether a client or not, can undertake the verification for themselves.
• Armanino and Mazars oversaw all of the PoR attestations where there was an auditor present, and they’re both out of the PoR market now. This is problematic for the sector. I’m hoping a few audit firms dip their toes back into the water. In the short term I expect these exchanges will mostly be unable to persuade audit firms to oversee their PoRs, as the public backlash against these audit firms has been pretty fierce.
• There’s still a lot of room for improvement. My six point test (elaborated below) lays out very simple ways to improve. For a few exchanges, simply committing to running a PoR on an ongoing basis would help their score. For others, covering a larger share of assets or introducing an auditor would help. Generally, it is not too difficult for exchanges to improve their scores, should they want to.

I’ll get into my assessments of some of these PoRs a bit later. First I want to address the lingering question of nomenclature, which has dominated the PoR narrative recently, and made it hard to discuss the substance of the issue.

Does ‘Proof of Reserve’ not count liabilities?

It’s frequently said that PoR refers to only half of the equation. This mostly stems from a terminology issue. When I, and most others that have been tracking PoR for a while, refer to PoR, we are talking about the procedure whereby both the assets and the liabilities are attested to. Proof of Reserve dates back to at least 2014 (arguably, Mt Gox’ infamous 424,242 BTC transaction in 2011 was the first attempt at a PoR, but it included only the asset side), and even back then, PoR was used to refer to both the assets and liabilities.

The earliest reference to the procedure I can find is a 2013 chat in which Greg Maxwell discusses the possibility of proving ‘reserves’ with an on-chain tx for assets, and a merkle proof of liabilities. So PoR has always referred to both sides of the equation. I moved away from calling it a ‘Proof of Solvency’ because solvency is a broader question which requires a fuller analysis of all of the claims on the exchange. This strays into more typical audit territory and can’t be cryptographically resolved.

Some people have suggested calling it a Proof of Reserve & Liabilities. I don’t love this, because ‘Reserve’ already includes Liabilities. You could also call it a Proof of Assets and Liabilities (PoAL) but that’s unnecessarily pedantic. Reserve is more concise — and it’s the existing term of art.

At the end of the day, the thing initially dubbed ‘Proof of Reserve’ as described by Maxwell is all about the liabilities. Saying “Proof of Reserve doesn’t count the liabilities” doesn’t make sense, because the original and enduring usage of the term refers to a process in which both assets and liabilities are attested to.

A reserve is something that’s being held … in reserve. Assets held on behalf of someone else. So the word ‘reserve’ implies a custodial relationship. The ‘Proof’ of ‘Reserve’ describes the orderly functioning of that relationship — in which assets equal liabilities. If it was called “Proof of Assets” I’d understand the claim that it’s incomplete. But it’s not. So I’m going to stick with ‘Proof of Reserve’ for now.

What do you make of the recent Proofs of Reserve?

I am very encouraged exchanges are taking up the practice. Not all of the recent PoRs are the same though. On my PoR tracking site, I have split up recent exchange attestations into ‘gold standard’, ‘good quality’, and ‘other’ to reflect these distinctions. To get the gold standard label, the exchange should do the following:

• Satisfy the basic qualities of a PoR: cryptographic attestation to assets held, and a disclosure of liabilities
• Optionally, but optimally, incorporate a third-party auditor in the process, to ensure that the attested-to liabilities match the internal database
• If no auditor is involved, demonstrate a high level of credibility by undertaking a PoR for substantially all of the assets on the platform, and allowing third party verifiers to check the completeness of the liability set, including the non-negativity of liabilities
• Commit to an ongoing procedure. There is a wide spectrum of frequency, and this is due to the different types of PoRs (supervised and unsupervised). I would like to see quarterly for PoRs supervised by auditors. For unsupervised ones, a more frequent cadence is possible

Kraken, BitMEX, Deribit, and Kucoin meet the above criteria, and get the ‘gold standard’ moniker on my tracker.

I gave the ‘good quality’ moniker to exchanges scoring at least a 4/6 on my rubric, and ‘other’ to exchanges scoring less than 4. The test assigns one point for each feature present, and a half point for partial credit:

Asset side:

• Did the entity perform cryptographic verification of assets held? (1 pt)
• Does the PoR cover the vast majority of assets on the platform? (1 pt)
• Is the procedure undertaken on a recurring basis with reasonable frequency? (1 pt)

Liability side:

• Can users verify their inclusion in the liability set? (1 pt)
• If the exchange has complexities around margin and lending, are these fully accounted for in the PoL? (1 pt)
• Did a credible auditor provide oversight over the liability attestation? (1 pt)

You can see my findings below. Note that some of these are subjective, and I am relying on my own judgment for these assessments. For exchanges that want to provide feedback, please email nic@niccarter.info.

Click for full size

Additional ‘extra credit’ line items I’m not insisting on, but I’d like to see, are:

• Is the entity able to prove exclusive ownership of assets held?
• Is the entity able to demonstrate that there is no window dressing to shore up cash positions prior to PoR attestations?
• Does the entity clearly stipulate the segregation of client and operating capital?
• Does the entity clearly ensure the seniority of client deposits in a bankruptcy or liquidation scenario?
• (Longer term) Is the entity part of a consortium of provers collectively attesting to the non-duplication of client deposits?

I’ll dive into a couple PoRs here, starting with Kraken. Kraken employed Armanino LLP in their attestation, which gives clients a good level of confidence that they aren’t hiding liabilities, publishing negative or undercounted balances in the merkle leaves, or engaging in window-dressing (aka borrowing funds on a short term basis to pass the attestation).

Kraken also did PoRs for BTC, ETH, USDT, USDC, XRP, ADA, and DOT, representing the majority of platform funds. They even covered staked funds for ETH, ADA, and DOT. Right now they are doing PoRs every six months, although I hope that becomes more frequent with time. In this post, they are realistic about PoR shortcomings, and do not represent it as a panacea for exchange issues.

BitMEX’s approach also deserves praise. They are not relying on an auditor, choosing instead a highly transparent model. On the asset side, they list all BTC balances held by the exchange and the execution scripts for these UTXOs which prove that they are spendable by the BitMEX multisig. On the liabilities side, they publish the Merkle tree of user balances in full. This is different from the standard Maxwell approach whereby users are only exposed to their leaf in the merkle tree (and path to the stem) in the interests of preserving privacy. This means that there are no issues with excluded or negative balances since anyone can vet the liability set in full. To deal with the privacy leakage, they randomly split user balances into two, so specific balances can’t be tracked over time. And impressively, BitMEX now publishes PoR attestations twice a week, a more frequent cadence than most other exchanges.

Deribit also appears to be following a model similar to BitMEX, which is why they score highly on my rubric. They release the full set of liabilities, meaning anyone — client or not — can evaluate the PoR for themselves. They actually split up the accounts into many pieces (rather than four as BitMEX does) to make forensic analysis difficult. And they release liability snapshots on a daily basis! Something I think is unique to Deribit as well is their publication of cumulative margin locked so third parties can evaluate overall client leverage relative to exchange assets.

And uniquely among exchanges I’ve evaluated is that Deribit’s PoR covers 100% of client assets (granted, BitMEX’s PoR coverage isn’t far away). This is because Deribit is a derivatives exchange with relatively few collateral types.

As you can tell, BitMEX/Deribit and Kraken trade off along the cryptographic versus institutional trust continuum. Kraken does not publish the full liability set to the general public but we can still trust that they extracted it faithfully given that they used a credible auditor with experience overseeing the procedure. Additionally, Kraken users can verify their inclusion in the liability set, so that gives us another layer of assurance, provided they do so (and have the wherewithal to spot an issue if there is one). BitMEX on the other hand did not use an auditor but publishes frequent, complete attestations which allow anyone to validate the PoR.

Binance’s PoR on the other hand scores poorly, as it is incomplete. CZ has extolled the virtues of PoR ever since FTX collapsed, but hasn’t yet risen to his own challenge. As an aside, it’s worth noting that CZ’s own history with PoR is mired in controversy. Whether the OkCoin PoR that CZ oversaw in 2014 was genuine or not remains an outstanding question. Whatever the truth of the matter, CZ has known about PoR for eight years and is quite familiar with its possible shortcomings. So he should naturally be unsurprised if some are skeptical of Binance’s PoR if it fails to include controls against similar understatements.

And indeed, Binance’s first PoR doesn’t grant strong assurances. It only covers Bitcoin, which only represents 16.5% of their client assets. It does allow individual users to verify their inclusion in the liability set but does not contain the entire liability list, making it hard for a third party to verify the procedure. Audit firm Mazars supervised the attestation with an ‘Agreed Upon Procedures’ and then deleted their AUP. Shortly thereafter they curtailed their entire PoR practice. Given the relatively small PoR relative to Binance’s depository base and balance sheet, questions arose regarding whether Binance used assets exclusively attributable to clients or marshalled other assets they controlled. Technically, security firm Mysten Labs found a number of issues in the Binance PoR, including four possible ways Binance could have understated liabilities.

Now Binance does mention that they intend to involve third party auditors in subsequent attestations, expand the set of assets covered, and explore ZK proofs to ensure that negative balances attributed to margin are fully matched by user collateral. Given that this appears to be a somewhat rough first draft, I’m inclined to be lenient for now, but I would want to see significant improvements before I am satisfied. I do think that some of the criticism around Binance has been overstated lately, as people appear to be hoping that Binance goes the way of FTX. But the situations are extremely dissimilar. Regardless, Binance could quell many rumors with a more complete PoR.

Of course, evaluating exchange credibility is broader than this simple rubric. For instance, we might aspire to privacy-preserving liability attestations that don’t leak anything material — not the distribution of user balances, or the number of users, or anything like that. Then there’s complexities around margin accounts, which some exchanges deal with by inserting negative user balances in the merkle tree, offset by positive collateral balances. The more the exchange permits margin and lending, the less straightforward the PoR becomes, and the more auditor oversight is warranted.

Proving the exclusivity of assets held is another tricky question. Exchanges can of course borrow funds and engage in window dressing. This isn’t a PoR problem exclusively, it’s a general well-known accounting issue. Incorporating auditors into the process could help provide assurances that exchanges are just borrowing funds on a short term basis. And lastly there’s legal and contractual questions which can obviate the need for a PoR or at the very least give clients very strong assurances that their assets are safe and that they are entitled to them in a variety of scenarios. Here we are getting into more legal and contractual territory, where cryptography is less addressable.

I’ll cover some of these additional considerations in the final section.

Should exchanges release only the merkle tree or a full list of liabilities?

There’s a degree of controversy over this. Exchanges aren’t keen on releasing a full dump of liabilities like BitMEX or Deribit do. If they have 100m+ users, like Binance does, such a file would also be insanely large and unwieldy. And even if you split up balances randomly, you’re still releasing a lot of data — and there’s always ways to extract some signal from such a large dataset.

So there’s a tradeoff between the desire for privacy on the part of the exchange and the auditability of the PoR. If the full liability set isn’t released, third parties can’t meaningfully evaluate the quality of the PoR: they simply have to trust that sufficient exchange clients are diligently performing the verification. And that doesn’t feel great.

Here we run into contradictions and paradoxes. On the one hand, I would say auditor oversight would likely be sufficient if an exchange doesn’t want to publish the full liability set — but in that case, why even bother with user verification and the merkle tree? Why not just do a simple AUP and ask everyone to trust the auditor?

Ultimately, the current state of PoR doesn’t lend itself to good answers in this case. I’m sympathetic to exchanges who don’t want to list a full liability set, even if they have obfuscated the distribution of user balances. But in my view PoR should be public facing, not just something that clients can verify for themselves. So I generally support the publication of as much liability information as possible.

I think the concern regarding privacy leaks from the merkle approach is not unfounded. While auditor oversight can naturally help here, I prefer crypto-native solutions if possible. So I am hoping that ZK approaches, such as the ones Vitalik mentions, can fill in the gap here. ZK liability proofs could potentially grant third parties — not just exchange clients — strong assurances, while maintaining privacy regarding the distribution of client balances. Such schemes date back to at least 2015 with Provisions so the idea is certainly not new. However, ZK tech has come a long way and is now generally trusted throughout the industry in the case of rollups which are deployed in production. The time is ripe for ZK liability schemes, in my view.

What’s next for Proof of Reserve?

Now that PoR appears to be catching on, there are many possible refinements. The core procedure hasn’t changed much since it was proposed by Maxwell in 2013 — improvements have been largely incremental. It was 2015 when the Provisions paper was first published proposing ZK proofs for a more privacy-preserving liabilities side, and yet we still find ourselves with no deployed implementation of the idea. Newer schemes like Ji and Chalkias’ Generalized Proof of Liabilities rely on Zk proofs (specifically, bulletproofs) and look promising, though.

Historically, attempts at PoR have suffered from a variety of technical issues, as a gulf between the academic literature and actual practice of PoR has persisted. Chalkias, Chatzigiannis, and Ji identify a number of vulnerabilities in historical liability assessments, including privacy constraints like leaking the number of users and leaking individual liabilities. There’s clearly scope for more academics to engage in the PoR space. I’m hoping the gulf between academia and industry in this sector is bridged fairly soon.

What I’d like to see in the future regarding PoR is a combination of a few things:

• ZK proofs of liabilities. These don’t leak client data but still provide credible attestations. The merkle approach, even if privacy is sought by splitting up accounts into random pieces, still leaks all sorts of data regarding client behavior. I think exchanges should be comfortable sharing aggregate deposits, but they may not wish to share the distribution of ownership on a weekly or daily basis. Eliminating these privacy concerns makes exchanges more likely to pursue PoR and on a more frequent basis.
• Legal and contractual assurances on top of PoR. PoR is not a substitute for clear terms that establish the seniority of depositors in a liquidation situation and the segregation of client and operating capital.
• Audit firms reentering the space. Right now, the major CPA firms that did AUPs for PoR have deprecated their practices. I would like to see some audit firms step up and start supervising PoR attestations again, as the non-supervised PoRs just don’t provide the same assurances. This is especially the case for more complex liabilities relating to margin and lending.
• Standardization of PoR. One issue we haven’t addressed is the possibility to engage in window dressing by borrowing prior to a PoR and returning the funds after. More frequent attestations mostly fixes this (hard to engage in window dressing if you are doing daily PoR attestations), but another way to address it would be getting a number of exchanges on a shared PoR standard. If they were in some kind of PoR consortium, they could attest to the respective uniqueness of capital and it would be relatively easy to verify that.
• Dedicated custodians building out their own PoR practice. Exchangesare unbundling and some are outsourcing custody. This is because we have good, high quality custodians, that now support all the requisite assets. So we can expect that crypto might end up more like tradfi, with order matching, clearing/settlement, and custody being distinct functions. In this world, a handful of custodians might end up being very important. Clients of the exchanges relying on these custodians deserve to know that their funds are accounted for. For this reason I want to see these custodians start to build their own PoR practice, so that they can cater to these requests as they emerge. This was the case with the Bitcoin held by Coinbase on behalf of GBTC — but Coinbase hadn’t built the proving infrastructure yet, so we were left with an unsatisfying answer.
• A larger set of ‘PoR watchers’. Right now most PoRs are being treated as equally valid or equally stupid, depending on your perspective (there’s a few BTC maxi cynics that hate PoR because they think it normalizes third party custody at the expense of the pure and holy self-custody). I would like to see more critical eyes affixed on PoRs so that exchanges were encouraged to provide better and more complete attestations. I would be much happier if there were dozens of people like me that took the time to evaluate these PoRs.
• DEXes that obsolete CEXes. Of course, functional DEXes are the equivalent of a continuous PoR, because clients generally retain their own assets until it is time to conduct a swap. Something a few folks have noted is that exchanges like StarkEx are kind of a middle ground between a pure on-chain DEX and a centralized exchange that does a PoR. You can think of a proof of reserve as an attempt to bring off chain functions on chain. DEXes are the end state there. If we can get performant and trustworthy DEXes, then we won’t have to worry about CEXes (and PoRs) as much.

To sum up my own feelings on PoR, I am cautiously excited and feeling somewhat validated. I’ve been pushing PoR/ Proof of Solvency for years now, and for a long time it felt like we were making no progress as an industry. Sadly, the catalytic event that caused this latest round of PoRs (with the exception of BitMEX and Kraken who were working on PoR beforehand) was the collapse of FTX. Unfortunately, reform only follows a crisis.

I do reflect with irony on my prior Coindesk column from 2020 — How to Stop the Next Quadriga: Make Exchanges Prove their Reserves. We did not sufficiently ask exchanges to prove reserves, and therefore suffered something much, much worse than Quadriga. Indeed, it appears that the FTX shortfall, if it ends up being $8b, would be 37 times worse in fiat terms that Quadriga’s loss. Incredibly, it seems that no one ever scrutinized the FTX hot or cold wallets carefully. A mere ‘proof of assets’ would probably have exposed them, let alone a full Proof of Reserve.

Each of Quadriga, Gox, and FTX would have been avoidable had PoR been entrenched in the industry, as none of those exchanges would have ever been able to ‘pass’ a PoR. Each was insolvent for some time before the collapse — in the case of Gox, for years. The way PoR works is, if enough exchanges do it, the few exchanges that don’t do it end up sticking out like a sore thumb.

This is why it’s so important that even the most credible exchanges actually commit to the procedure; otherwise, the signal is lost and the sketchy exchanges can hide in the crowd. So I would encourage every exchange that has undertaken the PoR process to continue to publish these attestations and grow their frequency and asset coverage. For the exchanges that don’t, take a careful look at the process. It will get more convenient, safer, and more secure as better tools emerge. And regulators and lawmakers may end up asking for exchanges to do PoRs anyway, so it would be best to get ahead of that and start doing them voluntarily now.