The post Ethereum R&D Roundup: Valentine’s Day Edition appeared first on Ethereum Blog.

]]>First, ongoing progress on the collaboration with the Zcash team and the implementation of zk-SNARKs:

- The full series of “zk-SNARK explainer” posts from myself is now finished, see https://medium.com/@VitalikButerin/quadratic-arithmetic-programs-from-zero-to-hero-f6d558cea649, https://medium.com/@VitalikButerin/exploring-elliptic-curve-pairings-c73c1864e627 and https://medium.com/@VitalikButerin/zk-snarks-under-the-hood-b33151a013f6
- An update on implementation from Christian Reitwiessner: https://blog.ethereum.org/2017/01/19/update-integrating-zcash-ethereum/

On the proof of stake front, myself and Vlad and others have continued to solidify the Casper specification and converge on a roadmap. A key focus of our work has been on a notion of “protocol armor”, which can turn many classes of traditional Byzantine-fault-tolerant consensus algorithms into “attributable-fault consensus algorithms”, where if there is a protocol failure then not only do you know that a large number of validators were faulty, but you also know whom to blame. The work has not yet been fully written out, though it will be further formalized and presented soon, and anyone interested is free to follow along at https://gitter.im/ethereum/casper-scaling-and-protocol-economics.

A post on parametrizing Casper was written here: https://medium.com/@VitalikButerin/parametrizing-casper-the-decentralization-finality-time-overhead-tradeoff-3f2011672735

We had two core dev meetings, and have approved the following EIPs for likely inclusion into Metropolis:

- http://github.com/ethereum/EIPs/issues/86 (abstraction)
- http://github.com/ethereum/EIPs/issues/100 (uncle mining incentive fix)
- https://github.com/ethereum/EIPs/issues/196 and https://github.com/ethereum/EIPs/issues/197 (elliptic curve pairings, the key ingredient for on-chain zk-SNARK verification)
- https://github.com/ethereum/EIPs/issues/198 (bigint modular exponentation, for use in RSA and various other big-number-based cryptographic algorithms)
- https://github.com/ethereum/EIPs/pull/206 (REVERT opcode, equivalent to the “throw” keyword in Solidity but without burning extra gas)
- https://github.com/ethereum/EIPs/pull/210 (blockhash refactoring, serving the triple duty of (i) simplifying consensus logic, (ii) allowing the BLOCKHASH opcode to refer much further back, and (iii) enabling much more efficient and secure light client initial-syncing protocols)

Additionally, there were a few changes to the EIP process itself: https://www.reddit.com/r/ethereum/comments/5rp8mr/update_to_eip_ethereum_improvement_proposal_system/

Work on Mist, Swarm, ENS and associated infrastructure is continuing at a rapid pace; Swarm is now at the stage where it can serve the wallet app, though the incentivization logic is not yet in place.

- Our new developer Victor Maia is working on Swarm integration in Mist; you can follow progress here: https://github.com/ethereum/mist/pull/1661
- The work on ENS includes a name resolver, a registrar, a Javascript implementation library and a user-facing dapp. Support from major Ethereum wallets is coming soon.

Work on programming languages is also moving forward:

- Solidity is adding a fully specified way to access the compiler input, settings and output: https://solidity.readthedocs.io/en/latest/using-the-compiler.html#compiler-input-and-output-json-description
- There are plans (not yet completed) to add an intermediate language to Solidity to help with understanding what the compiler does and auditing compiler output
- In the community, the fp-ethereum functional programming initiative got some attention
- Viper saw another round of improvements, including support for unit types (timestamp, timedelta, wei, wei per second, etc), bytearrays and more builtin functions: https://github.com/ethereum/viper/commits/master
- The Javascript-based development environment Remix added much more powerful debugging support

Work on implementations is progressing:

- The cpp-ethereum project is experimenting with the performance of the EVM. Some results suggest that the EVM might be able to get up to a small linear factor close to “raw metal” with the more structured EVM 1.5 model.
- With its newest 1.5.9 release, Geth supports hardware wallets: https://github.com/ethereum/go-ethereum/releases/tag/v1.5.9
- Work on integrating the recent pyethereum changes into pyethapp is continuing: https://github.com/ethereum/pyethereum/commits/state_revamp_for_stable

We wish everyone a happy Valentine’s day!

The post Ethereum R&D Roundup: Valentine’s Day Edition appeared first on Ethereum Blog.

]]>The post An Update on Integrating Zcash on Ethereum (ZoE) appeared first on Ethereum Blog.

]]>Ethereum’s flexible smart contract interface enables a large variety of applications, many of which have probably not yet been conceived. The possibilities grow considerably when adding the capacity for privacy. Imagine, for example, an election or auction conducted on the blockchain via a smart contract such that the results can be verified by any observer of the blockchain, but the individual votes or bids are not revealed. Another possible scenario may involve selective disclosure where users would have the ability to prove they are in a certain city without disclosing their exact location. The key to adding such capabilities to Ethereum is zero-knowledge succinct non-interactive arguments of knowledge (zk-SNARKs) – precisely the cryptographic engine underlying Zcash.

One of the goals of the Zcash company, codenamed Project Alchemy, is to enable a direct decentralized exchange between Ethereum and Zcash. Connecting these two blockchains and technologies, one focusing on programmability and the other on privacy, is a natural way to facilitate the development of applications requiring both.

As part of the Zcash/Ethereum technical collaboration, Ariel Gabizon from Zcash visited Christian Reitwiessner from the Ethereum hub at Berlin a few weeks ago. The highlight of the visit is a proof of concept implementation of a zk-SNARK verifier written in Solidity, based on pre-compiled Ethereum contracts implemented for the Ethereum C++ client. This work complements Baby ZoE , where a zk-SNARK precompiled contract was written for Parity (the Ethereum Rust client). The updates we’ve made involved adding tiny cryptographic primitives (elliptic curve multiplication, addition and pairing) and implementing the rest in Solidity, all of which allows for a greater flexibility and enables using a variety of zk-SNARK constructions without requiring a hard fork. Details will be shared as they are available later. We tested the new code by successfully verifying a real privacy-preserving Zcash transaction on a testnet of the Ethereum blockchain.

The verification took only 42 milliseconds, which shows that such precompiled contracts can be added, and the gas costs for using them can be made to be quite affordable.

The Zcash system can be reused on Ethereum to create shielded custom tokens. Such tokens already allow many applications like voting, (see below) or simple blind auctions where participants make bids without the knowledge of the amounts bid by others.

If you want to try compiling the proof of concept, you can use the following commands. If you need help, see https://gitter.im/ethereum/privacy-tech

`git clone https://github.com/scipr-lab/libsnark.git cd libsnark`

`sudo PREFIX=/usr/local make NO_PROCPS=1 NO_GTEST=1 NO_DOCS=1 \ CURVE=ALT_BN128 \`

`FEATUREFLAGS="-DBINARY_OUTPUT=1 -DMONTGOMERY_OUTPUT=1 \ -DNO_PT_COMPRESSION=1" \`

`lib install`

`cd ..`

`git clone --recursive -b snark https://github.com/ethereum/cpp-ethereum.git`

`cd cpp-ethereum`

`./scripts/install_deps.sh && cmake . -DEVMJIT=0 -DETHASHCL=0 && make eth`

`cd ..`

`git clone --recursive -b snarks https://github.com/ethereum/solidity.git`

`cd solidity`

`./scripts/install_deps.sh && cmake . && make soltest`

`cd ..`

`./cpp-ethereum/eth/eth --test -d /tmp/test`

`# And on a second terminal:`

`./solidity/test/soltest -t "*/snark" -- --ipcpath /tmp/test/geth.ipc --show-messages`

We also discussed various aspects of integrating zk-SNARKs into the Ethereum blockchain, upon which we now expand.

Recall that a SNARK is a short proof of some property, and what is needed for adding the privacy features to the Ethereum blockchain are clients that have the ability to verify such a proof.

In all recent constructions, the verification procedure consisted solely of operations on elliptic curves. Specifically, the verifier requires scalar multiplication and addition on an elliptic curve group, and would also require a heavier operation called a bilinear pairing.

As mentioned here, implementing these operations directly in the EVM is too costly. Thus, we would want to implement pre-compiled contracts that perform these operations. Now, the question debated is: what level of generality should these pre-compiled contracts aim for.

The security level of the SNARK corresponds to the parameters of the curve. Roughly, the larger the curve order is, and the larger something called the embedding degree is, and the more secure the SNARK based on this curve is. On the other hand, the larger these quantities are, naturally the more costly the operations on the corresponding curve are. Thus, a contract designer using SNARKs may wish to choose these parameters according to their own desired efficiency/security tradeoff. This tradeoff is one reason for implementing a pre-compiled contract with a high level of generality, where the contract designer can choose from a large family of curves. We indeed began by aiming for a high level of generality, where the description of the curve is given as part of the input to the contract. In such a case, a smart contract would be able to perform addition in any elliptic curve group.

A complication with this approach is assigning gas cost to the operation. You must assess, merely from the description of the curve, and with no access to a specific implementation, how expensive a group operation on that curve would be in the worst case. A somewhat less general approach is to allow all curves from a given family. We noticed that when working with the Barreto-Naehrig (BN) family of curves, one can assess roughly how expensive the pairing operation will be, given the curve parameters, as all such curves support a specific kind of optimal Ate pairing. Here’s a sketch of how such a precompile would work and how the gas cost would be computed.

We learned a lot from this debate, but ultimately, decided to “keep it simple” for this proof of concept: we chose to implement contracts for the specific curve currently used by Zcash. We did this by using wrappers of the corresponding functions in the libsnark library, which is also used by Zcash.

Note that we could have simply used a wrapper for the entire SNARK verification function currently used by Zcash, as was done in the above mentioned Baby ZoE project. However, the advantage of explicitly defining elliptic curve operations is enabling using a wide variety of SNARK constructions which, again, all have a verifier working by some combination of the three previously mentioned elliptic curve operations.

As you may have heard, using SNARKs requires a complex setup phase in which the so-called public parameters of the system are constructed. The fact that these public parameters need to be generated in a secure way every time we want to use a SNARK for a particular circuit significantly, hinders the usability of SNARKs. Simplifying this setup phase is an important goal that we have given thought to, but haven’t had any success in so far.

The good news is that someone desiring to issue a token supporting privacy-preserving transactions can simply reuse the public parameters that have already been securely generated by Zcash. It can be reused because the circuit used to verify privacy-preserving transactions is not inherently tied to one currency or blockchain. Rather, one of its explicit inputs is the root of a Merkle tree that contains all the valid notes of the currency. Thus, this input can be changed according to the currency one wishes to work with. Moreover, if it is easy to start a new anonymous token. You can already accomplish many tasks that do not look like tokens at first glance. For example, suppose we wish to conduct an anonymous election to choose a preferred option amongst two. We can issue an anonymous custom token for the vote, and send one coin to each voting party. Since there is no “mining”, it will not be possible to generate tokens any other way. Now each party sends their coin to one of two addresses according to their vote. The address with a larger final balance corresponds to the election result.

A non-token-based system that is fairly simple to build and allows for “selective disclosure” follows. You can, for example, post an encrypted message in regular intervals, containing your physical location to the blockchain (perhaps with other people’s signatures to prevent spoofing). If you use a different key for each message, you can reveal your location only at a certain time by publishing the key. However, with zk-SNARKs you can additionally prove that you were in a certain area without revealing exactly where you were. Inside the zk-SNARK, you decrypt your location and check that it is inside the area. Because of the zero-knowledge property, everyone can verify that check, but nobody will be able to retrieve your actual location.

Achieving the mentioned functionalities – creating anonymous tokens and verifying Zcash transactions on the Ethereum blockchain, will require implementing other elements used by Zcash in Solidity.

For the first functionality, we must have an implementation of tasks performed by nodes on the Zcash network such as updating the note commitment tree.

For the second functionality, we need an implementation of the equihash proof of work algorithm used by Zcash in Solidity. Otherwise, transactions can be verified as valid in themselves, but we do not know whether the transaction was actually integrated into the Zcash blockchain.

Fortunately, such an implementation was written; however, its efficiency needs to be improved in order to be used in practical applications.

**Acknowledgement**: We thank Sean Bowe for technical assistance. We also thank Sean and Vitalik Buterin for helpful comments, and Ming Chan for editing.

The post An Update on Integrating Zcash on Ethereum (ZoE) appeared first on Ethereum Blog.

]]>The post Introduction of the Light Client for DApp developers appeared first on Ethereum Blog.

]]>In most cases a properly designed application can work even without knowing what kind of client it is connected to, but we are looking into adding an API extension for communicating different client capabilities in order to provide a future proof interface. While minor details of LES are still being worked out, I believe it is time to clarify the most important differences between full and light clients from the application developer perspective.

Light clients do not receive pending transactions from the main Ethereum network. **The only pending transactions a light client knows about are the ones that have been created and sent from that client.** When a light client sends a transaction, it starts downloading entire blocks until it finds the sent transaction in one of the blocks, then removes it from the pending transaction set.

**Currently you can only find locally created transactions by hash.** These transactions and their inclusion blocks are stored in the database and can be found by hash later. Finding other transactions is a bit trickier. It is possible (though not implemented as of yet) to download them from a server and verify the transaction is actually included in the block if the server found it. Unfortunately, if the server says that the transaction does not exist, it is not possible for the client to verify the validity of this answer. It is possible to ask multiple servers in case the first one did not know about it, but the client can never be absolutely sure about the non-existence of a given transaction. For most applications this might not be an issue but it is something one should keep in mind if something important may depend on the existence of a transaction. A coordinated attack to fool a light client into believing that no transaction exists with a given hash would probably be difficult to execute but not entirely impossible.

The only thing a light client always has in its database is the last few thousand block headers. This means that retrieving anything else requires the client to send requests and get answers from light servers. The light client tries to optimize request distribution and collects statistical data of each server’s usual response times in order to reduce latency. **Latency is the key performance parameter of a light client. It is usually in the 100-200ms order of magnitude, and it applies to every state/contract storage read, block and receipt set retrieval.** If many requests are made sequentially to perform an operation, it may result in a slow response time for the user. Running API functions in parallel whenever possible can greatly improve performance.

Full clients employ a so-called “MIP mapped” bloom filter to find events quickly in a long list of blocks so that it is reasonably cheap to search for certain events in the entire block history. Unfortunately, using a MIP-mapped filter is not easy to do with a light client, as searches are only performed in individual headers, which is a lot slower. Searching a few days’ worth of block history usually returns after an acceptable amount of time, but **at the moment you should not search for anything in the entire history because it will take an extremely long time.**

Here is the good news: a light client does not need a big database since it can retrieve anything on demand. With garbage collection enabled (which scheduled to be implemented), the database will function more like a cache, and a light client will be able to run with **as low as 10Mb of storage space**. Note that the current Geth implementation uses around **200Mb of memory**, which can probably be further reduced. Bandwidth requirements are also lower when the client is not used heavily. Bandwidth used is usually well under **1Mb/hour when running idle, with an additional 2-3kb for an average state/storage request**.

Sometimes it is unnecessary to pass data back and forth multiple times between the client and the server in order to evaluate a function. It would be possible to execute functions on the server side, then collect all the Merkle proofs proving every piece of state data the function accessed and return all the proofs at once so that the client can re-run the code and verify the proofs. This method can be used for both read-only functions of the contracts as well as any application-specific code that operates on the blockchain/state as an input.

One of the main limitations we are working to improve is the slow search speed of log histories. Many of the limitations mentioned above, including the difficulty of obtaining MIP-mapped bloom filters, follow the same pattern: the server (which is a full node) can easily calculate a certain piece of information, which can be shared with the light clients. But the light clients currently have no practical way of checking the validity of that information, since verifying the entire calculation of the results directly would require so much processing power and bandwidth, which would make using a light client pointless.

Fortunately there is a safe and trustless solution to the general task of indirectly validating remote calculations based on an input dataset that both parties assume to be available, even if the receiving party does not have the actual data, only its hash. This is the exact the case in our scenario where the Ethereum blockchain itself can be used as an input for such a verified calculation. This means it is possible for light clients to have capabilities close to that of full nodes because they can ask a light server to remotely evaluate an operation for them that they would not be able to otherwise perform themselves. The details of this feature are still being worked out and are outside the scope of this document, but the general idea of the verification method is explained by Dr. Christian Reitwiessner in this Devcon 2 talk.

Complex applications accessing huge amounts of contract storage can also benefit from this approach by evaluating accessor functions entirely on the server side and not having to download proofs and re-evaluate the functions. Theoretically it would also be possible to use indirect verification for filtering events that light clients could not watch for otherwise. However, in most cases generating proper logs is still simpler and more efficient.

The post Introduction of the Light Client for DApp developers appeared first on Ethereum Blog.

]]>The post December Roundup appeared first on Ethereum Blog.

]]>First, privacy technologies on Ethereum, and particularly zk-SNARKs (or “zero knowledge proofs”), have been rapidly moving forward.

- A blog post, “zk-SNARKs in a Nutshell“, by Christian Reitwiessner
- A blog post explaining quadratic arithmetic programs, from myself
- An implementation of elliptic curve pairings, perhaps the most complex on-chain technical ingredient in zk-SNARK verification, from myself (with special thanks to Ariel and Sean from the Zcash team for their assistance)
- Experimental work in integrating a zk-SNARK precompile in C++ from Christian (also with special thanks to the Zcash team)

Vlad Zamfir has taken it upon himself to explain the history behind Casper, from his point of view:

On proof of stake from myself:

- A Proof of Stake Design Philosophy
- And while we’re at it, the Proof of Stake FAQ and Sharding FAQ continue to exist and and continue to be worked on.

Vlad has also taken it upon himself to rail against the evils of “economic abstraction” (ie. the goal of trying to create token-agnostic public economic consensus protocols):

Various discussions were had on monetary policy:

- A community-created EIP (186) proposed to decrease ETH issuance by ~3x before PoS
- Discussions on issuance in Casper in the Reddit thread for one of Vlad’s posts

Speaking of EIPs…

- Greg Colvin’s suggested modifications for adding further static analysis capability (184) as part of the move toward “EVM 1.5”
- The Ethereum Name System (launched on the Ropsten testnet in late November), saw an EIP opened (181) to support reverse resolution of Ethereum addresses

The data storage-focused “sister protocol” Swarm continues to move forward:

- Public pilot released! (by Viktor Tron)
- A reddit thread with community feedback

And from the core client development standpoint:

- Geth 1.5.5 was released, combining small but important fixes to various “bugs and annoyances”
- Jan Xie is continuing work on pyethereum to see how well it can pass through all of the denial-of-service blocks in September and October. Not always, though the good news is that there seem to be no quadratic memory issues that stop the client outright.
- Another grab bag of small but important security fixes and stability improvements from Mist 0.8.8, after an audit from Cure53

We wish the community a happy new year and look forward to more progress in January!

The post December Roundup appeared first on Ethereum Blog.

]]>The post Security alert [12/19/2016]: Ethereum.org Forums Database Compromised appeared first on Ethereum Blog.

]]>- The information that was recently accessed is a database backup from April 2016 and contained information about 16.5k forum users.
- The leaked information includes
- Messages, both public and private
- IP-addresses
- Username and email addresses
- Profile information
- Hashed passwords
- ~13k bcrypt hashes (salted)
- ~1.5k WordPress-hashes (salted)
- ~2k accounts without passwords (used federated login)

- The attacker self-disclosed that they are the same person/persons who recently hacked Bo Shen.
- The attacker used social engineering to gain access to a mobile phone number that allowed them to gain access to other accounts, one of which had access to an old database backup from the forum.

We are taking the following steps:

- Forum users whose information may have been compromised by the leak will be receiving an email with additional information.
- We have closed the unauthorized access points involved in the leak.
- We are enforcing stricter security guidelines internally such as removing the recovery phone numbers from accounts and using encryption for sensitive data.
- We are providing the email addresses that we believe were leaked to https://haveibeenpwned.com, a service that helps communicate with affected users.
- We are resetting all forum passwords, effective immediately.

If you were affected by the attack we recommend you do the following:

- Ensure that your passwords are not reused between services. If you have reused your forum.ethereum.org password elsewhere, change it in those places.

Additionally, we recommend this excellent blog post by Kraken that provides useful information about how to protect against these types of attacks.

We deeply regret that this incident occurred and are working diligently internally, as well as with external partners to address the incident.

Questions can be directed to security@ethereum.org.

The post Security alert [12/19/2016]: Ethereum.org Forums Database Compromised appeared first on Ethereum Blog.

]]>The post Swarm alpha public pilot and the basics of Swarm appeared first on Ethereum Blog.

]]>The current release ships with the `swarm`

command that launches a standalone Swarm daemon as separate process using your favourite IPC-compliant ethereum client if needed. Bandwidth accounting (using the Swarm Accounting Protocol = SWAP) is responsible for smooth operation and speedy content delivery by incentivising nodes to contribute their bandwidth and relay data. The SWAP system is functional but it is switched off by default. Storage incentives (punitive insurance) to protect availability of rarely-accessed content is planned to be operational in POC 0.4. So currently by default, the client uses the blockchain only for domain name resolution.

With this blog post we are happy to announce the launch of our shiny new Swarm testnet connected to the Ropsten ethereum testchain. The Ethereum Foundation is contributing a 35-strong (will be up to 105) Swarm cluster running on the Azure cloud. It is hosting the Swarm homepage.

We consider this testnet as the first public pilot, and the community is welcome to join the network, contribute resources, and help us find issues, identify painpoints and give feedback on useability. Instructions can be found in the Swarm guide. We encourage those who can afford to run persistent nodes (nodes that stay online) to get in touch. We have already received promises for 100TB deployments.

Note that the testnet offers no guarantees! Data may be lost or become unavailable. Indeed guarantees of persistence cannot be made at least until the storage insurance incentive layer is implemented (scheduled for POC 0.4).

We envision shaping this project with more and more community involvement, so we are inviting those interested to join our public discussion rooms on gitter. We would like to lay the groundwork for this dialogue with a series of blog posts about the technology and ideology behind Swarm in particular and about Web3 in general. The first post in this series will introduce the ingredients and operation of Swarm as currently functional.

Swarm is a distributed storage platform and content distribution service; a native base layer service of the ethereum Web3 stack. The objective is a peer-to-peer storage and serving solution that has zero downtime, is DDOS-resistant, fault-tolerant and censorship-resistant as well as self-sustaining due to a built-in incentive system. The incentive layer uses peer-to-peer accounting for bandwidth, deposit-based storage incentives and allows trading resources for payment. Swarm is designed to deeply integrate with the devp2p multiprotocol network layer of Ethereum as well as with the Ethereum blockchain for domain name resolution, service payments and content availability insurance. Nodes on the current testnet use the Ropsten testchain for domain name resolution only, with incentivisation switched off. The primary objective of Swarm is to provide decentralised and redundant storage of Ethereum’s public record, in particular storing and distributing dapp code and data as well as blockchain data.

There are two major features that set Swarm apart from other decentralised distributed storage solutions. While existing services (Bittorrent, Zeronet, IPFS) allow you to register and share the content you host on your server, Swarm provides the hosting itself as a decentralised cloud storage service. There is a genuine sense that you can just ‘upload and disappear’: you upload your content to the swarm and retrieve it later, all potentially without a hard disk. Swarm aspires to be the generic storage and delivery service that, when ready, caters to use-cases ranging from serving low-latency real-time interactive web applications to acting as guaranteed persistent storage for rarely used content.

The other major feature is the incentive system. The beauty of decentralised consensus of computation and state is that it allows programmable rulesets for communities, networks, and decentralised services that solve their coordination problems by implementing transparent self-enforcing incentives. Such incentive systems model individual participants as agents following their rational self-interest, yet the network’s emergent behaviour is massively more beneficial to the participants than without coordination.

Not long after Vitalik’s whitepaper the Ethereum dev core realised that a generalised blockchain is a crucial missing piece of the puzzle needed, alongside existing peer-to-peer technologies, to run a fully decentralised internet. The idea of having separate protocols (shh for Whisper, bzz for Swarm, eth for the blockchain) was introduced in May 2014 by Gavin and Vitalik who imagined the Ethereum ecosystem within the grand crypto 2.0 vision of *the third web*. The Swarm project is a prime example of a system where incentivisation will allow participants to efficiently pool their storage and bandwidth resources in order to provide global content services to all participants. We could say that the smart contracts of the incentives implement the *hive mind of the swarm*.

A thorough synthesis of our research into these issues led to the publication of the first two orange papers. Incentives are also explained in the devcon2 talk about the Swarm incentive system. More details to come in future posts.

Swarm is a network, a service and a protocol (rules). A Swarm network is a network of nodes running a wire protocol called bzz using the ethereum devp2p/rlpx network stack as the underlay transport. The Swarm protocol (bzz) defines a mode of interaction. At its core, Swarm implements a *distributed content-addressed chunk store*. Chunks are arbitrary data blobs with a fixed maximum size (currently 4KB). Content addressing means that the address of any chunk is deterministically derived from its content. The addressing scheme falls back on a hash function which takes a chunk as input and returns a 32-byte long key as output. A hash function is irreversible, collision free and uniformly distributed (indeed this is what makes bitcoin, and in general proof-of-work, work).

This hash of a chunk is the address that clients can use to retrieve the chunk (the hash’s *preimage*). Irreversible and collision-free addressing immediately provides integrity protection: no matter the context of how a client knows about an address,

it can tell if the chunk is damaged or has been tampered with just by hashing it.

Swarm’s main offering as a distributed chunkstore is that you can upload content to it.

The nodes constituting the Swarm all dedicate resources (diskspace, memory, bandwidth and CPU) to store and serve chunks. But what determines who is keeping a chunk?

Swarm nodes have an address (the hash of the address of their *bzz-account*) in the same keyspace as the chunks themselves. Lets call this address space the *overlay network*. If we upload a chunk to the Swarm, the protocol determines that it will eventually end up being stored at nodes that are closest to the chunk’s address (according to a well-defined distance measure on the overlay address space). The process by which chunks get to their address is called syncing and is part of the protocol. Nodes that later want to retrieve the content can find it again by forwarding a query to nodes that are close the the content’s address. Indeed, when a node needs a chunk, it simply posts a request to the Swarm with the address of the content, and the Swarm will forward the requests until the data is found (or the request times out). In this regard, Swarm is similar to a traditional *distributed hash table (DHT)* but with two important (and under-researched) features.

Swarm uses a set of TCP/IP connections in which each node has a set of (semi-)permanent peers. All wire protocol messages between nodes are relayed from node to node hopping on active peer connections. Swarm nodes actively manage their peer connections to maintain a particular set of connections, which enables syncing and content-retrieval by key-based routing. Thus, a chunk-to-be-stored or a content-retrieval-request message can always be efficiently routed along these peer connections to the nodes that are nearest to the content’s address. This flavour of the routing scheme is called *forwarding Kademlia*.

Combined with the SWAP incentive system, a node’s rational self-interest dictates opportunistic caching behaviour: The node caches all relayed chunks locally so they can be the ones to serve it next time it is requested. As a consequence of this behavior, popular content ends up being replicated more redundantly across the network, essentially decreasing the latency of retrievals** –** we say that [call this phemon/outcome/?] Swarm is ‘auto-scaling’ as a distribution network. Furthermore, this caching behaviour unburdens the original custodians from potential DDOS attacks. SWAP incentivises nodes to cache all content they encounter, until their storage space has been filled up. In fact, caching incoming chunks of average expected utility is always a good strategy even if you need to expunge older chunks.

The best predictor of demand for a chunk is the rate of requests in the past. Thus it is rational to remove chunks requested the longest time ago. So content that falls out of fashion, goes out of date, or never was popular to begin with, will be garbage collected and removed unless protected by insurance. The upshot is that nodes will end up fully utilizing their dedicated resources to the benefit of users. Such organic auto-scaling makes Swarm a kind of maximum-utilisation elastic cloud.

Now we’ve explained how Swarm functions as a distributed chunk store (fix-sized preimage archive), you may wonder, where do chunks come from and why do I care?

On the API layer Swarm provides a chunker. The chunker takes any kind of readable source, such as a file or a video camera capture device, and chops it into fix-sized chunks. These so-called data chunks or leaf chunks are hashed and then synced with peers. The hashes of the data chunks are then packaged into chunks themselves (called intermediate chunks) and the process is repeated. Currently 128 hashes make up a new chunk. As a result the data is represented by a merkle tree, and it is the root hash of the tree that acts as the address you use to retrieve the uploaded file.

When you retrieve this ‘file’, you look up the root hash and download its preimage. If the preimage is an intermediate chunk, it is interpreted as a series of hashes to address chunks on a lower level. Eventually the process reaches the data level and the content can be served. An important property of a merklised chunk tree is that it provides integrity protection (what you seek is what you get) even on partial reads. For example, this means that you can skip back and forth in a large movie file and still be certain that the data has not been tampered with. advantages of using smaller units (4kb chunk size) include parallelisation of content fetching and less wasted traffic in case of network failures.

On top of the chunk merkle trees, Swarm provides a crucial third layer of organising content: *manifest* files. A manifest is a json array of manifest entries. An entry minimally specifies a path, a content type and a hash pointing to the actual content. Manifests allow you to create a virtual site hosted on Swarm, which provides url-based addressing by always assuming that the host part of the url points to a manifest, and the path is matched against the paths of manifest entries. Manifest entries can point to other manifests, so they can be recursively embedded, which allows manifests to be coded as a compacted trie efficiently scaling to huge datasets (i.e., Wikipedia or YouTube). Manifests can also be thought of as sitemaps or routing tables that map url strings to content. Since each step of the way we either have merkelised structures or content addresses, manifests provide integrity protection for an entire site.

Manifests can be read and directly traversed using the bzzr url scheme. This use is demonstrated by the *Swarm Explorer*, an example Swarm dapp that displays manifest entries as if they were files on a disk organised in directories. Manifests can easily be interpreted as directory trees so a directory and a virtual host can be seen as the same. A simple decentralised dropbox implementation can be based on this feature. The Swarm Explorer is up on swarm: you can use it to browse any virtual site by putting a manifest’s address hash in the url: this link will show the explorer browsing its own source code.

Hash-based addressing is *immutable*, which means there is no way you can overwrite or change the content of a document under a fixed address. However, since chunks are synced to other nodes, Swarm is immutable in the stronger sense that if something is uploaded to Swarm, it cannot be unseen, unpublished, revoked or removed. For this reason alone, be extra careful with what you share. However you can change a site by creating a new manifest that contains new entries or drops old ones. This operation is cheap since it does not require moving any of the actual content referenced. The photo album is another Swarm dapp that demonstrates how this is done. the source on github. If you want your updates to show continuity or need an anchor to display the latest version of your content, you need name based mutable addresses. This is where the blockchain, the Ethereum Name Service and domain names come in. A more complete way to track changes is to use version control, like git or mango, a git using Swarm (or IPFS) as its backend.

In order to authorise changes or publish updates, we need domain names. For a proper domain name service you need the blockchain and some governance. Swarm uses the Ethereum Name Service (ENS) to resolve domain names to Swarm hashes. Tools are provided to interact with the ENS to acquire and manage domains. The ENS is crucial as it is the bridge between the blockchain and Swarm.

If you use the Swarm proxy for browsing, the client assumes that the domain (the part after bzz:/ up to the first slash) resolves to a content hash via ENS. Thanks to the proxy and the standard url scheme handler interface, Mist integration should be blissfully easy for Mist’s official debut with Metropolis.

Our roadmap is ambitious: Swarm 0.3 comes with an extensive rewrite of the network layer and the syncing protocol, obfuscation and double masking for plausible deniability, kademlia routed p2p messaging, improved bandwidth accounting and extended manifests with http header support and metadata. Swarm 0.4 is planned to ship client side redundancy with erasure coding, scan and repair with proof of custody, encryrption support, adaptive transmission channels for multicast streams and the long-awaited storage insurance and litigation.

In future posts, we will discuss obfuscation and plausible deniability, proof of custody and storage insurance, internode messaging and the network testing and simulation framework, and more. Watch this space, bzz…

The post Swarm alpha public pilot and the basics of Swarm appeared first on Ethereum Blog.

]]>The post The History of Casper – Chapter 2 appeared first on Ethereum Blog.

]]>Vitalik and I had each been reasoning about incentives as part of our research before we ever met, so the proposition that “getting the incentives right” was crucial in proof-of-stake was never a matter of debate. **We were never willing to take “half of the coins are honest” as a security assumption.** (It’s in bold because it’s important.) We knew that we needed some kind of “incentive compatibility” between bonded node incentives and protocol security guarantees.

It was always our view that the protocol could be viewed as a game that could easily result in “bad outcomes” if the protocol’s incentives encouraged that behaviour. We regarded this as a potential security problem. Security deposits gave us a clear way to punish bad behaviour; slashing conditions, which are basically programs that decide whether to destroy the deposit.

We had long observed that Bitcoin was more secure when the price of bitcoin was higher, and less secure when it was lower. We also now knew that security deposits provided slasher with more economic efficiency than slasher only on rewards. **It was clear to us that economic security existed and we made it a high priority.**

I’m not sure how much background Vitalik had in game theory (though it was clear he had more than I did). My own game theory knowledge at the start of the story was even more minimal than it is at the end. But I knew how to recognize and calculate Nash Equilibriums. If you haven’t learned about Nash Equilibriums yet, this next paragraph is for you.

A Nash Equilibrium is a strategy profile (the players’ strategy choices) with a corresponding payoff (giving $ETH or taking $ETH away) where no players individually have an incentive to deviate. “Incentive to deviate” means “they get more $ETH if they somehow change what they’re doing”. If you remember that, and every time you hear “Nash Equilbrium” you thought “no points for individual strategy changes”, you’ll have it.

Some time in late summer of 2014, I first ran into “the bribing attacker model” when I made an offhand response to an economic security question Vitalik asked me on a Skype call (“I can just bribe them to do it”). I don’t know where I got the idea. Vitalik then asked me again about this maybe a week or two later, putting me on the spot to develop it further.

**By bribing game participants you can modify a game’s payoffs, and through this operation change its Nash Equilibriums.** Here’s how this might look:

**The bribing attacker was our first useful model of economic security.**

Before the bribing attack, we usually thought about economic attacks as hostile takeovers by foreign, extra-protocol purchasers of tokens or mining power. A pile of external capital would have to come into the system to attack the blockchain. With the bribe attack, the question became “what is the price of bribing the currently existing nodes to get the desired outcome?”.

**We hoped that the bribing attacks of our yet-to-be-defined proof-of-stake protocol would have to spend a lot of money to compensate for lost deposits.**

Debate about “reasonableness” aside, this was our first step in learning to reason about economic security. It was fun and simple to use a bribing attacker. You just see how much you have to pay the players to do what the attacker wants. And we were already confident that we would be able to make sure that an attacker has to pay security-deposit-sized bribes to revert the chain in an attempted double-spend. We knew we could recognize “double-signing”. So we were pretty sure that this would give proof-of-stake a quantifiable economic security advantage over a proof-of-work protocol facing a bribing attacker.

Vitalik and I applied the bribing attacker to our proof-of-stake research. **We found that PoS protocols without security deposits could be trivially defeated with small bribes. You simply pay coin holders to move their coins to new addresses and give you the key to their now empty addresses.** (I’m not sure who originally thought of this idea.) Our insistence on using the briber model easily ruled out all of the proof-of-stake protocols we knew about. I liked that. (At the time we had not yet heard of Jae Kwon’s Tendermint, of Dominic William’s now-defunct Pebble, or of Nick Williamson’s Credits.)

This bribe attack also posed a challenge to security-deposit based proof-of-stake: The moment after a security deposit was returned to its original owner, the bribing adversary could buy the keys to their bonded stakeholder address at minimal cost.

**This attack is identical to the long range attack.** It is acquiring old keys to take control of the blockchain. It meant that the attacker can create “false histories” at will. But only if they start at a height from which all deposits are expired.

Before working on setting the incentives for our proof-of-stake protocol, therefore, we needed to address the long-range attack problem. **If we didn’t address the long range attack problem, then it would be impossible for clients to reliably learn who really had the security deposits.**

We did know that developer checkpoints could be used to deal with the long-range attack problem. We thought this was clearly way too centralized.

In the weeks following my conversion to proof-of-stake, while I was staying at Stephan Tual’s house outside of London, I discovered that there was a natural rule for client reasoning about security deposits. **Signed commitments are only meaningful if the sender currently has a deposit.** That is to say, after the deposit is withdrawn, the signatures from these nodes are no longer meaningful. Why would I trust you after you withdraw your deposit?

The bribing attack model demanded it. **It would cost the bribing attacker almost nothing to break the commitments after the deposit is withdrawn. **

This meant that a client would hold a list of bonded nodes, and stop blocks at the door if they were not signed by one of these nodes. **Ignoring consensus messages from nodes who don’t currently have security deposits solves circumvents the long-range attack problem. **Instead of authenticating the current state based on the history starting from the genesis block, we authenticate it based on a list of who currently has deposits.

**This is radically different from proof-of-work. **

In PoW, a block is valid if it is chained to the genesis block, and if the block hash meets the difficulty requirement for its chain. In this security deposit-based model, a block is valid if it was created by a stakeholder with a currently existing deposit. This meant that you would need to have current information in order to authenticate the blockchain. This subjectivity has caused a lot of people a lot of concern, but it is necessary for security-deposit based proof-of-stake to be secure against the bribing attacker.

This realization made it very clear to me that the proof-of-work security model and the proof-of-stake security model are fundamentally not compatible. I therefore abandoned any serious use of “hybrid” PoW/PoS solutions. Trying to authenticate a proof-of-stake blockchain from genesis now seemed very obviously wrong.

Beyond changing the authentication model, however, we did need to provide a way to manage these lists of security deposits. **We had to use signatures from bonded nodes to manage changes to the list of bonded nodes, and we had to do it after the bonded nodes come to consensus on these changes.** Otherwise, clients would have different lists of bonded validators, and they would therefore be unable to agree on the state of Ethereum.** **

**Bond time needed to be made long, so that clients have time to learn about the new, incoming set of bonded stakeholders.** As long as clients were online enough, they could keep up to date. I thought we would use twitter to share the bonded node list, or at least a hash, so that new and hibernating clients could get synchronized after their user enters a hash into the UI.

**If you have the wrong validator list you can get man-in-the-middled.** But it’s really not that bad. The argument was (and still is!) that **you only need to be able to trust an external source for this information once**. After that once, you will be able to update your list yourself – at least, if you are able to be online regularly enough to avoid the “long range” of withdrawn deposits.

I know that it might take some getting used to. But we can only rely on fresh security deposits. Vitalik was a bit uncomfortable with this argument at first, trying to hold onto the ability to authenticate from genesis, but eventually was convinced by the necessity of this kind of subjectivity in proof of stake protocols. Vitalik independently came up with his weak subjectivity scoring rule, which seemed to me like a perfectly reasonable alternative to my idea at the time, which was basically “have all the deposits sign every Nth block to update the bonded node list”.

**With the nails in the nothing-at-stake and long-range attack coffins completely hammered in, we were ready to start choosing our slashing conditions.**

The next chapter will document what we learned from our first struggles to define a consensus protocol by specifying slashing conditions. I’ll also tell you about what we learned from talking with fine people from our space about our research. The game theory and economic modelling story presented here will continue developing in Chapter 4.

NOTE: The views expressed here are solely my own personal views and do not represent those of the Ethereum Foundation. I am solely responsible for what I’ve written and am not am not acting as a spokesperson for the Foundation.

The post The History of Casper – Chapter 2 appeared first on Ethereum Blog.

]]>The post The History of Casper — Chapter 1 appeared first on Ethereum Blog.

]]>So here’s the Casper tech story, given as a chronological history of the evolution of the key technology, ideas and language that are involved in “Casper research”. Many of our favorite blockchain personalities are part of the story. This is my attempt to recount everything in an accessible, sequential way so that you can see where we are now (and where we’re going) with our research efforts (so don’t argue until the end of the story!). I’m going to try to release a chapter per day until it’s complete.

Also note that this is my personal point of view, understanding what little I could manage through the process of working on proof-of-stake. Vitalik and Greg Meredith’s accounts will vary, for example, as they each have their own view of Casper research.

I immediately got hooked on the Blockchain technology story when Bitcoin first (really) caught my attention in March of 2013. This was during the “Cyprus crisis” run-up in the price of Bitcoin. I learned about cryptographic hashes, digital signatures and public key cryptography. I also learned about Bitcoin mining, and the incentives that miners have to protect the network. I was interested in computer science and security for the first time in my life. It was great.

Set against a narrative of dystopian libertarian economics, it was underground developers (like Amir Taaki) versus central bankers in an epic global battle to save the world from the fractional reserve banking system. The blockchain revolution was better than fiction.

I consumed content on reddit, listened to Lets Talk Bitcoin and a lot of Peter Todd content. I lost money on BTC-e (once because I took advice from the trollbox). I argued with my friends Ethan Buchman and Zach Ramsay about technology. We learned about MasterCoin and the possibility of building systems of top of Bitcoin, taking advantage of its Proof-of-Work network effect. When I first heard about proof-of-stake (PoS) in the 2013 alt-coin scene (thanks PPCoin!), I thought it sounded like heretical voodoo magic. Replacing miners with coins seemed like an inherently strange thing to try to do. I ended up deciding that the long-range attack problem was fatal, and any solutions were going to involve developer checkpoints of one form or another (an opinion I learned from Peter Todd). Being a Bitcoiner in 2013 was one of the most intellectually stimulating experiences of my life.

In Janurary or Feburary 2014, I read about Ethereum for the first time. I watched Vitalik’s youtube videos, and I met him in person at the Toronto Decentral Bitcoin Meetups. He obviously knew way more of the tech story than I did, so I became hooked in, this time on Ethereum. Ethereum was the promise of decentralization made accessible to me, someone without much background. It was general purpose smart contracts that could do anything, disrupt any centralized system. It could be and do so many things that it wasn’t always clear to me what role ethereum would actually play in the blockchain ecosystem. The blockchain tech story (as I see it) took an exciting turn with Ethereum, and I got to be closer to the action

Having been invited by Russel Verbeeten at one of these meetups, Ethan and I went to the hackathon prior to the 2014 Bitcoin Expo in Toronto. (Vitalik taught me how to use Merkle trees at this event.) I was thinking about properly incentivizing and decentralizing the peer review system for a couple of weeks, having recently had a paper rejected from an academic journal. Ethan and I tried putting this kind of system together at the hackathon. Ethan did most of the hard work using pyethereum, while I very slowly put together the first GUI I ever made. We came in second place at the hackathon (after Amir’s “Dark Market”, which became Open Bazaar). We got to meet the whole Ethereum team at the Expo, and we got ourselves invited to the public Skype channels! Charles Hoskinson offered us jobs: It was then, in April 2014, that we started volunteering for Ethereum. We even got @ethereum.org email addresses.

So I got into the blockchain space because I got hooked on the Bitcoin tech story, and then on the Ethereum tech story. I then got hooked on the proof-of-stake tech story, which I now know to be very compelling. I’m going to share it, being as faithful as possible to the timeline and manner in which the parts of picture have been coming together, in an effort to help bring everyone up to speed on our efforts. It may take a few chapters, but story time ain’t over ’til it’s over.

When Vitalik first expressed interest in PoS to me in May 2014, first over Skype and then at a Bitcoin conference in Vienna, I was skeptical. Then he told me about slasher, which I think he had come up in January 2014. Slasher was the idea that you could lose your block reward if you sign blocks at the same height on two forks.

This gave Vitalik the ability to directly tackle (and arguably solve) the nothing-at-stake problem. (For the uninitiated, the “nothing-at-stake” problem refers to the fact that the PoS miners best strategy is to mine on all forks, because signatures are very cheap to produce). It also opened up our imaginations to a new space of interactive protocols for disincentivizing bad behaviour.

Still, I did not feel very satisfied with proof-of-stake at this time (despite Vitalik telling me a couple of times that he thinks “proof-of-stake is the future”) because I was really in love with proof-of-work. So during the summer I mostly worked on proof-of-work problems (ASIC-hard PoW, security sharing between PoW Chains via “Proofs-of-Proof-of-Work”, neither to completion). But I did suggest the use of security deposits to a couple of contract developers on a couple of different occasions. This planted the seed for insights made on the fateful post-Ethereum-meetup night of September 11th 2014 (kudos to Stephan Tual for organizing + getting me to that event!).

Ethan Buchman and I stayed up late talking about proof-of-stake at the “hacker” instead of the “party” section of Amir Taaki’s squat in London. I connected the dots and internalized the power of security deposits for proof-of-stake. This was the night that I became convinced that PoS would work, and that making it work would be a huge amount of fun. It was also the first time I experienced the surprising size of the PoS design space, through long arguments about attacks and possible protocol responses.

Since the early morning of September 12th, 2014 I have firmly advocated (to everyone who would listen) that blockchains move to PoS because it would be more secure. Amir Taaki was unimpressed by my enthusiasm for proof-of-stake. At least Ethan and I were having the best time.

The use of security deposits always significantly leveraged slasher’s effectiveness. Instead of forgoing some profit X, a provably faulty node would lose a security deposit (imagined to be on the order of size X/r) on which the block reward X was to be paid as interest (at rate r).

You place a deposit to play, and if you play nice you make a small return on your deposit, but if you play mean you lose your deposit. It feels economically ideal, and it’s so programmable.

**Adding deposits to slasher meant that the nothing at stake problem was officially solved.**

At least, I had made up my mind that it was solved to the point where we could no longer understand why anyone would want to build a proof-of-stake system without security deposits, for fear of nothing-at-stake problems.

Also on September 12th, 2014 I met Pink Penguin for the first time, due to an introduction from Stephan Tual. I breathlessly recounted my PoS insights made the night before. And after I respectfully declined a job from from Eris Industries (now Monax) that week, Pink Penguin began sponsoring this research! (Thanks <3!!)

At this point in the story I was unaware of the other, multiple independent discoveries of the use of security deposits in proof-of-stake systems made by Jae Kwon, Dominic Williams, and Nick Williamson.

Stay tuned… the next chapter is about the central role that ideas from game theory played in setting the design goals that led to Casper!

NOTE: The views expressed here are solely my own personal views and do not represent those of the Ethereum Foundation. I am solely responsible for what I’ve written and am not am not acting as a spokesperson for the Foundation.

The post The History of Casper — Chapter 1 appeared first on Ethereum Blog.

]]>The post zkSNARKs in a nutshell appeared first on Ethereum Blog.

]]>As a very short summary, zkSNARKs as currently implemented, have 4 main ingredients (don’t worry, we will explain all the terms in later sections):

**A) Encoding as a polynomial problem**

The program that is to be checked is compiled into a quadratic equation of polynomials: t(x) h(x) = w(x) v(x), where the equality holds if and only if the program is computed correctly. The prover wants to convince the verifier that this equality holds.

**B) Succinctness by random sampling**

The verifier chooses a secret evaluation point s to reduce the problem from multiplying polynomials and verifying polynomial function equality to simple multiplication and equality check on numbers: t(s)h(s) = w(s)v(s)

This reduces both the proof size and the verification time tremendously.

**C) Homomorphic encoding / encryption**

An encoding/encryption function E is used that has some homomorphic properties (but is not fully homomorphic, something that is not yet practical). This allows the prover to compute E(t(s)), E(h(s)), E(w(s)), E(v(s)) without knowing s, she only knows E(s) and some other helpful encrypted values.

**D) Zero Knowledge**

The prover permutes the values E(t(s)), E(h(s)), E(w(s)), E(v(s)) by multiplying with a number so that the verifier can still check their correct *structure* without knowing the actual encoded values.

The very rough idea is that checking t(s)h(s) = w(s)v(s) is identical to checking t(s)h(s) k = w(s)v(s) k for a random secret number k (which is not zero), with the difference that if you are sent only the numbers (t(s)h(s) k) and (w(s)v(s) k), it is impossible to derive t(s)h(s) or w(s)v(s).

This was the hand-waving part so that you can understand the essence of zkSNARKs, and now we get into the details.

Let us start with a quick reminder of how RSA works, leaving out some nit-picky details. Remember that we often work with numbers modulo some other number instead of full integers. The notation here is “a + b ≡ c (mod n)”, which means “(a + b) % n = c % n”. Note that the “(mod n)” part does not apply to the right hand side “c” but actually to the “≡” and all other “≡” in the same equation. This makes it quite hard to read, but I promise to use it sparingly. Now back to RSA:

The prover comes up with the following numbers:

- p, q: two random secret primes
- n := p q
- d: random number such that 1 < d < n – 1
- e: a number such that d e ≡ 1 (mod (p-1)(q-1)).

The public key is (e, n) and the private key is d. The primes p and q can be discarded but should not be revealed.

The message m is encrypted via

- E(m) := m
^{e}% n

and c = E(m) is decrypted via

- D(c) := c
^{d}% n.

Because of the fact that c^{d} ≡ (m^{e} % n)^{d} ≡ m^{ed} (mod n) and multiplication in the exponent of m behaves like multiplication in the group modulo (p-1)(q-1), we get m^{ed} ≡ m (mod n). Furthermore, the security of RSA relies on the assumption that n cannot be factored efficiently and thus d cannot be computed from e (if we knew p and q, this would be easy).

One of the remarkable feature of RSA is that it is **multiplicatively homomorphic**. In general, two operations are homomorphic if you can exchange their order without affecting the result. In the case of homomorphic encryption, this is the property that you can perform computations on encrypted data. *Fully homomorphic encryption*, something that exists, but is not practical yet, would allow to evaluate arbitrary programs on encrypted data. Here, for RSA, we are only talking about group multiplication. More formally: E(x) E(y) ≡ x^{e}y^{e} ≡ (xy)^{e} ≡ E(x y) (mod n), or in words: The product of the encryption of two messages is equal to the encryption of the product of the messages.

This homomorphicity already allows some kind of zero-knowledge proof of multiplication: The prover knows some secret numbers x and y and computes their product, but sends only the encrypted versions a = E(x), b = E(y) and c = E(x y) to the verifier. The verifier now checks that (a b) % n ≡ c % n and the only thing the verifier learns is the encrypted version of the product and that the product was correctly computed, but she neither knows the two factors nor the actual product. If you replace the product by addition, this already goes into the direction of a blockchain where the main operation is to add balances.

Having touched a bit on the zero-knowledge aspect, let us now focus on the other main feature of zkSNARKs, the succinctness. As you will see later, the succinctness is the much more remarkable part of zkSNARKs, because the zero-knowledge part will be given “for free” due to a certain encoding that allows for a limited form of homomorphic encoding.

SNARKs are short for *succinct non-interactive arguments of knowledge*. In this general setting of so-called interactive protocols, there is a *prover* and a *verifier* and the prover wants to convince the verifier about a statement (e.g. that f(x) = y) by exchanging messages. The generally desired properties are that no prover can convince the verifier about a wrong statement (*soundness*) and there is a certain strategy for the prover to convince the verifier about any true statement (*completeness*). The individual parts of the acronym have the following meaning:

- Succinct: the sizes of the messages are tiny in comparison to the length of the actual computation
- Non-interactive: there is no or only little interaction. For zkSNARKs, there is usually a setup phase and after that a single message from the prover to the verifier. Furthermore, SNARKs often have the so-called “public verifier” property meaning that anyone can verify without interacting anew, which is important for blockchains.
- ARguments: the verifier is only protected against computationally limited provers. Provers with enough computational power can create proofs/arguments about wrong statements (Note that with enough computational power, any public-key encryption can be broken). This is also called “computational soundness”, as opposed to “perfect soundness”.
- of Knowledge: it is not possible for the prover to construct a proof/argument without knowing a certain so-called
*witness*(for example the address she wants to spend from, the preimage of a hash function or the path to a certain Merkle-tree node).

If you add the **zero-knowledge** prefix, you also require the property (roughly speaking) that during the interaction, the verifier learns nothing apart from the validity of the statement. The verifier especially does not learn the *witness string* – we will see later what that is exactly.

As an example, let us consider the following transaction validation computation: f(σ_{1}, σ_{2}, s, r, v, p_{s}, p_{r}, v) = 1 if and only if σ_{1} and σ_{2} are the root hashes of account Merkle-trees (the pre- and the post-state), s and r are sender and receiver accounts and p_{s}, p_{r} are Merkle-tree proofs that testify that the balance of s is at least v in σ_{1} and they hash to σ_{2} instead of σ_{1} if v is moved from the balance of s to the balance of r.

It is relatively easy to verify the computation of f if all inputs are known. Because of that, we can turn f into a zkSNARK where only σ_{1} and σ_{2} are publicly known and (s, r, v, p_{s}, p_{r}, v) is the witness string. The zero-knowledge property now causes the verifier to be able to check that the prover knows some witness that turns the root hash from σ_{1} to σ_{2} in a way that does not violate any requirement on correct transactions, but she has no idea who sent how much money to whom.

The formal definition (still leaving out some details) of zero-knowledge is that there is a *simulator* that, having also produced the setup string, but does not know the secret witness, can interact with the verifier — but an outside observer is not able to distinguish this interaction from the interaction with the real prover.

In order to see which problems and computations zkSNARKs can be used for, we have to define some notions from complexity theory. If you do not care about what a “witness” is, what you will *not* know after “reading” a zero-knowledge proof or why it is fine to have zkSNARKs only for a specific problem about polynomials, you can skip this section.

First, let us restrict ourselves to functions that only output 0 or 1 and call such functions *problems*. Because you can query each bit of a longer result individually, this is not a real restriction, but it makes the theory a lot easier. Now we want to measure how “complicated” it is to solve a given problem (compute the function). For a specific machine implementation M of a mathematical function f, we can always count the number of steps it takes to compute f on a specific input x – this is called the *runtime* of M on x. What exactly a “step” is, is not too important in this context. Since the program usually takes longer for larger inputs, this runtime is always measured in the size or length (in number of bits) of the input. This is where the notion of e.g. an “n^{2} algorithm” comes from – it is an algorithm that takes at most n^{2} steps on inputs of size n. The notions “algorithm” and “program” are largely equivalent here.

Programs whose runtime is at most n^{k} for some k are also called “polynomial-time programs”.

Two of the main classes of problems in complexity theory are P and NP:

- P is the class of problems L that have polynomial-time programs.

Even though the exponent k can be quite large for some problems, P is considered the class of “feasible” problems and indeed, for non-artificial problems, k is usually not larger than 4. Verifying a bitcoin transaction is a problem in P, as is evaluating a polynomial (and restricting the value to 0 or 1). Roughly speaking, if you only have to compute some value and not “search” for something, the problem is almost always in P. If you have to search for something, you mostly end up in a class called NP.

There are zkSNARKs for all problems in the class NP and actually, the practical zkSNARKs that exist today can be applied to all problems in NP in a generic fashion. It is unknown whether there are zkSNARKs for any problem outside of NP.

All problems in NP always have a certain structure, stemming from the definition of NP:

- NP is the class of problems L that have a polynomial-time program V that can be used to verify a fact given a polynomially-sized so-called witness for that fact. More formally:

L(x) = 1 if and only if there is some polynomially-sized string w (called the*witness) s*uch that V(x, w) = 1

As an example for a problem in NP, let us consider the problem of boolean formula satisfiability (SAT). For that, we define a boolean formula using an inductive definition:

- any variable x
_{1}, x_{2}, x_{3},… is a boolean formula (we also use any other character to denote a variable - if f is a boolean formula, then ¬f is a boolean formula (negation)
- if f and g are boolean formulas, then (f ∧ g) and (f ∨ g) are boolean formulas (conjunction / and, disjunction / or).

The string “((x_{1}∧ x_{2}) ∧ ¬x_{2})” would be a boolean formula.

A boolean formula is *satisfiable* if there is a way to assign truth values to the variables so that the formula evaluates to true (where ¬true is false, ¬false is true, true ∧ false is false and so on, the regular rules). The satisfiability problem SAT is the set of all satisfiable boolean formulas.

- SAT(f) := 1 if f is a satisfiable boolean formula and 0 otherwise

The example above, “((x_{1}∧ x_{2}) ∧ ¬x_{2})”, is not satisfiable and thus does not lie in SAT. The witness for a given formula is its satisfying assignment and verifying that a variable assignment is satisfying is a task that can be solved in polynomial time.

If you restrict the definition of NP to witness strings of length zero, you capture the same problems as those in P. Because of that, every problem in P also lies in NP. One of the main tasks in complexity theory research is showing that those two classes are actually different – that there is a problem in NP that does not lie in P. It might seem obvious that this is the case, but if you can prove it formally, you can win US$ 1 million. Oh and just as a side note, if you can prove the converse, that P and NP are equal, apart from also winning that amount, there is a big chance that cryptocurrencies will cease to exist from one day to the next. The reason is that it will be much easier to find a solution to a proof of work puzzle, a collision in a hash function or the private key corresponding to an address. Those are all problems in NP and since you just proved that P = NP, there must be a polynomial-time program for them. But this article is not to scare you, most researchers believe that P and NP are not equal.

Let us get back to SAT. The interesting property of this seemingly simple problem is that it does not only lie in NP, it is also NP-complete. The word “complete” here is the same complete as in “Turing-complete”. It means that it is one of the hardest problems in NP, but more importantly — and that is the definition of NP-complete — an input to any problem in NP can be transformed to an equivalent input for SAT in the following sense:

For any NP-problem L there is a so-called *reduction function* f, which is computable in polynomial time such that:

- L(x) = SAT(f(x))

Such a reduction function can be seen as a compiler: It takes source code written in some programming language and transforms in into an equivalent program in another programming language, which typically is a machine language, which has the some semantic behaviour. Since SAT is NP-complete, such a reduction exists for any possible problem in NP, including the problem of checking whether e.g. a bitcoin transaction is valid given an appropriate block hash. There is a reduction function that translates a transaction into a boolean formula, such that the formula is satisfiable if and only if the transaction is valid.

In order to see such a reduction, let us consider the problem of evaluating polynomials. First, let us define a polynomial (similar to a boolean formula) as an expression consisting of integer constants, variables, addition, subtraction, multiplication and (correctly balanced) parentheses. Now the problem we want to consider is

- PolyZero(f) := 1 if f is a polynomial which has a zero where its variables are taken from the set {0, 1}

We will now construct a reduction from SAT to PolyZero and thus show that PolyZero is also NP-complete (checking that it lies in NP is left as an exercise).

It suffices to define the reduction function r on the structural elements of a boolean formula. The idea is that for any boolean formula f, the value r(f) is a polynomial with the same number of variables and f(a_{1},..,a_{k}) is true if and only if r(f)(a_{1},..,a_{k}) is zero, where true corresponds to 1 and false corresponds to 0, and r(f) only assumes the value 0 or 1 on variables from {0, 1}:

- r(x
_{i}) := (1 – x_{i}) - r(¬f) := (1 – r(f))
- r((f ∧ g)) := (1 – (1 – r(f))(1 – r(g)))
- r((f ∨ g)) := r(f)r(g)

One might have assumed that r((f ∧ g)) would be defined as r(f) + r(g), but that will take the value of the polynomial out of the {0, 1} set.

Using r, the formula ((x ∧ y) ∨¬x) is translated to (1 – (1 – (1 – x))(1 – (1 – y))(1 – (1 – x)),

Note that each of the replacement rules for r satisfies the goal stated above and thus r correctly performs the reduction:

- SAT(f) = PolyZero(r(f)) or f is satisfiable if and only if r(f) has a zero in {0, 1}

**Witness Preservation**

From this example, you can see that the reduction function only defines how to translate the input, but when you look at it more closely (or read the proof that it performs a valid reduction), you also see a way to transform a valid witness together with the input. In our example, we only defined how to translate the formula to a polynomial, but with the proof we explained how to transform the witness, the satisfying assignment. This simultaneous transformation of the witness is not required for a transaction, but it is usually also done. This is quite important for zkSNARKs, because the the only task for the prover is to convince the verifier that such a witness exists, without revealing information about the witness.

In the previous section, we saw how computational problems inside NP can be reduced to each other and especially that there are NP-complete problems that are basically only reformulations of all other problems in NP – including transaction validation problems. This makes it easy for us to find a generic zkSNARK for all problems in NP: We just choose a suitable NP-complete problem. So if we want to show how to validate transactions with zkSNARKs, it is sufficient to show how to do it for a certain problem that is NP-complete and perhaps much easier to work with theoretically.

This and the following section is based on the paper GGPR12 (the linked technical report has much more information than the journal paper), where the authors found that the problem called Quadratic Span Programs (QSP) is particularly well suited for zkSNARKs. A Quadratic Span Program consists of a set of polynomials and the task is to find a linear combination of those that is a multiple of another given polynomial. Furthermore, the individual bits of the input string restrict the polynomials you are allowed to use. In detail (the general QSPs are a bit more relaxed, but we already define the *strong* version because that will be used later):

A QSP over a field F for inputs of length n consists of

- a set of polynomials v
_{0},…,v_{m}, w_{0},…,w_{m}over this field F, - a polynomial t over F (the target polynomial),
- an injective function f: {(i, j) | 1 ≤ i ≤ n, j ∈ {0, 1}} → {1, …, m}

The task here is roughly, to multiply the polynomials by factors and add them so that the sum (which is called a *linear combination*) is a multiple of t. For each binary input string u, the function f restricts the polynomials that can be used, or more specific, their factors in the linear combinations. For formally:

An input u is *accepted* (verified) by the QSP if and only if there are tuples a = (a_{1},…,a_{m}), b = (b_{1},…,b_{m}) from the field F such that

- a
_{k},b_{k}= 1 if k = f(i, u[i]) for some i, (u[i] is the ith bit of u) - a
_{k},b_{k}= 0 if k = f(i, 1 – u[i]) for some i and - the target polynomial t divides v
_{a}w_{b}where v_{a}= v_{0}+ a_{1}v_{0}+ … + a_{m}v_{m}, w_{b}= w_{0}+ b_{1}w_{0}+ … + b_{m}w_{m}.

Note that there is still some freedom in choosing the tuples a and b if 2n is smaller than m. This means QSP only makes sense for inputs up to a certain size – this problem is removed by using non-uniform complexity, a topic we will not dive into now, let us just note that it works well for cryptography where inputs are generally small.

As an analogy to satisfiability of boolean formulas, you can see the factors a_{1},…,a_{m}, b_{1},…,b_{m} as the assignments to the variables, or in general, the NP witness. To see that QSP lies in NP, note that all the verifier has to do (once she knows the factors) is checking that the polynomial t divides v_{a} w_{b}, which is a polynomial-time problem.

We will not talk about the reduction from generic computations or circuits to QSP here, as it does not contribute to the understanding of the general concept, so you have to believe me that QSP is NP-complete (or rather complete for some non-uniform analogue like NP/poly). In practice, the reduction is the actual “engineering” part – it has to be done in a clever way such that the resulting QSP will be as small as possible and also has some other nice features.

One thing about QSPs that we can already see is how to verify them much more efficiently: The verification task consists of checking whether one polynomial divides another polynomial. This can be facilitated by the prover in providing another polynomial h such that t h = v_{a} w_{b} which turns the task into checking a polynomial identity or put differently, into checking that t h – v_{a} w_{b} = 0, i.e. checking that a certain polynomial is the zero polynomial. This looks rather easy, but the polynomials we will use later are quite large (the degree is roughly 100 times the number of gates in the original circuit) so that multiplying two polynomials is not an easy task.

So instead of actually computing v_{a}, w_{b} and their product, the verifier chooses a secret random point s (this point is part of the “toxic waste” of zCash), computes the numbers t(s), v_{k}(s) and w_{k}(s) for all k and from them, v_{a}(s) and w_{b}(s) and only checks that t(s) h(s) = v_{a}(s) w_{b} (s). So a bunch of polynomial additions, multiplications with a scalar and a polynomial product is simplified to field multiplications and additions.

Checking a polynomial identity only at a single point instead of at all points of course reduces the security, but the only way the prover can cheat in case t h – v_{a} w_{b} is not the zero polynomial is if she manages to hit a zero of that polynomial, but since she does not know s and the number of zeros is tiny (the degree of the polynomials) when compared to the possibilities for s (the number of field elements), this is very safe in practice.

We now describe the zkSNARK for QSP in detail. It starts with a setup phase that has to be performed for every single QSP. In zCash, the circuit (the transaction verifier) is fixed, and thus the polynomials for the QSP are fixed which allows the setup to be performed only once and re-used for all transactions, which only vary the input u. For the setup, which generates the *common reference string* (CRS), the verifier chooses a random and secret field element s and encrypts the values of the polynomials at that point. The verifier uses some specific encryption E and publishes E(v_{k}(s)) and E(w_{k}(s)) in the CRS. The CRS also contains several other values which makes the verification more efficient and also adds the zero-knowledge property. The encryption E used there has a certain homomorphic property, which allows the prover to compute E(v(s)) without actually knowing v_{k}(s).

Let us first look at a simpler case, namely just the encrypted evaluation of a polynomial at a secret point, and not the full QSP problem.

For this, we fix a group (an elliptic curve is usually chosen here) and a generator g. Remember that a group element is called *generator* if there is a number n (the group order) such that the list g^{0}, g^{1}, g^{2}, …, g^{n-1} contains all elements in the group. The encryption is simply E(x) := g^{x}. Now the verifier chooses a secret field element s and publishes (as part of the CRS)

- E(s
^{0}), E(s^{1}), …, E(s^{d}) – d is the maximum degree of all polynomials

After that, s can be (and has to be) forgotten. This is exactly what zCash calls toxic waste, because if someone can recover this and the other secret values chosen later, they can arbitrarily spoof proofs by finding zeros in the polynomials.

Using these values, the prover can compute E(f(s)) for arbitrary polynomials f without knowing s: Assume our polynomial is f(x) = 4x^{2} + 2x + 4 and we want to compute E(f(s)), then we get E(f(s)) = E(4s^{2} + 2s + 4) = g^{4s^2 + 2s + 4} = E(s^{2})^{4} E(s^{1})^{2} E(s^{0})^{4}, which can be computed from the published CRS without knowing s.

The only problem here is that, because s was destroyed, the verifier cannot check that the prover evaluated the polynomial correctly. For that, we also choose another secret field element, α, and publish the following “shifted” values:

- E(αs
^{0}), E(αs^{1}), …, E(αs^{d})

As with s, the value α is also destroyed after the setup phase and neither known to the prover nor the verifier. Using these encrypted values, the prover can similarly compute E(α f(s)), in our example this is E(4αs^{2} + 2αs + 4α) = E(αs^{2})^{4} E(αs^{1})^{2} E(αs^{0})^{4}. So the prover publishes A := E(f(s)) and B := E(α f(s))) and the verifier has to check that these values match. She does this by using another main ingredient: A so-called *pairing function* e. The elliptic curve and the pairing function have to be chosen together, so that the following property holds for all x, y:

- e(g
^{x}, g^{y}) = e(g, g)^{xy}

Using this pairing function, the verifier checks that e(A, g^{α}) = e(B, g) — note that g^{α} is known to the verifier because it is part of the CRS as E(αs^{0}). In order to see that this check is valid if the prover does not cheat, let us look at the following equalities:

e(A, g^{α}) = e(g^{f(s)}, g^{α}) = e(g, g)^{α f(s)}

e(B, g) = e(g^{α f(s)}, g) = e(g, g)^{α f(s)}

The more important part, though, is the question whether the prover can somehow come up with values A, B that fulfill the check e(A, g^{α}) = e(B, g) but are not E(f(s)) and E(α f(s))), respectively. The answer to this question is “we hope not”. Seriously, this is called the “d-power knowledge of exponent assumption” and it is unknown whether a cheating prover can do such a thing or not. This assumption is an extension of similar assumptions that are made for proving the security of other public-key encryption schemes and which are similarly unknown to be true or not.

Actually, the above protocol does not really allow the verifier to check that the prover evaluated the polynomial f(x) = 4x^{2} + 2x + 4, the verifier can only check that the prover evaluated *some* polynomial at the point s. The zkSNARK for QSP will contain another value that allows the verifier to check that the prover did indeed evaluate the correct polynomial.

What this example does show is that the verifier does not need to evaluate the full polynomial to confirm this, it suffices to evaluate the pairing function. In the next step, we will add the zero-knowledge part so that the verifier cannot reconstruct anything about f(s), not even E(f(s)) – the encrypted value.

For that, the prover picks a random δ and instead of A := E(f(s)) and B := E(α f(s))), she sends over A’ := E(δ + f(s)) and B := E(α (δ + f(s)))). If we assume that the encryption cannot be broken, the zero-knowledge property is quite obvious. We now have to check two things: 1. the prover can actually compute these values and 2. the check by the verifier is still true.

For 1., note that A’ = E(δ + f(s)) = g^{δ + f(s)} = g^{δ}g^{f(s)} = E(δ) E(f(s)) = E(δ) A and similarly, B’ = E(α (δ + f(s)))) = E(α δ + α f(s))) = g^{α δ + α f(s)} = g^{α δ} g^{α f(s)}

= E(α)^{δ}E(α f(s)) = E(α)^{δ} B.

For 2., note that the only thing the verifier checks is that the values A and B she receives satisfy the equation A = E(a) und B = E(α a) for some value a, which is obviously the case for a = δ + f(s) as it is the case for a = f(s).

Ok, so we now know a bit about how the prover can compute the encrypted value of a polynomial at an encrypted secret point without the verifier learning anything about that value. Let us now apply that to the QSP problem.

Remember that in the QSP we are given polynomials v_{0},…,v_{m}, w_{0},…,w_{m,} a target polynomial t (of degree at most d) and a binary input string u. The prover finds a_{1},…,a_{m, }b_{1},…,b_{m} (that are somewhat restricted depending on u) and a polynomial h such that

- t h = (v
_{0}+ a_{1}v_{1}+ … + a_{m}v_{m}) (w_{0}+ b_{1}w_{1}+ … + b_{m}w_{m}).

In the previous section, we already explained how the common reference string (CRS) is set up. We choose secret numbers s and α and publish

- E(s
^{0}), E(s^{1}), …, E(s^{d}) and E(αs^{0}), E(αs^{1}), …, E(αs^{d})

Because we do not have a single polynomial, but sets of polynomials that are fixed for the problem, we also publish the evaluated polynomials right away:

- E(t(s)), E(α t(s)),
- E(v
_{0}(s)), …, E(v_{m}(s)), E(α v_{0}(s)), …, E(α v_{m}(s)), - E(w
_{0}(s)), …, E(w_{m}(s)), E(α w_{0}(s)), …, E(α w_{m}(s)),

and we need further secret numbers β_{v}, β_{w}, γ (they will be used to verify that those polynomials were evaluated and not some arbitrary polynomials) and publish

- E(γ), E(β
_{v}γ), E(β_{w}γ), - E(β
_{v}v_{1}(s)), …, E(β_{v}v_{m}(s)) - E(β
_{w}w_{1}(s)), …, E(β_{w}w_{m}(s)) - E(β
_{v}t(s)), E(β_{w}t(s))

This is the full common reference string. In practical implementations, some elements of the CRS are not needed, but that would complicated the presentation.

Now what does the prover do? She uses the reduction explained above to find the polynomial h and the values a_{1},…,a_{m, }b_{1},…,b_{m}. Here it is important to use a witness-preserving reduction (see above) because only then, the values a_{1},…,a_{m, }b_{1},…,b_{m} can be computed together with the reduction and would be very hard to find otherwise. In order to describe what the prover sends to the verifier as proof, we have to go back to the definition of the QSP.

There was an injective function f: {(i, j) | 1 ≤ i ≤ n, j ∈ {0, 1}} → {1, …, m} which restricts the values of a_{1},…,a_{m, }b_{1},…,b_{m}. Since m is relatively large, there are numbers which do not appear in the output of f for any input. These indices are not restricted, so let us call them I_{free} and define v_{free}(x) = Σ_{k} a_{k}v_{k}(x) where the k ranges over all indices in I_{free}. For w(x) = b_{1}w_{1}(x) + … + b_{m}w_{m}(x), the proof now consists of

- V
_{free}:= E(v_{free}(s)), W := E(w(s)), H := E(h(s)), - V’
_{free}:= E(α v_{free}(s)), W’ := E(α w(s)), H’ := E(α h(s)), - Y := E(β
_{v}v_{free}(s) + β_{w}w(s)))

where the last part is used to check that the correct polynomials were used (this is the part we did not cover yet in the other example). Note that all these encrypted values can be generated by the prover knowing only the CRS.

The task of the verifier is now the following:

Since the values of a_{k}, where k is not a “free” index can be computed directly from the input u (which is also known to the verifier, this is what is to be verified), the verifier can compute the missing part of the full sum for v:

- E(v
_{in}(s)) = E(Σ_{k}a_{k}v_{k}(s)) where the k ranges over all indices*not*in I_{free}.

With that, the verifier now confirms the following equalities using the pairing function e (don’t be scared):

- e(V’
_{free}, g) = e(V_{free}, g^{α}), e(W’, E(1)) = e(W, E(α)), e(H’, E(1)) = e(H, E(α)) - e(E(γ), Y) = e(E(β
_{v}γ), V_{free}) e(E(β_{w}γ), W) - e(E(v
_{0}(s)) E(v_{in}(s)) V_{free}, E(w_{0}(s)) W) = e(H, E(t(s)))

To grasp the general concept here, you have to understand that the pairing function allows us to do some limited computation on encrypted values: We can do arbitrary additions but just a single multiplication. The addition comes from the fact that the encryption itself is already additively homomorphic and the single multiplication is realized by the two arguments the pairing function has. So e(W’, E(1)) = e(W, E(α)) basically multiplies W’ by 1 in the encrypted space and compares that to W multiplied by α in the encrypted space. If you look up the value W and W’ are supposed to have – E(w(s)) and E(α w(s)) – this checks out if the prover supplied a correct proof.

If you remember from the section about evaluating polynomials at secret points, these three first checks basically verify that the prover did evaluate some polynomial built up from the parts in the CRS. The second item is used to verify that the prover used the correct polynomials v and w and not just some arbitrary ones. The idea behind is that the prover has no way to compute the encrypted combination E(β_{v} v_{free}(s) + β_{w} w(s))) by some other way than from the exact values of E(v_{free}(s)) and E(w(s)). The reason is that the values β_{v} are not part of the CRS in isolation, but only in combination with the values v_{k}(s) and β_{w} is only known in combination with the polynomials w_{k}(s). The only way to “mix” them is via the equally encrypted γ.

Assuming the prover provided a correct proof, let us check that the equality works out. The left and right hand sides are, respectively

- e(E(γ), Y) = e(E(γ), E(β
_{v}v_{free}(s) + β_{w}w(s))) = e(g, g)^{γ(βv vfree(s) + βw w(s))} - e(E(β
_{v}γ), V_{free}) e(E(β_{w}γ), W) = e(E(β_{v}γ), E(v_{free}(s))) e(E(β_{w}γ), E(w(s))) = e(g, g)^{(βv γ) vfree(s)}e(g, g)^{(βw γ) w(s)}= e(g, g)^{γ(βv vfree(s) + βw w(s))}

The third item essentially checks that (v_{0}(s) + a_{1}v_{1}(s) + … + a_{m}v_{m}(s)) (w_{0}(s) + b_{1}w_{1}(s) + … + b_{m}w_{m}(s)) = h(s) t(s), the main condition for the QSP problem. Note that multiplication on the encrypted values translates to addition on the unencrypted values because E(x) E(y) = g^{x} g^{y} = g^{x+y} = E(x + y).

As I said in the beginning, the remarkable feature about zkSNARKS is rather the succinctness than the zero-knowledge part. We will see now how to add zero-knowledge and the next section will be touch a bit more on the succinctness.

The idea is that the prover “shifts” some values by a random secret amount and balances the shift on the other side of the equation. The prover chooses random δ_{free}, δ_{w} and performs the following replacements in the proof

- v
_{free}(s) is replaced by v_{free}(s) + δ_{free}t(s) - w(s) is replaced by w(s) + δ
_{w}t(s).

By these replacements, the values V_{free} and W, which contain an encoding of the witness factors, basically become indistinguishable form randomness and thus it is impossible to extract the witness. Most of the equality checks are “immune” to the modifications, the only value we still have to correct is H or h(s). We have to ensure that

- (v
_{0}(s) + a_{1}v_{1}(s) + … + a_{m}v_{m}(s)) (w_{0}(s) + b_{1}w_{1}(s) + … + b_{m}w_{m}(s)) = h(s) t(s), or in other words - (v
_{0}(s) + v_{in}(s) + v_{free}(s)) (w_{0}(s) + w(s)) = h(s) t(s)

still holds. With the modifications, we get

- (v
_{0}(s) + v_{in}(s) + v_{free}(s) + δ_{free}t(s)) (w_{0}(s) + w(s) + δ_{w}t(s))

and by expanding the product, we see that replacing h(s) by

- h(s) + δ
_{free}(w_{0}(s) + w(s)) + δ_{w}(v_{0}(s) + v_{in}(s) + v_{free}(s)) + (δ_{free}δ_{w}) t(s)

will do the trick.

As you have seen in the preceding sections, the proof consists only of 7 elements of a group (typically an elliptic curve). Furthermore, the work the verifier has to do is checking some equalities involving pairing functions and computing E(v_{in}(s)), a task that is linear in the input size. Remarkably, neither the size of the witness string nor the computational effort required to verify the QSP (without SNARKs) play any role in verification. This means that SNARK-verifying extremely complex problems and very simple problems all take the same effort. The main reason for that is because we only check the polynomial identity for a single point, and not the full polynomial. Polynomials can get more and more complex, but a point is always a point. The only parameters that influence the verification effort is the level of security (i.e. the size of the group) and the maximum size for the inputs.

It is possible to reduce the second parameter, the input size, by shifting some of it into the witness:

Instead of verifying the function f(u, w), where u is the input and w is the witness, we take a hash function h and verify

- f'(H, (u, w)) := f(u, w) ∧ h(u) = H.

This means we replace the input u by a hash of the input h(u) (which is supposed to be much shorter) and verify that there is some value x that hashes to H(u) (and thus is very likely equal to u) in addition to checking f(x, w). This basically moves the original input u into the witness string and thus increases the witness size but decreases the input size to a constant.

This is remarkable, because it allows us to verify arbitrarily complex statements in constant time.

Since verifying arbitrary computations is at the core of the Ethereum blockchain, zkSNARKs are of course very relevant to Ethereum. With zkSNARKs, it becomes possible to not only perform secret arbitrary computations that are verifiable by anyone, but also to do this efficiently.

Although Ethereum uses a Turing-complete virtual machine, it is currently not yet possible to implement a zkSNARK verifier in Ethereum. The verifier tasks might seem simple conceptually, but a pairing function is actually very hard to compute and thus it would use more gas than is currently available in a single block. Elliptic curve multiplication is already relatively complex and pairings take that to another level.

Existing zkSNARK systems like zCash use the same problem / circuit / computation for every task. In the case of zCash, it is the transaction verifier. On Ethereum, zkSNARKs would not be limited to a single computational problem, but instead, everyone could set up a zkSNARK system for their specialized computational problem without having to launch a new blockchain. Every new zkSNARK system that is added to Ethereum requires a new secret trusted setup phase (some parts can be re-used, but not all), i.e. a new CRS has to be generated. It is also possible to do things like adding a zkSNARK system for a “generic virtual machine”. This would not require a new setup for a new use-case in much the same way as you do not need to bootstrap a new blockchain for a new smart contract on Ethereum.

There are multiple ways to enable zkSNARKs for Ethereum. All of them reduce the actual costs for the pairing functions and elliptic curve operations (the other required operations are already cheap enough) and thus allows also the gas costs to be reduced for these operations.

- improve the (guaranteed) performance of the EVM
- improve the performance of the EVM only for certain pairing functions and elliptic curve multiplications

The first option is of course the one that pays off better in the long run, but is harder to achieve. We are currently working on adding features and restrictions to the EVM which would allow better just-in-time compilation and also interpretation without too many required changes in the existing implementations. The other possibility is to swap out the EVM completely and use something like eWASM.

The second option can be realized by forcing all Ethereum clients to implement a certain pairing function and multiplication on a certain elliptic curve as a so-called precompiled contract. The benefit is that this is probably much easier and faster to achieve. On the other hand, the drawback is that we are fixed on a certain pairing function and a certain elliptic curve. Any new client for Ethereum would have to re-implement these precompiled contracts. Furthermore, if there are advancements and someone finds better zkSNARKs, better pairing functions or better elliptic curves, or if a flaw is found in the elliptic curve, pairing function or zkSNARK, we would have to add new precompiled contracts.

The post zkSNARKs in a nutshell appeared first on Ethereum Blog.

]]>The post Ethereum Research Update appeared first on Ethereum Blog.

]]>In the midst of these events, we have seen great progress from the C++ and Go development teams, including improvements to Solidity tools and the release of the Geth light client, and the Parity, EthereumJ and other external development teams have continued pushing forward on their own with technologies such as Parity’s warp sync; many of these innovations have already made their way into the hands of the average user, and still others are soon to come. At the same time, however, a large amount of quiet progress has been taking place on the research side, and while that progress has in many cases been rather blue-sky in nature and low-level protocol improvements necessarily take a while to make it into the main Ethereum network, we expect that the results of the work will start to bear fruit very soon.

Metropolis is the next major planned hardfork for Ethereum. While Metropolis is not quite as ambitious as Serenity and will not include proof of stake, sharding or any other similarly large sweeping changes to how Ethereum works, it is expected to include a series of small improvements to the protocol, which are altogether much more substantial than Homestead. Major improvements include:

- EIP 86 (account security abstraction) – move the logic for verifying signatures and nonces into contracts, allowing developers to experiment with new signature schemes, privacy-preserving technologies and modifications to parts of the protocol without requiring further hard forks or support at the protocol level. Also allows contracts to pay for gas.
- EIP 96 (blockhash and state root changes) – simplifies the protocol and client implementations, and allows for upgrades to light client and fast-syncing protocols that make them much more secure.
- Precompiled/native contracts for elliptic curve operations and big integer arithmetic, allowing for applications based on ring signatures or RSA cryptography to be implemented efficiently
- Various improvements to efficiency that allow faster transaction processing

Much of this work is part of a long-term plan to move the protocol toward what we call *abstraction*. Essentially, instead of having complex protocol rules governing contract creation, transaction validation, mining and various other aspects of the system’s behavior, we try to put as much of the Ethereum protocol’s logic as possible into the EVM itself, and have protocol logic simply be a set of contracts. This reduces client complexity, reduces the long-run risk of consensus failures, and makes hard forks easier and safer – potentially, a hard fork could be specified simply as a config file that changes the code of a few contracts. By reducing the number of “moving parts” at the bottom level of the protocol in this way, we can greatly reduce Ethereum’s attack surface, and open up more parts of the protocol to user experimentation: for example, instead of the protocol upgrading to a new signature scheme all at the same time, users are free to experiment and implement their own.

Over the past year, research on proof of stake and sharding has been quietly moving forward. The consensus algorithm that we have been working on, Casper, has gone through several iterations and proof-of-concept releases, each of which taught us important things about the combination of economics and decentralized consensus. PoC release 2 came at the start of this year, although that approach has now been abandoned as it has become obvious that requiring every validator to send a message every block, or even every ten blocks, requires far too much overhead to be sustainable. The more traditional chain-based PoC3, as described in the Mauve Paper, has been more successful; although there are imperfections in how the incentives are structured, the flaws are much less serious in nature.

Myself, Vlad and many volunteers from Ethereum research team came together at the bootcamp at IC3 in July with university academics, Zcash developers and others to discuss proof of stake, sharding, privacy and other challenges, and substantial progress was made in bridging the gap between our approach to proof of stake and that of others who have been working on similar problems. A newer and simpler version of Casper began to solidify, and myself and Vlad continued on two separate paths: myself aiming to create a simple proof of stake protocol that would provide desirable properties with as few changes from proof of work as possible, and Vlad taking a “correct-by-construction” approach to rebuild consensus from the ground up. Both were presented at Devcon2 in Shanghai in September, and that’s where we were at two weeks ago.

At the end of November, the research team (temporarily joined by Loi Luu, of validator’s dilemma fame), along with some of our long-time volunteers and friends, came together for two weeks for a research workshop in Singapore, aiming to bring our thoughts together on various issues to do with Casper, scalability, consensus incentives and state size control.

A major topic of discussion was coming up with a rigorous and generalizable strategy for determining optimal incentives in consensus protocols – whether you’re creating a chain-based protocol, a scalable sharding protocol, or even an incentivized version of PBFT, can we come up with a generalized way to correctly assign the right rewards and penalties to all participants, using only verifiable evidence that could be put into a blockchain as input, and in a way that would have optimal game-theoretic properties? We had some ideas; one of them, when applied to proof of work as an experiment, immediately led to a new path toward solving selfish mining attacks, and has also proven extremely promising in addressing long-standing issues in proof of stake.

A key goal of our approach to cryptoeconomics is ensuring as much incentive-compatibility as possible even under a model with majority collusions: even if an attacker controls 90% of the network, is there a way to make sure that, if the attacker deviates from the protocol in any harmful way, the attacker loses money? At least in some cases, such as short-range forks, the answer seems to be yes. In other cases, such as censorship, achieving this goal is much harder.

A second goal is bounding “griefing factors” – that is, ensuring that there is no way for an attacker to cause other players to lose money without losing close to the same amount of money themselves. A third goal is ensuring that the protocol continues to work as well as possible under other kinds of extreme conditions: for example, what if 60% of the validator nodes drop offline simultaneously? Traditional consensus protocols such as PBFT, and proof of stake protocols inspired by such approaches, simply halt in this case; our goal with Casper is for the chain to continue, and even if the chain can’t provide all of the guarantees that it normally does under such conditions the protocol should still try to do as much as it can.

One of the main beneficial results of the workshop was bridging the gap between my current “exponential ramp-up” approach to transaction/block finality in Casper, which rewards validators for making bets with increasing confidence and penalizes them if their bets are wrong, and Vlad’s “correct-by-construction” approach, which emphasizes penalizing validators only if they equivocate (ie. sign two incompatible messages). At the end of the workshop, we began to work together on strategies to combine the best of both approaches, and we have already started to use these insights to improve the Casper protocol.

In the meantime, I have written some documents and FAQs that detail the current state of thinking regarding proof of stake, sharding and Casper to help bring anyone interested up to speed:

https://github.com/ethereum/wiki/wiki/Proof-of-Stake-FAQ

https://github.com/ethereum/wiki/wiki/Sharding-FAQ

https://docs.google.com/document/d/1maFT3cpHvwn29gLvtY4WcQiI6kRbN_nbCf3JlgR3m_8 (Mauve Paper; now slightly out of date but will be updated soon)

Another important area of protocol design is state size control – that is, how to we reduce the amount of state information that full nodes need to keep track of? Right now, the state is about a gigabyte in size (the rest of the data that a geth or parity node currently stores is the transaction history; this data can theoretically be pruned once there is a robust light-client protocol for fetching it), and we saw already how protocol usability degrades in several ways if it grows much larger; additionally, sharding becomes much more difficult as sharded blockchains require nodes to be able to quickly download parts of the state as part of the process of serving as validators.

Some proposals that have been raised have to do with deleting old non-contract accounts with not enough ether to send a transaction, and doing so safely so as to prevent replay attacks. Other proposals involve simply making it much more expensive to create new accounts or store data, and doing so in a way that is more decoupled from the way that we pay for other kinds of costs inside the EVM. Still other proposals include putting time limits on how long contracts can last, and charging more to create accounts or contracts with longer time limits (the time limits here would be generous; it would still be affordable to create a contract that lasts several years). There is currently an ongoing debate in the developer community about the best way to achieve the goal of keeping state size small, while at the same time keeping the core protocol maximally user and developer-friendly.

Other areas of low-level-protocol improvement on the horizon include:

- Several “EVM 1.5” proposals that make the EVM more friendly to static analysis, facilitating compatibility with WASM
- Integration of zero knowledge proofs, likely through either (i) an explicit ZKP opcode/native contract, or (ii) an opcode or native contract for the key computationally intensive ingredients in ZKPs, particularly elliptic curve pairing computations
- Further degrees of abstraction and protocol simplification

Expect more detailed documents and conversations on all of these topics in the months to come, especially as work on turning the Casper specification into a viable proof of concept release that could run a testnet continues to move forward.

The post Ethereum Research Update appeared first on Ethereum Blog.

]]>