Roundup #6

Metropolis is finally (almost) here! The fork for Byzantium, the first and larger part of Metropolis, succeeded on the testnet over two weeks ago, and the likely date for the fork on the mainnet has been set to block 4.37 million, which is expected to be on Oct 17. New features include opcodes such as REVERT and RETURNDATACOPY, as well as precompiles that can be used to support a wide array of cryptographic algorithms. At the same time, we have been seeing many improvements to Ethereum core code, Whisper, Swarm as well as Ethereum’s future scaling plans.

• Casper PoC4 has been released. This includes an implementation of the fork choice rule, the Casper contract, and a complete pyethereum library, though not yet a full node that can connect to the network.
• A “testing language” has been implemented that allows us to quickly implement tests for the Casper chain. This can also theoretically be used for the proof of work chain, and an extension to sharding is in progress.
• Implementation of a proof of concept for sharding is in progress.
• Implementation of the account redesign in the sharding PoC is in progress.
• The Casper papers continue to be in progress.
• The number of message types in Casper FFG has been reduced from 2 to 1, which will also simplify the incentive structure. A formal proof of the safety property has been written. This will be incorporated in PoC5.
• The fork choice rule has also been simplified.
• Pyethapp now supports python 3.
• The "scalable light client data availability verification" note has been edited with an improved scheme

• Cross-contract ABI calling
• ABI logging
• Support for ECADD (Elliptic Curve Addition) , ECMUL (Elliptic Curve Multiplication) and other precompiles
• Support for features specific to the current sharding specification
• General code refactoring
• More examples
• Easier compatibility with ERC20 and other Solidity-developed ABIs

We are making progress with syncing to the test and main networks. Now cpp-ethereum can import warp snapshot and continue syncing from the snapshot block.The Byzantium implementation has been finished. Since the last roundup:

• Documentation on DB layout was added
• The ice-age delay and the reward change
• Simpler rules for address collision were implemented
• Several deadlock and race condition bugs were fixed
• Unit test improvements have been made
• Sync assertion failure fix was implemented
• More race conditions are being discovered and fixed
• The final pull request for the first candidate of C API for EVM (EVM-C). Tracing is still missing though. See the evm.h header file.

Since the last roundup, Geth saw a major release (v1.7.0) and a follow up maintenance release (v1.7.1) with primary focus being on the upcoming Byzantium hard fork (enabled for all networks as of v1.7.1).

In addition to Byzantium, a few features finalized (some of which you may have seen in the v1.7.0 Megara blogpost):

• EVM log storage and indexing was completely reworked, cutting the filtering time of the entire chain for contract events by 2-3 orders of magnitude, from minutes to under a second.
• The base peer-to-peer protocol used by all Ethereum sub-protocols was updated to use Snappy compression, cutting the bandwidth needed for a fast sync from 33.6GB to 13.5GB and similar reductions are expected for day-to-day operational bandwidth requirements too. Since operational bandwidth relies on updated peers, its effect will become visible in the longer term.
• Rinkeby was extended with an authorized signer and a bootnode courtesy of the INFURA team, which should both help find peers faster, as well as provide a stabler experience due to the excess available bandwidth capacity on the test network.

• Generational disk writes to keep the recent trie modifications in system memory and only periodically store checkpoint tries. This feature greatly speeds up processing batches of blocks as well as significantly reduces final consumed disk space. However there are still some minor roadblocks to solve to ensure we don’t break sync for remote nodes due to the new data availability pattern in the network.
• A new variation of fast sync that does not download the entire state-trie node-by-node, but rather only pulls trie leaf nodes (along with supporting Merkle proofs), and assembles the final trie locally, which should have a huge impact on the number of network packets required to do the initial sync. That said, besides implementing it, this feature needs proper benchmarking to support it and an EIP process to debate its inclusion as the next version of the `eth` protocol.
• Version 5 of the DEVp2p discovery protocol (currently experimental and used by the light clients) to enable it to run concurrently with v4 on the same UDP port, as well as to iron out the kinks noticed from running the light client on top.
• Version 2 of the light client protocol, based on the new bloombits filtering mechanism, which will finally allow light clients to retrieve and subscribe to EVM events. The new version should also support tracking transactions for chain inclusion without the overhead of downloading entire blocks just to verify their transaction contents.
• Research into an ultra-light mode of operation where clients connect to a set of user defined backing servers similar to the classical client-server model, but where the node can cryptographically cross-validate the multiple backends to prevent them from cheating.

A plethora of bugfixes have also been merged into both releases, which you can skim through at the GitHub milestones for v1.7.0 and v1.7.1.

Testing

Test cases for Byzantium are ready and some clients passed all of them. After the last round up, we:

• introduced JSON schema checking. This ensures more stable formats of the testing suite, and prevents accidental typos in field names.
• separated inputs and outputs of doTests() functions. Before this PR, typos in the input files were copied into the output files.
• applied `const` attributes on inputs, and now the output files only contain fields that were explicitly added.
• consolidated options of testeth.
• added more graceful error handling in testeth.
• Replaced log entries in the test files with a keccak hash of the log RLP.
• opened a fix that will remedy some unwanted side-effects where translation of GeneralStateTests into BlockchainTests destroys the originals.

Yellow Paper

All Byzantium changes have been documented on pull-requests. The next issue to address is the treatment of the empty account states on precompiled contracts, where different clients do different things. A GitHub issue was created to discuss this.

Formal Verification

@pirapira gave talks at the ARM Research Summit and at TU Berlin. We Implemented RLP in OCaml, available with opam install rlp.

Bamboo

We started writing a specification. External interfaces are being implemented.

EVM 1.5

Solidity assembly can now handle EIP #615 opcodes for subroutines and the rest, thanks to Alex.  Initial implementations of both EIP #615 and EIP #616 are written, and reviewed by Andrei, and testing has begun.  Greg is working to present initial performance measurements at Devcon.

web3.js

web3.js 1.0 beta received a great deal of community contributions which is exciting as it appears 1.0 is already being used a lot in the wild. Please be careful using the account generation and signing, as it has not been audited. Those accounts should not hold any significant amount of ether. We are currently at beta.22. If everything goes well, we hope to have a 1.0.0 Release candidate by Devcon3.

web3.py

Web3.py recently added features and improvements including:

• Release of the Middleware API
• Reworking of the testing infrastructure to test against go-ethereum, pyethereum, and py-evm.
• Bugfixes and API improvements for the web3.eth.sign and web3.sha3 APIs as well as progress towards a unified approach to how different string encodings are handled.
• Experimental integration with the ethereum-tester library.

Development continues to progress, moving us closer to an early alpha release, which will allow running py-evm as a light client.

• Implementation of the basic DevP2P networking API.
• Journaling for state database changes.

Ethereum Tester is a new tool for python developers design to provide a standard API for testing applications that interacts with the EVM. The library uses a pluggable backend system that allows swapping out the underlying VM implementation. This library can also be used transparently with web3.py by using the experimental EthereumTesterProvider

Remix

Two very important PRs has been pushed recently: (1) a terminal that allows tracking transactions and interacting with web3, and (2) a revamp of the UI.

We are now fixing bugs and helping the community to adjust to this new GUI.

A number of new features are being worked on for release in the coming weeks:

- Highlight of declarations and references, basic autocomplete and renaming

https://github.com/ethereum/browser-solidity/pull/819

• Command interpreter for the terminal (used to command remix from the terminal)

• Save transactions as scenario that can be then replayed in any context

• Make remix UI themable

Solidity

We are close to finishing the new ABI en/decoder using the iulia language. This allows passing structs and arbitrarily nested arrays across function calls. It is also a little stricter when it comes to invalid input data including invalid array lengths. You can use pragma experimental ABIEncoderV2; to activate it. With a similar statement, pragma experimental “v0.5.0”; you can activate upcoming breaking changes, which we are adding one after the other. You can now use </span><a href="https://github.com/ethereum/solidity/pull/2473"><span style="font-weight: 400;">f.selector</span></a><span style="font-weight: 400;"> to access the selector (also called the “function hash”) of a function. Using the default visibility of a function will generate a warning now, and should make you aware of unintended public functions. Finally, Solidity is warning about (and more strictly enforcing if you opt to already activate v0.5.0 features) violations of “view” (formerly called “constant”) and “pure” annotations for functions. After Byzantium, a call to a “view” function will use the new STATICCALL opcode and thus enforce the “view” aspect of the function at VM level.

Mist

Released version 0.9.1:

• Merged the light client feature as an experimental option
• Updated electron, geth and solidity to enhance security
• A new core developer @marcgarreau started working on refactoring state management in Mist using Redux

A cross-language differential fuzzer of operations on the alt_bn128 elliptic curve, which are required to perform zkSNARK verifications, in three different clients (Parity, Geth and CPP), in order to find discrepancies between these implementations, has been built. Driven by the libFuzzer engine, the program autonomously creates inputs that lead to new code paths in any of the implementations. In addition to that, it validates that the output of each implementation for any input is the same.

Based on this work, a fuzzer that tests the Parity and Geth implementations of the EVM for equivalence is being developed.

We have been continuing cross-EVM fuzzing efforts in order to find discrepancies between EVM-implementations (Geth, CPP and Parity), and have doubled the bounty rewards for bugs related to consensus or denial-of-service (geth).

The hive-testing framework has been updated to test four clients (geth, cpp, py, parity) with the new Byzantium tests.

An external audit of Mist is in the final stages, and an RCE-vulnerability has been patched and released (credits to Yoonho Kim via the bug bounty program). We strongly recommend updating to the most recent version of Mist!

EthereumJS

Since the last roundup, several new contributors have stepped in to assist with development.  EthereumJS is now passing most Byzantium state and blockchain tests. The development effort continues with general bug fixes and code refactoring. The most notable of these efforts is a refactor of the library to add support for JavaScript big integers.

Swarm

The swarm team merged important new updates and additions into master including the network testing and simulations framework for p2p with a 3D visualiser, improved scheduling for breaking up and reassembling data -- the pyramid chunker with the new Binary Merkle Tree  chunk hash, which brings  compact inclusion proofs verifiable on the blockchain, and improved more informative error messages. The network layer rewrite (our major features for POC3) has completed the first phase and pss v1.0 is dev ready now with diffie hellman key-exchange and continually updated keystream.

Our team is engaging in ever more community cooperation to develop certain features on our roadmap by allowing external contributors to feel part of the wider team. We are launching 5 working groups about 5 of our most relevant subprojects: network simulation, pss node to node messaging, media and data streaming, database services and the swap swear and swindle framework for incentivizing bandwidth sharing and long-term storage of data. The latter is now gaining momentum as the third orange paper is being worked on with the draft version first to be presented on 11th October in the London Ethereum Meetup.

We just published the tentative program for Devcon3 breakout session on p2p technology and the blockchain (10:30am-4:30pm, 4th November, 2017).

Whisper

• A new core developer @gballet started working on fixing Whisper v5 and updating the documentation.
• v5 message delivery was broken by a slew of updates over the summer; fixes have been made and will be integrated soon.
• EIP 627 is finalized and implementation will begin shortly.