The network upgrade is being postponed to a later date.
This post is an expansion on the post published originally at https://blog.ethereum.org/2019/01/15/security-alert-ethereum-constantinople-postponement/.
The Ethereum Core Developers and the Ethereum Security Community were made aware of the potential Constantinople-related issues identified by ChainSecurity on January 15, 2019.
Out of an abundance of caution, key stakeholders around the Ethereum community have determined that the best course of action will be to delay the planned Constantinople fork that would have occurred at block 7,080,000 on January 16, 2019.
This will require anyone running a node (node operators, exchanges, miners, wallet services, etc.) to update to a new version of Geth or Parity before block 7080000.
Block 7080000 will occur in approximately 32 hours from the time of this publishing or at approximately Jan 16, 8:00pm PT / Jan 16, 11:00pm ET / Jan 17, 4:00am GMT.
What You Need To Do
If you are a person who simply interacts with Ethereum (you do not run a node), you do not need to do anything.
Miners, Exchanges, Node Operators
- Update your Geth and/or Parity instances when they are released.
- These releases are not released yet. We will update this post when they are available.
- Links and version numbers and instructions will be provided here when they are available.
- We expect to have updated releases in 3–4 hours from the time this blog is published.
Geth
- Upgrade to Geth Version 1.8.21 (recommended)
- Downgrade to Geth Version 1.8.19
- Remain on 1.8.20, but use the switch — override.constantinople=9999999 to postpone the Constantinople fork indefinitely.
Parity
- Upgrade to Parity Ethereum 2.2.7-stable (recommended)
- Upgrade to Parity Ethereum 2.3.0-beta
- Downgrade to Parity Ethereum 2.2.4-beta (not recommended)
Nethermind
- Use the latest Nethermind
Pantheon
- Use Pantheon 0.8.5
Everyone Else
Ledger, Trezor, Safe-T, Parity Signer, WallEth, Paper Wallets, MyCrypto, MyEtherWallet and other users or token holders that do not participate in the network by syncing and running a node:
- You do not have to do anything.
Contract owners:
- You do not have to do anything.
- You may choose to examine the analysis of the potential vulnerability and check your contracts.
- However, you do not have to do anything as the change that would introduce this potential vulnerability will not be enabled.
Background
The article by ChainSecurity dives deep into the potential vulnerability and how smart contracts can be checked for the vulnerability. Very briefly:
- EIP-1283 introduces cheaper gas cost for SSTORE operations
- Some smart contracts (that are already on chain) may utilize code patterns that would make them vulnerable to a re-entrancy attack after the Constantinople upgrade takes place
- These smart contracts would not have been vulnerable before the Constantinople upgrade
ChainSecurity’s article goes into details about the attack and what can be done to check if your smart contract code is vulnerable.
Contracts that increase their probability to being vulnerable are contracts that utilize a transfer() or send() function followed by a state-changing operation. An example of such a contract would be one where two parties jointly receive funds, decide on how to split said funds, and initiate a payout of those funds.
How the decision to postpone Constantinople was made
Security researchers like ChainSecurity and TrailOfBits ran (and are still running) analysis across the entire blockchain. They did not find any cases of this vulnerability in the wild. However, there is still a non-zero risk that some contracts could be affected.
The discussions around this decision mostly revolved risk analysis and attempting to answer the following core questions:
- “Are there any contracts currently on chain that are vulnerable to this attack?”
- “How long will it take to determine if there are any vulnerable smart contracts?”
- “What action should be taken in regards to the upcoming Constantinople fork now?”
- “Do we have enough time to analyze the risk?”
- “Do smart-contract developers currently assume that using transfer() and send() is this way is safe?”
Because the risk is non-zero and the amount of time required to determine the risk with confidence is longer the amount of time available before the planned Constantinople upgrade, a decision was reached to postpone the fork out of an abundance of caution.
Parties involved in the discussions included, but were not limited to:
- Security researchers.
- Ethereum stakeholders.
- Ethereum client developers.
- Smart contract owners / developers.
- Wallet providers.
- Node operators.
- Dapp developers.
- Media.
Discussions happened across a variety of channels, including but not limited to developer channels across gitter, security channels across Skype and Telegram, a Zoom conference call, and many many others.
Response Time
- 3:09am PT: ChainSecurity responsibly discloses potentially vulnerability via Ethereum Foundation’s bug bounty program.
- 8:04am PT: Original article by ChainSecurity was published.
- 8:52am PT: Martin Holst Swende posts in ethsecurity and AllCoreDevs Gitter channel: “Please read: https://medium.com/chainsecurity/constantinople-enables-new-reentrancy-attack-ace4088297d9 @/all We need a quick decision on potential consequences and how to move forward. We have about 37 hours left until the fork happens”
- 8:52am PT — 10:15am PT: Discussion occurs across various channels regarding potential risks, on-chain analysis, and what steps need to be taken
- 10:15am PT — 12:40pm PT: Discussion via Zoom audio call with key stakeholders. Discussion continues in gitter and other channels as well.
- 12:08pm PT: Decision made to delay Constantinople upgrade.
- 1:15pm PT: Public blog post released across various channels and social media.
“Why wasn’t this detected sooner?”
It’s quite difficult to detect as it requires examining the entire stack and what types of smart contract patterns being actually being used.
This isn’t a bug in the EVM or the Geth/Parity/client implementations. Instead, it’s change that could potentially cause existing smart contracts to be exploited if they use a very specific pattern and take very specific actions consecutively. How this could play out:
- A developer wrote, audited, tested and deployed a smart contract in the past
- It is not possible to exploit the smart contract
- The Constantinople update goes live
- It is now possible to exploit the smart contract, due to the changes made in EIP1283
These exploits aren’t found by auditing the EIP or Geth or Parity. It is found by auditing every existing contract while that contract is on an already-updated chain, or by researchers imagining what devs could write that could be inadvertently exploitable.
Diving Deeper: Action Time
Even before the decision to delay Constantinople was made, key developers and stake-holders were already preparing for the delay, just in case. If the decision was made to not delay the fork, no action would need to be taken. However, if the decision was made to delay the fork, the work would already be underway. This includes but isn’t limited to:
- Security firms analyzing smart contracts and collaborating on analysis tools and types
- Geth developers determining plan of action for Geth users, including preparing and update and instructions.
- Parity developers preparing a plan of action for Parity users, including a potential new release.
- Communications people working together on plan of action for notifying exchanges, miners, node operators, as well as beginning a detailed blog post.
Diving Deeper: Some Key Points From The Discussions
It’s always impressive to see various stakeholders come together in times like these. Here is a small sampling of the discussions being had and points made in the initial hours (all timestamps in PT).
Diving Deeper: The Audio Call
For those who weren’t able to join the call, we tried to take notes but there was a lot happening and a lot of things happening simultaneously. This is not exhaustive and information below may not be 100% verbatim of what was said.
- Brief description of potential vulnerability outlined by ChainSecurity
- What types of contracts have highest risk of being affected (Multisigs? Token contracts?)
- Do we continue with Constantinople? Do we postpone?
- Do we remove the change around SSTORE? (EIP 1283)? (Decided that this should be saved for later discussion — first we need to decide on postponement of Constantinople first)
- We are waiting for second analysis from TrailOfBits team to help make the decision
- Anyone familiar with Solidity should be looking for any real-world cases
- If decide to postpone… (1) Geth and Parity do new releases (2) Everyone (miners/exchanges) needs to be informed and pushed to update their software ASAP
- TrailOfBits: Hasn’t found any vulnerable contracts yet but needs more time to be more confident. Tool they use is a static analyzer. Looking for if a send is followed by an access / read / write variable.
- Discussion about vulnerability and how to find the vuln
- Points out it is a bit different than The DAO re-ent case and is also a bit harder to find that the “classic” re-ent attack vector
- Martin: “how much time would it take for you to complete analysis / be more confident in findings”
- TrailOfBits: “Will probably be difficult to be fully confident before hard fork block. We will be working hard on it today / tonight.”
- Conclusion that we will not be exhaustive analysis done before Constantinople
- Shared lists of smart contracts that have been checked, etc. is created
- Recommendation to delay the fork: “Impossible to say for sure that there is nothing out there, but haven’t found any yet”
- Ethereum Cat Herders can contact 75% of miners, exchanges via Skype group, exchange email list
- Comms group created on Telegram for coordinating reaching out to people, creating messaging, answering questions, reducing FUD, scammers, etc.
- Risk of postpone: (1) Attacks on small exchanges (2) Scammers taking advantage of confusion (3) Possible continued fork of people who have spun up nodes (4) FUD exacerbation
- Conversation around how exactly postponement would work: (1) Do we hardcore a block number? (2) What does the “fix” include (3) Need to update tests and testnets as well (4) Suspect a new testnet may be needed (5) Moving forward, do we pull EIP 1283? Implement a fix?
Some Lessons Learned
- Modifying opcodes / the EVM can have unintended consequences on already-deployed contracts
- Changes of these types need to be made exceedingly carefully
- Smart contract developer assumptions and patterns being used by developers should also be analyzed when reviewing EIPs
- Ethereum clients should have a way for node operators to simply run with specific flags so creating and deploying releases isn’t necessary in situations like this
- Detecting unintended consequences of updates require imagination / research across the entire stack, not just technical reviews or audits
Key Takeaways
- A lot of people react very quickly when potential security vulnerabilities come to light.
- The diversity of people adding to the conversation and contributing to a solution was impressive: technical people, not technical people, dapp developers, smart contract developers, Ethereum Foundation people, services like Infura, MyCrypto, and Status all chimed in.
- Making decisions when you don’t have all the information necessary AND there is not enough time to get that information is difficult.
- Collaboration and respectful communication and healthy debate makes things much better.
- While the technical discussions and analysis always come first, having people who are focused on communicating that information is also a necessary priority.
- Response time was pretty epic.
- Making decisions out of an abundance of caution isn’t always fun, but better safe than sorry, especially when dealing with things that hold immense monetary value.
- How should EIP’s be analyzed? Is excessive community discussion / time enough? External audits? In this case, this EIP was open for a long time and had a lot of eyes on it.
This article was put together in a collaborative effort by EvanVanNess, Infura, MyCrypto, Parity, Status, The Ethereum Foundation, and the Ethereum Cat Herders. Additional information and deep-dives written by MyCrypto.
