HomeEIPsNewsletter
EIPsEIP-7688
EIP-7688

Forward compatible consensus data structures

Transition consensus SSZ data structures to ProgressiveContainer
DraftStandards Track: Core
Created: 2024-04-15
Requires: EIP-6110, EIP-7002, EIP-7251, EIP-7495, EIP-7549, EIP-7569, EIP-7916
Etan Kissling (@etan-status), Cayman (@wemeetagain)
Discussions ForumOriginal Proposal LinkEdit
1 min read
Anyone may contribute to propose contents.
Go propose
Video
Anyone may contribute to propose contents.
Go propose
Original

Abstract

This EIP defines the changes needed to adopt ProgressiveContainer from EIP-7495 and ProgressiveList from EIP-7916 in consensus data structures.

Motivation

Ethereum's consensus data structures make heavy use of Simple Serialize (SSZ) Container, which defines how they are serialized and merkleized. The merkleization scheme allows application implementations to verify that individual fields (and partial fields) have not been tampered with. This is useful, for example, in smart contracts of decentralized staking pools that wish to verify that participating validators have not been slashed.

While SSZ Container defines how data structures are merkleized, the merkleization is prone to change across the different forks. When that happens, e.g., because new features are added or old features get removed, existing verifier implementations need to be updated to be able to continue processing proofs.

ProgressiveContainer, of EIP-7495, is a forward compatible alternative that guarantees a forward compatible merkleization scheme. By transitioning consensus data structures to use ProgressiveContainer, smart contracts that contain verifier logic no longer have to be maintained in lockstep with Ethereum's fork schedule as long as the underlying features that they verify don't change. For example, as long as the concept of slashing is represented using the boolean slashed field, existing verifiers will not break when unrelated features get added or removed. This is also true for off-chain verifiers, e.g., in hardware wallets or in operating systems for mobile devices that are on a different software update cadence than Ethereum.

Specification

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 and RFC 8174.

Container conversion

Container types that are expected to evolve over forks SHALL be redefined as ProgressiveContainer[active_fields=[1] * len(type.fields())].

For example, given a type in the old fork:

class Foo(Container): a: uint8 b: uint16

This type can be converted to support stable Merkleization in the new fork:

class Foo(ProgressiveContainer[active_fields=[1, 1]]): a: uint8 b: uint16

As part of the conversion, a stable generalized index (gindex) is assigned to each field that remains valid in future forks.

  • If a fork appends a field, active_fields MUST be extended with a trailing 1.
  • If a fork removes a field, the corresponding active_fields bit MUST be changed to 0.
  • Compatibility rules SHOULD be enforced, e.g., by defining a CompatibleUnion[fork_1.Foo, fork_2.Foo, fork_3.Foo, ...] type in the unit test framework.

List[type, N] / Bitlist conversion

List types frequently have been defined with excessively large capacities N with the intention that N is never reached in practice. In other cases, the capacity itself has changed over time.

  • List types with dynamic or unbounded capacity semantics SHALL be redefined as ProgressiveList[type], and the application logic SHALL be updated to check for an appropriate limit at runtime.
  • Bitlist types with dynamic or unbounded capacity semantics SHALL be redefined as ProgressiveBitlist

As part of the conversion, a stable generalized index (gindex) is assigned to each list element that remains valid regardless of the number of added elements.

Converted types

The following types SHALL be converted to ProgressiveContainer:

  • Attestation
    • The aggregation_bits field is redefined to use ProgressiveBitlist
  • IndexedAttestation
    • The attesting_indices field is redefined to use ProgressiveList
  • ExecutionPayload
    • The transactions and withdrawals fields are redefined to use ProgressiveList
  • ExecutionPayloadHeader
  • ExecutionRequests
    • The deposits, withdrawals and consolidation fields are redefined to use ProgressiveList
  • BeaconBlockBody
    • The proposer_slashings, attester_slashings, attestations, deposits, voluntary_exits, bls_to_execution_changes and blob_kzg_commitments fields are redefined to use ProgressiveList
  • BeaconState
    • The validators, balances, previous_epoch_participation, current_epoch_participation, inactivity_scores, pending_deposits, pending_partial_withdrawals and pending_consolidations fields are redefined to use ProgressiveList

Immutable types

These types are used as part of the ProgressiveContainer definitions, and, as they are not ProgressiveContainer themselves, are considered to have immutable Merkleization. If a future fork requires changing these types in an incompatible way, a new type SHALL be defined and assigned a new field name.

TypeDescription
SlotSlot number on the beacon chain
EpochEpoch number on the beacon chain, a group of slots
CommitteeIndexIndex of a committee within a slot
ValidatorIndexUnique index of a beacon chain validator
GweiAmount in Gwei (1 ETH = 10^9 Gwei = 10^18 Wei)
RootByte vector containing an SSZ Merkle root
Hash32Byte vector containing an opaque 32-byte hash
VersionConsensus fork version number
BLSPubkeyCryptographic type representing a BLS12-381 public key
BLSSignatureCryptographic type representing a BLS12-381 signature
KZGCommitmentG1 curve point for the KZG polynomial commitment scheme
ForkConsensus fork information
CheckpointTuple referring to the most recent beacon block up through an epoch's start slot
ValidatorInformation about a beacon chain validator
AttestationDataVote that attests to the availability and validity of a particular consensus block
Eth1DataTarget tracker for importing deposits from transaction logs
DepositDataLog data emitted as part of a transaction's receipt when depositing to the beacon chain
BeaconBlockHeaderConsensus block header
ProposerSlashingTuple of two equivocating consensus block headers
DepositTuple of deposit data and its inclusion proof
VoluntaryExitConsensus originated request to exit a validator from the beacon chain
SignedVoluntaryExitTuple of voluntary exit request and its signature
SyncAggregateCryptographic type representing an aggregate sync committee signature
ExecutionAddressByte vector containing an account address on the execution layer
TransactionByte list containing an RLP encoded transaction
WithdrawalIndexUnique index of a withdrawal from any validator's balance to the execution layer
WithdrawalWithdrawal from a beacon chain validator's balance to the execution layer
DepositRequestTuple of flattened deposit data and its sequential index
WithdrawalRequestExecution originated request to withdraw from a validator to the execution layer
ConsolidationRequestExecution originated request to consolidate two beacon chain validators
BLSToExecutionChangeRequest to register the withdrawal account address of a beacon chain validator
SignedBLSToExecutionChangeTuple of withdrawal account address registration request and its signature
ParticipationFlagsParticipation tracker of a beacon chain validator within an epoch
HistoricalSummaryTuple combining a historical block root and historical state root
PendingDepositPending operation for depositing to a beacon chain validator
PendingPartialWithdrawalPending operation for withdrawing from a beacon chain validator
PendingConsolidationPending operation for consolidating two beacon chain validators

Rationale

Best timing?

Applying this EIP breaks hash_tree_root and Merkle tree verifiers a single time, while promising forward compatibility from the fork going forward. It is best to apply it before merkleization would be broken by different changes. Merkleization is broken by a Container reaching a new power of 2 in its number of fields.

Can this be applied retroactively?

While Profile serializes in the same way as the legacy Container, the merkleization and hash_tree_root of affected data structures changes. Therefore, verifiers that wish to process Merkle proofs of legacy variants still need to support the corresponding legacy schemes.

Immutability

Once a field in a ProgressiveContainer has been published, its name can no longer be used to represent a different type in the future. This is in line with historical management of certain cases:

  • Phase0: BeaconState contained previous_epoch_attestations / current_epoch_attestations
  • Altair: BeaconState replaced these fields with previous_epoch_participation / current_epoch_participation

Furthermore, new fields have to be appended at the end of ProgressiveContainer. This is in line with historical management of other cases:

  • Capella appended historical_summaries to BeaconState instead of squeezing the new field next to historical_roots

With ProgressiveContainer, stable Merkleization requires these rules to become strict.

Cleanup opportunity

Several fields in the BeaconState are no longer relevant in current specification versions.

  • The eth1_data, eth1_data_votes, eth1_deposit_index and deposit_requests_start_index fields could be dropped as they are no longer needed post Fulu.
  • historical_summaries could be redefined to use ProgressiveList and also integrate the historical historical_roots data by merging in full HistoricalSummary data from an archive (historical_root is frozen since Capella), simplifying access to historical block and state roots.

BeaconBlockHeader?

Updating the BeaconBlockHeader to ProgressiveContainer is tricky as is breaks hash_tree_root(latest_block_header) in the BeaconState. One option could be to store latest_block_header_root separately, possibly also incorporating the block proposer signature into the hash to avoid proposer signature checks while backfilling historical blocks.

Validator?

Updating the Validator to ProgressiveContainer would add an extra hash for each validator; validators are mostly immutable so rarely need rehashing. Due to the large hash count, implementations may have to incrementally construct the new Validator entries ahead of the fork. It should be evaluated whether the hashing overhead is worth a clean transition to future fields, e.g., for holding postquantum keys.

Backwards Compatibility

Existing Merkle proof verifiers need to be updated to support the new Merkle tree shape. This includes verifiers in smart contracts on different blockchains and verifiers in hardware wallets, if applicable.

Security Considerations

None

Copyright and related rights waived via CC0.

Further reading
Anyone may contribute to propose contents.
Go propose

Not miss a beat of EIPs' update?

Subscribe EIPs Fun to receive the latest updates of EIPs Good for Buidlers to follow up.

View all
Serve Ethereum Builders, Scale the Community.
Resources
GitHub