QUIC for P2P Networking

Tessera uses QUIC over UDP for secure, multiplexed validator communication. Built on aioquic, it provides TLS 1.3 encryption, stream multiplexing, and fast connection establishment.

Why QUIC?

Feature	Benefit
Stream Multiplexing	Multiple protocols over one connection
Built-in TLS 1.3	No separate encryption layer
Fast Handshake	0-RTT/1-RTT connection setup
Connection Migration	Survives IP changes
No Head-of-Line Blocking	Independent stream delivery

Architecture

+----------------------------------------------+
|           QuicNode (UDP Server)              |
|  - Listens on UDP socket                     |
|  - Routes datagrams to connections           |
|  - Manages ~62 neighbor connections          |
+--------+------------------------------------- +
         | manages
         v
+----------------------------------------------+
|      NodeConnection (Per-Peer)               |
|  - QUIC connection state                     |
|  - UP0 persistent stream (block announcements)|
|  - CE ephemeral streams (request-response)   |
+--------+-------------------------------------+
         | uses
         v
+----------------------------------------------+
|    Ed25519 Certificate Authentication        |
|  - Self-signed X.509 certificates            |
|  - SAN = base32(Ed25519 public key)          |
|  - Mutual TLS verification                   |
+----------------------------------------------+

Ed25519 Authentication

Certificate Structure

Validators use self-signed X.509 certificates containing Ed25519 public keys:

Generation Process:

1. Derive Ed25519 key: blake2b("jam_val_key_ed25519" + seed)
2. Generate SAN: "e" + base32(Ed25519_pubkey) → 53 chars like e4a2f3b5c7d8...
3. Create X.509 cert with 1-year validity

SAN Format: Unique DNS name derived from public key

Mutual Verification

On handshake, both peers verify:

• ✓ Certificate validity period
• ✓ Public key is Ed25519
• ✓ SAN matches generate_san(pubkey)
• ✓ Signature algorithm is Ed25519

Invalid certificates → connection rejected.

Connection Lifecycle

1. Connection Establishment

Initiator                          Responder
   |                                   |
   | ------  Initial Packet -------->  |
   |      (includes ClientHello)       |
   |                                   |
   |  -----  Handshake Packets -----   |
   |      (ServerHello + cert)         |
   |                                   |
   | ------  Handshake Packets ----->  |
   |      (cert + Finished)            |
   |                                   |
   |  -----  1-RTT Packets -------->   |
   |      (encrypted data)             |

Key Steps:

1. UDP datagram received -> routed to QuicNode
2. New connection ID? -> Create NodeConnection
3. TLS handshake completes -> verify peer certificate
4. Connection ready -> open UP0 persistent stream

2. Neighbor Management

Validators maintain connections to ~62 neighbors based on grid topology:

Grid Layout (√1023 × √1023 = 31 × 31):

     Col 0  Col 1  ...  Col 30
Row 0  [V0]  [V1]  ...  [V30]
Row 1  [V31] [V32] ...  [V61]
  ...
Row 30 [V930] ...  ...  [V992]

Neighbor Selection:

• Same row validators (30 peers)
• Same column validators (30 peers)
• Overlap at own position (-1)
• Plus: previous epoch key, pending key, staging key (up to +3)
• Total: ~62 neighbors

3. Stream Types

Stream	Type	Purpose	Lifetime
UP0	Unidirectional	Block announcements	Persistent
CE	Bidirectional	Request-response	Ephemeral

UP0 Stream:

• Opened immediately after handshake
• Validator -> neighbors direction only
• Broadcasts new block availability
• Never closes (unless connection drops)

CE Streams:

• Created per request
• Full duplex communication
• Closed after response received
• Multiple concurrent CE streams allowed

QuicNode - Central Server

Main networking component managing all connections:

Responsibilities:

• UDP socket handling
• Datagram routing by connection ID
• Neighbor list maintenance
• Connection lifecycle management

Datagram Routing:

UDP Packet arrives
    |
Extract connection ID from packet
    |
Lookup connection in table
    |
    +- Found? Forward to NodeConnection
    +- New? Create NodeConnection, add to table

NodeConnection - Per-Peer Handler

Each peer connection managed by a NodeConnection instance:

Responsibilities:

• TLS handshake & certificate verification
• Stream multiplexing (UP0 + CE streams)
• Protocol message routing
• Stream lifecycle management

Key State:

• Peer's SAN identifier
• Ed25519 public key
• UP0 stream ID (persistent)
• Stream buffers (per stream)
• Handshake status

Connection Initiation Logic

Initiator Selection: Higher Ed25519 public key initiates (prevents duplicate connections)

Decision Formula (XOR-based tie-breaking):
  (peer.ed25519[31] > 127) 
    XOR (our.ed25519[31] > 127)
    XOR (peer.ed25519 < our.ed25519)

If True -> We initiate
If False -> We wait for peer

Why: Ensures exactly one party initiates while being deterministic and symmetric.

Stream Management

UP0 - Persistent Stream

Purpose: Broadcast block announcements to all neighbors

Flow:

Validator produces block
    |
Encode announcement
    |
Send on UP0 stream to all neighbors
    |
Neighbors receive & process
    |
Stream remains open for next announcement

Properties:

• Stream ID: Always 0 (unidirectional)
• Direction: Initiator -> responder only
• Lifetime: Persistent (never closes)
• Data: Serialized block availability announcements

CE - Ephemeral Streams

Purpose: Request-response for data fetching

Types:

• CE0: Block fetch by hash
• CE1: Segment fetch by root
• CE2: Work package fetch
• CE3: Audit data fetch

Flow:

Node A needs data
    |
Open new CE stream
    |
Send request (prefix + parameters)
    |
Node B processes request
    |
Node B sends response
    |
Node B closes stream (FIN)
    |
Node A receives data & closes

Properties:

• Stream ID: Dynamically allocated
• Direction: Bidirectional
• Lifetime: Request-response duration
• Multiple concurrent CE streams allowed

Protocol Multiplexing

Stream Prefixes

First byte identifies protocol:

Prefix	Protocol	Type	Purpose
0	UP0	Unidirectional	Block announcements
128	CE128	Bidirectional	Block fetch
129	CE129	Bidirectional	State fetch
131	CE131	Bidirectional	Work-package
132-134	CE132-134	Bidirectional	Shards (WP/extrinsic/export)

Data Flow Patterns

UP0 (Push):

Producer                    Neighbor 1 ... Neighbor N
   |                            |              |
   |-- Block announcement ----->|------------->|
   |    (stream stays open)     |              |
   |                            |              |
   |-- Next announcement ------->|------------->|

CE (Request-Response):

Requester                   Provider
   |                            |
   |-- Open stream ------------>|
   |                            |
   |-- Request (prefix+data) -->|
   |    (send FIN)              |
   |                            |
   |                         Process
   |                            |
   |<----- Response ------------|
   |       (send FIN)           |
   |                            |
   |   Stream closed both ways  |

Message Handling

Outgoing Messages

UP0 (Persistent):

• Use stream ID 0
• Send without FIN flag
• Stream stays open for next message

CE (Ephemeral):

• Get new stream ID
• Send with FIN flag
• Wait for response
• Stream auto-closes

Incoming Messages

Processing Flow:

1. Receive StreamDataReceived event
2. Check if UP0 stream → process immediately
3. Otherwise buffer data until FIN received
4. Extract prefix byte → route to protocol handler
5. Process message → send response (if CE)
6. Clear buffer

Configuration

QUIC Parameters

Parameter	Value	Purpose
max_data	100 MB	Connection-level flow control
max_stream_data	10 MB	Per-stream flow control
max_datagram_size	1350 bytes	Safe MTU (avoids fragmentation)
idle_timeout	120s	Connection keepalive
verify_mode	CERT_NONE	Custom Ed25519 verification

ALPN Protocol

Format: jam/{version}/{genesis_hash}/{protocol_version}

Example: jam/0.1/476243ad/0

Ensures peers are on same network and protocol version.

Session Tickets (0-RTT)

Purpose: Fast reconnection without full handshake

Flow:

First Connection:
  Client --------------> Server
         1-RTT handshake
  Client <-------------- Server
         (session ticket)
  
  [Save ticket to disk]

Reconnection:
  Client --------------> Server
         0-RTT + early data
  Client <-------------- Server
         (immediate response)

Implementation:

• Tickets stored in sessions/{port}/session_tickets.pkl
• Persists across restarts
• Reduces latency from ~50ms to ~1ms

Active Peer Tracking

Active Peer: Neighbor with established UP0 stream

Tracking:

active_peers = {
    connection for neighbor in neighbors
    if has_connection(neighbor) and has_up0_stream(neighbor)
}

Used for:

• Block announcement broadcasting
• Network topology monitoring
• Connection health checks

Error Handling

Connection Failures

Error	Cause	Action
Certificate invalid	SAN mismatch, expired	Reject connection
ALPN mismatch	Different network	Reject connection
Handshake timeout	Network issues	Retry connection
Stream reset	Peer closed stream	Clean up buffer
Connection timeout	Idle > 120s	Reconnect if neighbor

Reconnection Logic

On Connection Termination:

1. Check if peer is in neighbor list
2. If yes: Schedule reconnect with exponential backoff
3. If no: Remove from connection pool

Keep-Alive: Periodic PING every 10 seconds to prevent timeout

Performance Optimizations

Key Optimizations:

• ✓ Connection ID pooling (multiple streams per connection)
• ✓ Stream buffering for efficient frame assembly
• ✓ Neighbor caching (computed once per epoch)
• ✓ 0-RTT session resumption
• ✓ ~62 parallel neighbor connections

Typical Metrics:

• New connection: ~50ms (1-RTT handshake)
• Resumed connection: ~1ms (0-RTT)
• Stream latency: ~5-20ms (depending on network)
• Concurrent streams: 100+ per connection

Security

Layer	Protection
Authentication	Ed25519 public key verification
Encryption	TLS 1.3 (mandatory)
Network Isolation	ALPN protocol matching
Certificate Validation	SAN = base32(pubkey) check
Connection Integrity	QUIC's built-in protection

References

• QUIC: RFC 9000
• aioquic: github.com/aiortc/aioquic
• Gray Paper: Appendix D - Networking
• Implementation: tessera/jam/network/

---

Next: Protocol Specification