Warpnet CRDT-Based Consensus Implementation

This implementation adds Conflict-free Replicated Data Type (CRDT) based consensus to Warpnet for managing distributed tweet statistics (likes, retweets, replies, and views) across autonomous nodes without centralized coordination.

Overview

1. Core CRDT Statistics Store

• Complete Merkle CRDT implementation using "github.com/ipfs/go-ds-crdt"

• Four statistic types: Likes, Retweets, Replies, Views

• Automatic state synchronization via libp2p PubSub

• Merkle-DAG based IPLD storage for efficient data replication

• Graceful error handling and fallback mechanisms

2. Database Integration Layer

• Maintains backward compatibility with existing code

• Updates both local storage and CRDT counters

• Falls back to local counts if CRDT unavailable

• Provides transparent upgrade path

3. Testing

• TODO

Each statistic uses a Merkle CRDT with the following structure:

Key Pattern: /CRDT/incr/{key}/{nodeID} Value: 64-bit unsigned integer

Key Design Decisions

1. Merkle CRDT Choice

• Why: Could behave as G-Counter. Statistics are monotonically increasing (likes grow over time)

• Benefits: Simple aggregation, deterministic, commutative

2. Per-Node Sets

• Why: Each node maintains independent counter

• Benefits: True offline operation, no coordination needed

• Tradeoff: O(n) space where n = active nodes

3. Hybrid Approach

• CRDT: Global aggregated counts

• Local DB: User-specific actions (who liked what)

• Benefits: Best of both worlds - distributed consistency + local tracking

4. Graceful Degradation

• Fallback: Always fall back to local counts if CRDT fails

• Benefits: Resilience, backward compatibility

• Tradeoff: Temporary inconsistency during failures

Benefits Achieved

1. True Offline-First Operation

• Nodes can like, retweet, reply while disconnected

• Changes sync automatically when reconnected

• No data loss during network partitions

2. Automatic Conflict Resolution

• No manual intervention needed

• Deterministic outcomes

• Commutative operations

3. Scalability

• Linear scaling with nodes

• No coordination bottleneck

• Efficient delta-only sync

4. Fault Tolerance

• No single point of failure

• Nodes can join/leave freely

• Resilient to network issues

5. Consistency Without Coordination

• Eventually consistent

• Deterministic convergence

• Strong mathematical guarantees

Performance Characteristics

Space Complexity

• Per stat: O(n) where n = number of nodes that updated it

• Typical: Small constant factor (10-100 nodes per popular tweet)

• Worst case: Bounded by active node count

Time Complexity

• Update: O(1) - single key write + PubSub broadcast

• Query: O(n) - iterate and sum node counters

• Sync: O(log n) - Merkle-DAG sync

Network Usage

• Bandwidth: Minimal - only deltas transmitted

• Latency: Depends on PubSub propagation (typically < 1s)

• Overhead: ~64 bytes per stat update

Future Enhancements

Near-Term

1. Compaction: Merge old node entries to reduce storage

2. Caching: LRU cache for frequently accessed stats

3. Metrics: Prometheus instrumentation

Long-Term

1. OR-Set: Track who liked/retweeted (not just count)

2. LWW-Register: Last-write-wins for timestamps

3. Selective Sync: Only sync stats for viewed tweets

4. Sharding: Partition stats across CRDT instances

Security Considerations

• DoS Protection: Counters are per-node, limiting attack surface

• Data Integrity: Updates propagated via signed PubSub messages

• Sybil Resistance: Node IDs tied to libp2p peer IDs

• Spam Prevention: Local validation before CRDT update

• Rate Limiting: Should be added in production (future work)