Introduction
DataStream is a real-time CDC (Change Data Capture) pipeline that delivers database changes to clients using multiple protocols: REST API, WebSocket, SSE (Server-Sent Events), WebTransport, and NATS WebSocket (direct broker connection).
Built with Hexagonal Architecture in Go, it provides sub-100ms latency from database INSERT to client notification.
Why Data Stream?
Traditional approaches to real-time data synchronization often involve:
- Polling databases (high latency, wasteful)
- Custom pub/sub implementations (complex to maintain)
- Vendor lock-in solutions (expensive, inflexible)
Data Stream solves these problems by:
- CDC-based: Captures database changes at the source (PostgreSQL WAL)
- Protocol agnostic: Supports REST API, WebSocket, SSE, and WebTransport
- Decoupled: Read model (Redis) separated from write model (PostgreSQL)
- Observable: Built-in metrics, logging, and tracing
- Production-ready: Deployed on Fly.io with automatic scaling
Architecture Overview
Key Features
Multi-Protocol Support
| Protocol | Latency | Direction | Best For |
|---|---|---|---|
| WebTransport | 0.5-2ms | Bidirectional | Ultra-low latency, QUIC/HTTP3 |
| NATS WebSocket | 0.5-2ms | Bidirectional | Direct broker, wildcards |
| WebSocket | 1-5ms | Bidirectional | Interactive apps, gaming |
| SSE | 10-50ms | Server → Client | Dashboards, feeds |
| REST API | 50-200ms | Request/Response | History, latest data, polling |
Hexagonal Architecture
Clean separation of concerns:
- Domain: Pure business logic, no external dependencies
- Ports: Interface definitions (contracts)
- Adapters: External integrations (Redis, NATS, HTTP)
CDC Pipeline
Real-time data capture without polling:
- PostgreSQL writes to WAL (Write-Ahead Log)
- Debezium captures changes and publishes to NATS
- Go consumer processes and publishes to Redis
- Backend streams to connected clients
Technology Stack
| Component | Technology | Purpose |
|---|---|---|
| Database | PostgreSQL 16 | Source of truth |
| CDC | Debezium 2.5 | Change capture |
| Message Broker | NATS JetStream | Event streaming |
| Cache | Redis 7 | Read model + Pub/Sub |
| Backend | Go + Fiber | API + Streaming |
| Deployment | Fly.io | Global edge deployment |
Quick Links
- Quickstart Guide - Get running in 5 minutes
- Architecture - Deep dive into the design
- WebSocket Protocol - Real-time bidirectional
- NATS WebSocket - Direct broker connection
- SSE Protocol - Simple server push
- WebTransport Protocol - Next-gen streaming
- API Reference - REST endpoints
- Client Examples - Interactive protocol clients
Security Notice
Authentication, authorization, and encryption layers are currently being implemented. The following security features are planned and in active development:
- Authentication: JWT tokens, API keys, OAuth2
- Authorization: Role-based access control (RBAC)
- Encryption: TLS 1.3 for all connections, encrypted payloads
- Rate Limiting: Per-client and per-IP limits (already implemented)
- Input Validation: Request sanitization and validation
This is a demo/development environment. Production deployments will include full security hardening.
Live Demo
All clients are running live at https://datastream.hypetech.games
Interactive Clients
| Client | Description | Try Online |
|---|---|---|
| Dashboard | Overview of all protocols | Open Dashboard |
| WebSocket | Real-time bidirectional | Open Client |
| SSE | Server-Sent Events | Open Client |
| NATS | Direct broker connection | Open Client |
| WebTransport | QUIC/HTTP3 streaming | Open Client |
| REST API | HTTP polling | Open Client |
| Comparison | Side-by-side protocols | Open Client |
API Endpoints
- Games List: /api/games
- Latest Result: /api/latest/crash
- History: /api/history/crash
Streaming Endpoints
- WebSocket:
wss://datastream.hypetech.games/ws/crash - SSE:
https://datastream.hypetech.games/sse/crash - WebTransport:
https://datastream.hypetech.games:4433/wt - NATS WebSocket:
ws://localhost:8443(local only)