Real-Time Analytics Dashboard
A production-grade real-time analytics platform with live metrics streaming every 2 seconds, threshold-based alerting, and full API key management
Overview
A production-grade real-time analytics platform built as a pnpm monorepo: Next.js 16 web client, NestJS 12 WebSocket backend, and a shared TypeScript contract package. Live metrics stream to the browser every 2 seconds with threshold-based alerting, searchable log streams, and full API key management — deployed on Vercel + Render with MongoDB Atlas and Upstash Redis.
Architecture
┌────────────────────┐ BFF /api/* ┌────────────────────┐
│ Next.js 16 (web) │ ── session ───▶ │ NestJS 12 API │
│ Vercel :3000 │ + x-api-key │ Render :4000 │
│ - React Query │ │ - REST controllers│
│ - Zustand store │ ◀─ socket.io ── │ - Socket Gateway │
│ - Virtualized UI │ org rooms │ - @Cron 2s sim │
└────────────────────┘ └─────────┬──────────┘
│ │
│ ┌─────────────┼─────────────┐
└──────────────────────────▶ MongoDB Atlas │ Upstash Redis
│ TTL 30d │ cache + rate limit
└───────────────┴─────────────┘
- apps/web — Next.js frontend with BFF API routes that authenticate the session and proxy to NestJS with a server-side BFF_API_KEY
- apps/server — NestJS backend: REST controllers, Socket.io gateway, cron simulation, throttler storage
- packages/shared — Zod schemas, env validation, and Auth.js cookie decryption (HKDF + JWE) so the server can trust sessions issued by the web app
- loadtest/ — k6 scenario scriptsKey Design Decisions
- pnpm 10 workspaces for monorepo management
- Shared package for Zod schemas, env validation, and Auth.js cookie decryption
- BFF pattern with Next.js API routes proxying to NestJS backend
- Socket.io with org-based rooms for real-time updates
- Upstash Redis for caching and rate limiting (HTTP-based, not ioredis)
- 30-day TTL indexes on metrics and logs for automatic cleanup
Features Implemented
Real-time Metrics Engine
- @Cron('*/2 * * * * *') simulation service generates jittered metric batches per organization every 2 seconds
- Bulk-inserts with insertMany and broadcasts each org's batch to its Socket.io room (org:{id})
- Client dedupes points by timestamp in a Zustand store (Map keyed by ts) and merges with React Query's initial 24h timeseries
Triple-mode WebSocket Auth
- httpOnly Auth.js session cookie (decrypted server-side)
- Short-lived (5-min) HS256 JWT minted by BFF /api/socket-token route
- Raw API key for machine-to-machine auth
- Org rooms enforce authorization on join:org — only members or admins can subscribe
Threshold Alerts with Cooldown
- When error rate exceeds 5%, AlertService.evaluate fires
- Only if no unacknowledged alert exists for that org in the last 5 minutes (prevents alert storms)
- Alerts stream over socket, surface as toast notifications with value-vs-threshold context
- Supports acknowledge workflow
Searchable Log Stream
- Cursor-based pagination (fetch limit + 1 to detect hasMore)
- Regex search across message/path, level filtering, sort toggle
- 300ms debounced input
- Virtualized table (@tanstack/react-virtual, 48px rows, overscan 20) over an infinite query
- Smooth with 10k seeded entries
API Key Lifecycle
- 64-char hex keys (32 random bytes), only SHA-256 hash persisted
- 8-char prefix enables fast lookup
- Create, revoke, regenerate, set-default, expiry dates, and lastUsedAt tracking
- ApiKeyGuard accepts x-api-key header or Bearer token
Server-side Time Bucketing
- Timeseries endpoint aggregates raw 2s points into 1m / 5m / 1h buckets for 1h / 24h / 7d ranges
- Charts stay fast as data grows
- Metric and log collections carry 30-day TTL indexes
Caching & Rate Limiting
- Metrics summary cached in Upstash Redis with 5s setex (plus in-memory fallback in BFF)
- Rate limiting uses custom ThrottlerStorage on Upstash sorted sets (atomic zadd + zremrangebyscore + zcard in multi())
- In-memory fallback when Redis is unavailable
Challenges & Solutions
httpOnly cookies don't cross origins with Socket.io
The web app (Vercel) and API (Render) are separate origins, so the browser can't attach the session cookie to the WS handshake. Solution: BFF issues a 5-minute JWT (/api/socket-token) that the client passes as a query param. The guard tries cookie first, then JWT, then API key — one guard, three transport strategies.
Socket.io's built-in reconnection breaks with expiring tokens
Disabled native reconnection and implemented manual exponential backoff (1s doubling to 30s cap, 10 attempts) that re-mints the token through the BFF on every attempt (apps/web/src/hooks/useSocket.ts).
Upstash Redis is HTTP-based, not ioredis
@nestjs/throttler ships an ioredis storage that can't talk to Upstash's REST API. Implemented the ThrottlerStorage interface directly against @upstash/redis using sorted sets as a sliding window — atomic via multi(), with graceful in-memory fallback.
Cross-app session trust without per-request auth calls
packages/shared replicates Auth.js key derivation (HKDF-SHA256 with the cookie salt, both authjs.session-token and __Secure- variants) and decrypts the JWE with the shared AUTH_SECRET using jose. Both apps verify the same cookie independently.
MongoDB Atlas DNS resolution failures on Windows
Connection attempts intermittently failed during development; fixed by pinning resolvers (dns.setServers(['8.8.8.8', '1.1.1.1'])) in both the server bootstrap and the seed script.
Mongoose model duplication in dev hot-reload
nest start --watch re-registered models and threw OverwriteModelError; guarded with mongoose.models.X ?? model('X', schema). ESM required .js extensions on all relative imports.
CI flakiness
Lint and test steps removed from GitHub Actions workflow after false failures, leaving format:check, typecheck, and build as the CI gate — a deliberate trade-off to keep main green while the test suite matured.
Duplicate/out-of-order socket batches
Rapid emissions could deliver overlapping points; the store dedupes by timestamp and re-sorts ascending before render, keeping charts correct without server-side coordination.
Performance & Reliability
- k6 load test: 250 concurrent WebSocket VUs (30s ramp, 2m hold) plus 50 RPS REST for 2 minutes
- Pass thresholds: p95 latency < 500ms, HTTP failure rate < 1%, WS connect success ≥ 99%
- Unit tests (Vitest): API key service, alert service, gateway, throttler storage, and simulation specs
- Degradation paths: in-memory cache when Redis is unconfigured, in-memory throttler fallback, REST polling (5s refetch) as baseline
Future Enhancements
- OAuth providers (credentials-only auth currently)
- Multi-org self-signup
- Horizontal WebSocket scaling (sticky sessions + Socket.io Redis adapter)
- Docker Compose for local development
Deployment
Frontend
Vercel (apps/web)
Backend
Render / Fly.io / Railway (apps/server)
Database
MongoDB Atlas M0 + Upstash Redis
CI/CD
GitHub Actions