System Design: Problems & Solutions Matrix

At single-server stage, focus on feature velocity. Monitor uptime and treat failures as learning.

Contingency: Set up automated backups to external storage.

Add CPU, RAM, and disk storage to the single server.

Cost-effective short-term solution (~weeks to months).

Buys time before reaching hardware limits (max ~256 GB RAM, 64 vCPUs on most commodity hardware).

Set up daily/hourly backups to external storage (S3, backup service).

Document recovery procedure for manual failover.

Recovery time: hours to days (RTO) — not acceptable for production.

Move database server to separate machine (10.0.0.100).

Web server handles requests on 10.0.0.1, database serves data independently.

Can scale each tier independently: add CPU to web without touching DB, and vice versa.

Web tier: Add load balancer + multiple web servers (Stage 3).

DB tier: Add database replication — master/slave (Stage 4).

Data is structured (users, posts, comments with relationships).

Need ACID transactions (financial data, orders).

Need JOIN operations across tables.

Data model is known upfront.

Examples: MySQL, PostgreSQL, Oracle.

Data is unstructured or semi-structured (logs, events, documents).

Need super-low latency (<10 ms response).

Horizontal scaling is critical (millions of records/sec).

Data model evolves rapidly (schema flexibility).

Examples: MongoDB, DynamoDB, Cassandra, Redis.

Add more CPU, RAM, disk to single server.

Cheap, simple, works for moderate growth.

BUT: hardware limit reached (AWS max ~768 GB RAM, 96 vCPUs); cost becomes exponential; still SPOF.

Add multiple servers (10.0.0.1, 10.0.0.2, 10.0.0.3…).

Unlimited scalability: can keep adding servers as needed.

Provides redundancy: if one server fails, others still serve traffic.

Requires load balancer to distribute requests.

Single public IP that users connect to.

Load balancer routes requests to private IPs using round-robin, least-connections, or other algorithms.

Users never see private IPs; load balancer handles routing transparently.

Round-robin: Server 1, then 2, then 3, then repeat.

Least-connections: route to server with fewest active connections.

IP hash: same client always routes to same server (if sticky sessions needed).

Balancer pings each server every 10 seconds: "Are you alive?"

If Server 1 doesn't respond, balancer marks it DOWN and removes it from rotation.

Requests now route only to Servers 2 & 3; users don't see the outage.

Ops team can add a replacement Server 4; balancer detects it and adds to rotation.

Deploy 2+ load balancers in active-passive or active-active config.

DNS points to primary; if primary fails, DNS fails over to secondary.

Or: use managed load balancer service (AWS ALB, GCP LB) — provider handles redundancy.

Master DB (10.0.0.100): handles ALL writes, updates, deletes.

Slave DB(s) (10.0.0.101, 10.0.0.102): receive copies of data from master in real-time.

Web servers send reads to any slave, writes to master only.

If master fails: promote one slave to become new master.

Replication lag: typically <100 ms; eventual consistency.

Tools: MGR (MySQL Group Replication), Patroni (PostgreSQL), etc.

Automatically detect master failure, promote best slave, update all clients.

Failover time: <30 seconds; minimises data loss.

For most apps, eventual consistency is fine: <100 ms lag is imperceptible.

Critical reads (financial, current user profile): route to MASTER only.

Less critical reads (user feed, analytics): read from SLAVE (stale is OK).

Before hitting DB, web server checks cache: "Is this data in memory?"

Cache hit (data found): return in <1 ms.

Cache miss (data not found): query DB, store result in cache with TTL, return.

Typical hit rate: 90%+ → 90% of requests bypass the database entirely.

Result: DB load drops 10×, latency drops 100×.

Cache all data with expiration: cache.set("user:123", {...}, ttl=60)

After 60 s, cache entry expires automatically; next read hits DB for fresh data.

Invalidation: proactively delete from cache on write: cache.delete("user:123")

Deploy cache as cluster (2–3 replicas) instead of single instance.

If one node fails, cluster routes to others.

Overprovision capacity: cache 10 GB data, allocate 15 GB (50% headroom).

LRU (Least-Recently-Used): delete oldest-accessed entry to make room.

LFU (Least-Frequently-Used): delete least-popular entry.

FIFO (First-In-First-Out): delete oldest entry.

Most common: LRU (hottest data stays, cold data gets evicted).

When to use cache. Consider using cache when data is read frequently but modified infrequently. Since cached data is stored in volatile memory, a cache server is not ideal for persisting data. If a cache server restarts, all data in memory is lost. Important data should always be saved in persistent data stores.

Expiration policy. It is good practice to implement an expiration policy. Once cached data expires, it is removed from the cache. Without an expiration policy, cached data remains in memory permanently. The expiration date should not be too short (causes frequent database reloads) or too long (data can become stale).

Consistency. Keeping the data store and the cache in sync is critical. Inconsistency can occur because data-modifying operations on the data store and cache are not in a single transaction. When scaling across multiple regions, maintaining consistency is especially challenging. For further details, refer to "Scaling Memcache at Facebook" published by Facebook.

Mitigating failures. A single cache server represents a potential single point of failure (SPOF) — a part of a system that, if it fails, will stop the entire system from working. Multiple cache servers across different data centres are recommended to avoid SPOF. Another approach is to overprovision the required memory by a certain percentage, providing a buffer as memory usage increases.

Eviction policy. Once the cache is full, any request to add items may cause existing items to be removed (cache eviction). Least-Recently-Used (LRU) is the most popular policy. Other policies such as Least Frequently Used (LFU) or First In First Out (FIFO) can be adopted to satisfy different use cases.

CDN is a global network of servers physically located near users.

Static assets cached on CDN edges near users.

User in London → served from CDN UK edge (30 ms) instead of US origin (120 ms).

Set TTL on CDN: cache static assets for 1 hour, then re-fetch from origin.

Use versioning in URL: logo.jpg?v=2 → new cache key, CDN fetches fresh.

Purge CDN manually via API when deploying new assets.

HTML includes fallback: if CDN request fails, browser requests asset from origin.

Ensures graceful degradation: worse latency, but site stays functional.

Store session data in Redis/Memcached or database, NOT in server memory.

Load balancer can route User A to ANY server; all servers fetch session from Redis.

Servers become stateless: can be killed/restarted without losing user sessions.

Auto-scaling now works: spin up 10 servers, kill 5, users don't notice.

With sessions in Redis, can spin up/down servers freely without user impact.

Monitor CPU/memory; if >80%, add servers; if <20%, remove servers.

Tools: Kubernetes auto-scaler, AWS auto-scaling groups.

Deploy identical infrastructure in US-East (DC1) and US-West (DC2).

Each DC has: web servers, cache, master DB, workers, etc.

If US-East fails, GeoDNS routes 100% traffic to DC2. RTO: <30 s.

GeoDNS resolves domain based on user location: US-East → DC1, US-West → DC2.

Route to nearest DC → lower latency, better user experience.

Master DB in each DC handles local writes; replicates to other DCs asynchronously.

Eventual consistency: within seconds/minutes, all DCs converge to same state.

For critical data (financial): stick to single DC for writes.

Terraform/Bicep: define entire DC infrastructure once, deploy to both regions with one command.

Centralised monitoring: one dashboard showing both DCs' health.

Web server publishes task to queue: queue.publish("photo_process", {photo_id: 123})

Returns immediately: "Your photo is processing" (200 ms response).

Worker picks up task from queue asynchronously and processes offline.

Web servers only handle requests and publish to queue.

Workers independently consume from queue and process photos.

If queue backlog grows, spin up more workers. If queue empties, kill workers.

Web and worker tiers scale independently.

Queue persists all messages to disk before worker processes them.

If crash before ACK, queue re-delivers to another worker.

Guarantees: "at least once" or "exactly once" (more complex, slower).

Deploy queue as 3-node cluster (RabbitMQ cluster, Kafka brokers).

If one node fails, cluster rebalances; publishers/consumers reconnect to other nodes.

Add massive CPU/RAM to single server: 512 GB RAM, 96 vCPUs.

Works for medium scale (~100 M records).

BUT: costs exponential; hits hardware limits; still SPOF.

Partition data across multiple databases (Shard 0, 1, 2, 3).

Sharding key: user_id. Hash function: user_id % 4.

Each shard holds 1/4 of total data; queries 4× faster than monolith.

Can add more shards as data grows; unlimited horizontal scale.

Avoid cross-shard JOINs: denormalise data (duplicate in each shard).

Resharding: split Shard 0 into 0a, 0b; move half the data.

Hotspot keys: split that shard further, allocate dedicated replicas.

Shard key must be: (1) immutable, (2) even distribution, (3) queries mostly within shard.

Good: user_id (every user unique, query by user common).

Bad: timestamp (new data all goes to latest shard, old shards empty).

Every component logs to central location.

Search all logs by user_id: see every request, DB query, worker task.

Alerts: if error rate > 5%, page on-call engineer.

Collect metrics: CPU, memory, disk, request latency, error rate, queue size, DB query time.

Dashboard shows: latency, queue backlog, cache hit rate — spot problems before users notice.

Tag every resource by service: "photo_service", "payment_service", "worker_tier".

Track: cost per service, cost per user, cost per transaction.

Alerts: if service cost > budget, shut down unused resources.

Tools: LangSmith for AI workload tracking; Kubecost for Kubernetes; CloudHealth for AWS.

Continuous Integration: every commit runs tests automatically.

Continuous Deployment: tests pass → deploy to staging → approve → auto-deploy to production.

Result: deploy in 2 minutes with 0 human steps; rollback if needed.