Scalability and Performance Principle

Overview

Principle: Systems must handle growth in load, data, and users without proportional increases in cost or complexity.

Scalability enables sustainable growth. Performance ensures user experience remains good as scale increases.

Rationale

• Success means growth
• Doubling users shouldn’t require doubling cost
• Performance degradation harms user experience
• Scalability must be designed in, not added later
• Different architectures have different limits

Core Implications

1. Scalability Patterns

Horizontal Scaling:

Load Balancer
├─ Server 1 (handles 1000 req/s)
├─ Server 2 (handles 1000 req/s)
└─ Server 3 (handles 1000 req/s)
Total: 3000 req/s

Add load → Add more servers
Advantage: Unlimited scaling potential
Challenge: Stateless design required

Vertical Scaling:

Single Server
├─ 1 core → 4 cores (4x performance)
├─ 8 GB RAM → 64 GB RAM (8x performance)
└─ 1 disk → 4 disks (4x I/O)
Total: 128x performance increase

Add load → Bigger server
Advantage: Simple, stateful OK
Challenge: Finite limits exist

Caching as Scalability:

Direct Access (1000 requests/second):
Request → Database (1000 DB queries/second)
Result: Database becomes bottleneck

With Cache (1000 requests/second):
90% cache hits → 100 DB queries/second
Result: 10x scalability improvement without infra

Caching Strategy:
- Hot data (recently accessed): Cache in memory
- Warm data: Cache in Redis
- Cold data: Database only

2. Performance Metrics

Latency Tiers:

< 100ms: Excellent (feels instant)
100-300ms: Good (acceptable)
300-1000ms: Fair (noticeable delay)
1-3 seconds: Poor (user frustrated)
> 3 seconds: Unacceptable (user abandons)

Example Timeline:
User Click
├─ Network latency: 20ms
├─ Server processing: 50ms
├─ Database query: 80ms
├─ Format response: 10ms
└─ Browser render: 40ms
Total: 200ms (Good)

If database query becomes 300ms:
Total: 500ms (Fair → Poor)
User experience degrades significantly

Scalability Metrics:

1 User: 100ms response time
10 Users: 110ms response time (slight increase)
100 Users: 150ms response time (acceptable)
1000 Users: 500ms response time (concerning)
10000 Users: 5000ms response time (failure)

Good Architecture:
Response time increases logarithmically (slight with load)

Bad Architecture:
Response time increases exponentially (severe with load)

3. Bottleneck Analysis

Identifying Bottlenecks:

System Performance Profile:
├─ API Response Time: 50%
├─ Database Query Time: 30%
└─ Cache Miss: 20%

Optimization Effort:
├─ Optimize API: -25% (50% → 25%)
│   Total improvement: 6.25%
│
├─ Optimize Database: -15% (30% → 15%)
│   Total improvement: 4.5%
│
└─ Reduce Cache Misses: -18% (20% → 2%)
   Total improvement: 3.6%

Result: Focus on API first (biggest impact)

Common Bottlenecks:

Tier 1 (Most Common):
- Database queries (slow, blocking)
- Unindexed columns
- N+1 query problem

Tier 2:
- External API calls
- Network round-trips
- Synchronous processing

Tier 3:
- Memory/CPU constraints
- Disk I/O
- Garbage collection

4. Scalability Design Patterns

Stateless Design:

❌ Stateful (Can't scale horizontally):
Server holds user session data
├─ User1 → Server A (session in memory)
└─ User1 next request → Server B (session not there)
Result: User session lost, can't add more servers

✅ Stateless (Can scale):
Server retrieves session from shared store
├─ User1 → Server A (reads session from Redis)
└─ User1 next request → Server B (reads session from Redis)
Result: Any server can handle any request, add servers freely

Database Partitioning:

Single Database (bottleneck):
├─ All users data in single database
├─ Queries must scan all data
└─ 10K users → 1M queries/day becomes slow

Partitioned Database:
├─ User 1-1000: Database A
├─ User 1001-2000: Database B
├─ User 2001-3000: Database C
└─ Each database handles 1/3 queries
Result: 3x scalability improvement

Asynchronous Processing:

Synchronous (Blocks):
User Request
├─ Process payment (5s)
├─ Send confirmation email (3s)
├─ Update analytics (2s)
└─ Total: 10s (user waits)

Asynchronous (Non-blocking):
User Request
├─ Queue payment (10ms)
├─ Return immediately
└─ Background processes handle rest
   ├─ Process payment (5s)
   ├─ Send confirmation email (3s)
   └─ Update analytics (2s)

Result: User sees response in 10ms instead of 10s

Implementation Practices

1. Load Testing

Load Test Profile:

Ramp-up: Gradually increase users
├─ 0 min: 0 users
├─ 5 min: 100 users
├─ 10 min: 500 users
├─ 15 min: 1000 users
└─ Hold at 1000 for 10 minutes

Monitor:
├─ Response time
├─ Error rate
├─ Server CPU/Memory
└─ Database connections

Identify Breaking Point:
└─ At what load does system degrade?
   (If at 500 users, that's max capacity)

Load Testing Tools:

Apache JMeter: HTTP load testing
Locust: Python-based, easy scripting
k6: Modern, cloud-native load testing
LoadRunner: Enterprise load testing
Gatling: Scala-based, high performance

Example Load Test:

# Locust example
from locust import HttpUser, task, between

class WebsiteUser(HttpUser):
    wait_time = between(1, 5)
    
    @task
    def get_products(self):
        self.client.get("/api/products")
    
    @task
    def add_to_cart(self):
        self.client.post("/api/cart", json={
            "product_id": "123",
            "quantity": 1
        })
    
    @task
    def checkout(self):
        self.client.post("/api/orders")

# Run: locust -f locustfile.py --host=http://localhost:8000

2. Caching Strategies

Cache Tiers:

L1 Cache (CPU):
- In-memory, built-in
- < 1ms access time
- Example: Function-level caching

L2 Cache (Local):
- In-process memory
- < 10ms access time
- Example: Memoization, LRU cache

L3 Cache (Distributed):
- Redis, Memcached
- 1-50ms access time
- Example: Session cache, DB query cache

L4 Cache (Edge/CDN):
- Content delivery network
- 10-200ms (but geographically distributed)
- Example: Static assets, API responses

Cache Invalidation:

❌ Problem: Stale data
Cache has old data, user sees outdated info

Strategies:
1. Time-based (TTL):
   - Cache for 5 minutes
   - After 5 minutes, refresh
   - Simple but may serve stale data

2. Event-based:
   - User edits profile → invalidate cache
   - Precise but requires coupling

3. Lazy loading:
   - Check if data changed
   - If yes, refresh cache
   - Balanced approach

4. Cache aside pattern:
   - Check cache first
   - If miss, fetch from source
   - Update cache
   - Serve result

Cache Warm-up:

Problem: Cold cache is slow
- Service starts
- First requests hit database
- Response times slow until cache warms

Solution: Pre-load cache
- Start service
- Load popular/critical data into cache
- First requests hit warm cache
- Result: Consistent fast response times

3. Database Optimization

Query Optimization:

Slow Query:
SELECT * FROM users 
WHERE created_at > '2025-01-01' 
AND country = 'US'

Problems:
- No index on created_at
- No index on country
- SELECT * includes unnecessary columns

Optimized Query:
CREATE INDEX idx_users_date_country 
ON users(created_at, country)

SELECT id, name, email 
FROM users 
WHERE created_at > '2025-01-01' 
AND country = 'US'

Result: 100x faster

Connection Pooling:

❌ Without pooling:
Each request → Create new database connection → Use → Close
Overhead: 100ms per request

✅ With pooling:
├─ Pool maintains 20 connections
├─ Request grabs available connection
├─ Uses connection
├─ Returns to pool
└─ Overhead: 1ms per request

Configuration:
├─ Min connections: 10
├─ Max connections: 100
├─ Max idle time: 5 minutes

4. Performance Monitoring

APM (Application Performance Monitoring):

APM Tracks:
├─ Response times (per endpoint)
├─ Error rates
├─ Slow transactions
├─ Database queries
└─ External service calls

Example Alert:
"Checkout endpoint avg response time > 2s"
└─ Investigation: Database connection pool exhausted
└─ Action: Increase pool size

Common Scenarios

Scenario 1: Database Becomes Bottleneck

Situation: Response times slow down as queries increase.

Investigation:

1. Monitor database metrics
2. Identify slow queries
3. Check query execution plans
4. Find missing indexes

Solution:

1. Add appropriate indexes
2. Optimize query logic (reduce columns, joins)
3. Implement caching
4. Consider database partitioning

Result: 10x performance improvement

Scenario 2: Server Runs Out of Memory

Situation: Service crashes periodically under load.

Investigation:

1. Monitor memory metrics
2. Identify memory leak (gradual increase)
3. Profile application
4. Find object not being released

Solution:

1. Fix memory leak
2. Implement caching limits (eviction policy)
3. Increase server resources temporarily
4. Test with load

Result: Stable operation under load

Scenario 3: Peak Traffic Event

Situation: Black Friday sales cause traffic spike.

Preparation:

1. Load test with peak projections
2. Set up auto-scaling
3. Pre-warm caches
4. Brief on-call team

During Event:

1. Monitor metrics closely
2. Scale automatically
3. Watch for unexpected behaviors
4. Have rollback plan

After Event:

1. Analyze metrics
2. Identify bottlenecks
3. Plan optimizations
4. Update load test profiles

Risks of Ignoring This Principle

❌ System Degradation: Response times get worse with success ❌ Cascading Failures: One bottleneck causes others to fail ❌ User Abandonment: Slow sites lose users ❌ Expensive Growth: Doubling users requires 4x infrastructure ❌ Fire-fighting: Constantly dealing with performance crises

Best Practices

✅ Design for scale from start Horizontal scaling from day one.

✅ Monitor continuously Real-time visibility into performance.

✅ Load test regularly Know capacity limits.

✅ Cache strategically Cache hot data, not everything.

✅ Optimize databases Indexes, queries, partitioning.

✅ Make systems stateless Enable horizontal scaling.

✅ Use asynchronous processing Don’t block on slow operations.

✅ Profile before optimizing Fix actual bottlenecks, not assumptions.

✅ Plan for failure Assume parts will fail, design accordingly.

Related Principles

• Cloud-First, Cloud-Native - Cloud enables elastic scaling
• Observability Built-In - Monitoring enables performance optimization
• Cost Optimization - Scalability enables cost efficiency

References

• Designing Data-Intensive Applications (Martin Kleppmann)
• The Art of Capacity Planning (Arun Kejariwal)
• Performance Testing Handbook (Egor Bogatov)

---

Last Updated: November 2025 Principle Category: Architecture Applies To: All systems expecting growth Related Documents: Performance Guide, Capacity Planning