High-Level Design: URL Shortener System

1. Overview

This document describes the High-Level Design (HLD) for a scalable URL shortener system similar to Bitly or TinyURL.

The system allows users to submit a long URL and receive a short URL. When someone visits the short URL, they are redirected to the original long URL.

2. Goals

Functional Requirements

• Create a short URL from a long URL.
• Redirect users from a short URL to the original long URL.
• Support custom aliases, for example: sho.rt/kush.
• Track basic analytics:
  • Total clicks
  • Timestamp
  • Referrer
  • User agent
  • Approximate location
• Allow URL expiration.
• Prevent malicious or spam URLs.

Non-Functional Requirements

• High availability for redirects.
• Low latency redirects, ideally under 50 ms.
• Scalability to handle millions or billions of URLs.
• Durability so generated links are not lost.
• Security against abuse, phishing, and brute-force attacks.

3. Out of Scope

• Full marketing dashboard.
• Advanced user segmentation.
• Deep fraud detection using machine learning.
• Payment or subscription management.

4. System Architecture

5. Major Components

5.1 Client

The client can be:

• Web app
• Mobile app
• Browser
• Third-party API consumer

Users can create and access short URLs through the client.

5.2 CDN / Edge Cache

The CDN improves redirect performance by caching frequently accessed short URL mappings.

Example:

sho.rt/abc123 -> https://example.com/some/long/path

Benefits:

• Reduces load on backend services.
• Improves global latency.
• Helps absorb traffic spikes.

5.3 API Gateway

The API Gateway handles:

• Request routing
• Rate limiting
• Authentication checks
• Request validation
• TLS termination
• Logging

Example routes:

5.4 Auth Service

The Auth Service manages:

• User login
• API keys
• Access tokens
• User permissions

Anonymous users may be allowed to create links with stricter rate limits.

5.5 URL Shortener Service

This service handles short URL creation.

Responsibilities:

• Validate long URLs.
• Check if the URL is safe.
• Generate or validate short code.
• Store mapping in database.
• Cache mapping in Redis.
• Handle expiration date.
• Handle custom alias availability.

5.6 Redirect Service

This is the most performance-critical service.

Responsibilities:

• Accept short code.
• Look up long URL from cache.
• Fall back to database if cache misses.
• Return HTTP redirect.
• Emit analytics event asynchronously.

Redirect response:

HTTP/1.1 302 Found
Location: https://example.com/original-url

For permanent links, 301 Moved Permanently can also be used, but 302 is safer if analytics and future edits are needed.

5.7 ID Generator Service

The ID Generator Service generates globally unique IDs that can be encoded into short strings.

Possible strategies:

Recommended approach:

Use Snowflake-style distributed IDs and encode them using Base62.

Example:

Numeric ID: 1257894001
Base62: bK9xQ2
Short URL: https://sho.rt/bK9xQ2

6. Database Design

6.1 URL Mapping Table

Indexes:

CREATE UNIQUE INDEX idx_short_code ON url_mapping(short_code);
CREATE INDEX idx_user_id ON url_mapping(user_id);
CREATE INDEX idx_expires_at ON url_mapping(expires_at);

6.2 Analytics Events Table

For large scale, analytics should go to a separate analytics store.

Possible databases:

• ClickHouse
• BigQuery
• Apache Druid
• Elasticsearch
• Cassandra

7. API Design

7.1 Create Short URL

POST /api/v1/urls
Content-Type: application/json
Authorization: Bearer <token>

Request:

{
  "longUrl": "https://example.com/products/summer-sale?campaign=abc",
  "customAlias": "summer-sale",
  "expiresAt": "2026-12-31T23:59:59Z"
}

Response:

{
  "id": "987654321",
  "shortUrl": "https://sho.rt/summer-sale",
  "shortCode": "summer-sale",
  "longUrl": "https://example.com/products/summer-sale?campaign=abc",
  "expiresAt": "2026-12-31T23:59:59Z",
  "createdAt": "2026-05-12T10:30:00Z"
}

7.2 Redirect Short URL

GET /summer-sale

Response:

HTTP/1.1 302 Found
Location: https://example.com/products/summer-sale?campaign=abc

7.3 Get Analytics

GET /api/v1/urls/987654321/analytics
Authorization: Bearer <token>

Response:

{
  "shortCode": "summer-sale",
  "totalClicks": 15420,
  "uniqueClicks": 8211,
  "topCountries": [
    {
      "country": "India",
      "clicks": 7800
    },
    {
      "country": "United States",
      "clicks": 4300
    }
  ],
  "topReferrers": [
    {
      "referrer": "twitter.com",
      "clicks": 5200
    },
    {
      "referrer": "linkedin.com",
      "clicks": 3100
    }
  ]
}

8. URL Creation Flow

9. Redirect Flow

10. Caching Strategy

Cache Key

short_code:{shortCode}

Example:

short_code:abc123

Cache Value

{
  "longUrl": "https://example.com/article",
  "status": "active",
  "expiresAt": "2026-12-31T23:59:59Z"
}

Cache TTL

Cache Invalidation

Cache should be invalidated when:

• URL is deleted.
• URL expires.
• Destination URL is edited.
• URL is marked malicious.

11. Scalability Considerations

11.1 Read-Heavy System

URL shorteners are usually read-heavy.

Typical ratio:

Redirects : Creates = 100:1 or higher

Therefore, optimize heavily for redirects.

11.2 Database Sharding

Shard URL mappings by short_code hash or numeric ID.

Example:

shard = hash(short_code) % number_of_shards

Benefits:

• Distributes load.
• Avoids one huge database.
• Allows horizontal scaling.

11.3 Replication

Use:

• Primary database for writes.
• Read replicas for lookup traffic.
• Multi-region replicas for global redirects.

11.4 Async Analytics

Redirect latency should not depend on analytics processing.

Instead:

1. Redirect Service publishes event to Kafka or another queue.
2. Analytics workers consume events.
3. Events are stored in analytics database.
4. Aggregations are computed asynchronously.

12. Availability and Fault Tolerance

Failure Handling

High Availability Techniques

• Multi-AZ deployment
• Load balancing
• Database replication
• CDN caching
• Health checks
• Circuit breakers
• Graceful degradation

13. Security Considerations

Abuse Prevention

• Rate limit anonymous users.
• Scan URLs against malware/phishing databases.
• Block suspicious domains.
• Prevent alias squatting.
• Add CAPTCHA for suspicious activity.
• Monitor abnormal click patterns.

Privacy

• Hash IP addresses.
• Avoid storing unnecessary personal data.
• Respect regional privacy laws.
• Provide URL deletion support.

Brute Force Protection

Short codes should not be too short.

Recommended minimum length:

6-8 Base62 characters

Base62 capacity:

14. Capacity Estimation

Assume:

• 10 million new URLs per day.
• 1 billion redirects per day.
• Average long URL size: 500 bytes.
• Average metadata size: 500 bytes.

Storage for URL Mappings

10 million URLs/day × 1 KB = 10 GB/day

Yearly storage:

10 GB/day × 365 = 3.65 TB/year

With replication factor of 3:

3.65 TB × 3 = 10.95 TB/year

Redirect Traffic

1 billion redirects/day
= 11,574 redirects/second average

Peak traffic may be 5-10x average:

~60,000 to 120,000 redirects/second

15. Technology Choices

16. Trade-Offs

16.1 301 vs 302 Redirects

Recommended default:

Use 302 redirects unless the user explicitly wants a permanent redirect.

16.2 SQL vs NoSQL

Recommended approach:

Use SQL for early scale, then move hot lookup paths or large deployments to NoSQL/sharded storage.

17. Observability

Track the following metrics:

System Metrics

• Request rate
• Redirect latency
• Error rate
• Cache hit ratio
• Database read/write latency
• Queue lag
• CPU and memory usage

Business Metrics

• URLs created per day
• Redirects per day
• Active users
• Top domains
• Spam detection rate
• Expired links count

18. Example SLAs

19. Future Enhancements

• QR code generation.
• Branded domains.
• Team workspaces.
• Advanced campaign analytics.
• Geo-based redirects.
• Device-based redirects.
• A/B testing destinations.
• Link preview pages.
• Bulk URL creation.
• Webhooks for click events.

20. Summary

This URL shortener system is designed for high read throughput, low-latency redirects, and horizontal scalability.

The key design decisions are:

• Use Base62 encoded unique IDs for compact short codes.
• Use Redis and CDN caching for fast redirects.
• Store URL mappings in a durable primary database.
• Process analytics asynchronously through a queue.
• Use rate limiting and safe browsing checks to prevent abuse.
• Scale globally using replicas, sharding, and edge caching.