How to design a System or Software Step by Step?

Let's understand process by designing a URL shortener. While it may seem simple, designing a URL shortener requires thinking through multiple aspects of system design, such as scalability, database management, and handling high traffic. The first step in any system design is to clarify the requirements. You want to make sure you understand what is expected, and asking questions and making sum calculated assumptions can help refine the scope.

Before diving into the system design process, it's important to first understand what a URL shortener is, as it is a prerequisite for this tutorial.

What is a URL Shortener?

A URL shortener is a service that takes long URLs and converts them into shorter, more manageable links.
The shortened URL redirects users to the original, long URL when clicked.
It’s commonly used to make links more user-friendly, especially on platforms with character limits (e.g., Twitter) or in marketing campaigns.

Key Features of a URL Shortener:

Short URL Generation: Automatically generates a compact version of long URLs.
Redirection: Redirects users from the short URL to the original URL when accessed.
Custom Short URLs: Allows users to create personalized short URLs (e.g., short.ly/customname).
URL Expiration: Option to set expiration dates for short URLs, after which they are no longer valid.
Click Analytics: Tracks the number of clicks on each short URL and provides insights like geographic location and device type.
Bulk URL Shortening: Supports shortening multiple URLs at once for large-scale campaigns.
Security Features: Includes protections like spam filtering, malware detection, and the ability to block certain domains.

Note: The information above is provided to help you understand the terminology used in URL shorteners, ensuring you can follow the topics discussed later.

Step 1: Clarifying the Requirements

When you're asked to design a system like a URL shortener, the first thing you should do is fully understand what is expected. It's important not to jump straight into the design. Instead, take a moment to clarify the requirements and the scope of the problem. This step ensures that you’re solving the right problem and designing a system that meets the real-world needs of the service.

Here’s how you can think through this process in a real-life scenario:

1. What is the primary use case?

Start by understanding the core functionality of the service. Is the main goal just to shorten URLs, or does the service require additional features like analytics, expiration, or security?

Ask:

“Are we only focusing on generating and resolving short URLs, or should we also track analytics like click counts?”
“Will users need advanced features like custom short URLs, or should they be randomly generated?”

Why it matters:
This helps you understand the key functionalities that are expected. If analytics and custom URLs are required, that will affect your database design and the components you choose.

2. How long should the short URLs last?

Determine whether the shortened URLs will expire after a certain time or if they should exist indefinitely.

Ask:

“Do the URLs have an expiration date, or are they permanent?”
“If URLs expire, how long is the lifespan, and do we need to notify users before expiration?”

Why it matters:

If URLs are meant to expire, you’ll need to account for this in your database schema and implement a mechanism to clean up expired URLs. If they don’t expire, you'll need to design the system to handle a growing database over time.

3. How many users and how much traffic should the system support?

Understanding the expected user base and traffic load is critical for determining the scale of the system. The way you design a URL shortener for a small app will be different from one built for a global audience.

Ask:

“Is this a global service? How many users are expected to use it daily?”
“How many requests per second should the system handle during peak times?”

Why it matters:
This helps you gauge how to build the infrastructure. A global service with millions of users will need multiple data centers, robust load balancing, and caching. A smaller service might be fine with fewer resources but should still allow for future scalability.

4. Do we need to support custom short URLs?

Custom short URLs allow users to create their own short links, such as short.ly/mycustomurl. This feature can make the service more user-friendly but also introduces complexity, like handling collisions when multiple users want the same custom URL.

Ask:

“Do we need to support custom short URLs, or should all short URLs be auto-generated?”
“If we support custom URLs, what should happen if a requested URL is already taken?”

Why it matters:
Supporting custom URLs means you'll need logic to handle collisions, and you may also need to introduce user authentication to ensure users can claim and manage their custom links.

5. Should we ensure security and prevent malicious links?

URL shorteners can sometimes be used to hide malicious links. It’s important to understand whether the system needs to verify or filter links before shortening them.

Ask:

“Are there security concerns with preventing users from shortening malicious or harmful URLs?”
“Should we implement spam protection, or block certain domains from being shortened?”

Why it matters:
If security is a priority, you’ll need to integrate checks, like scanning URLs for malware or blacklisting certain domains. This can add complexity but ensures that the system is safer for users.

Defining the Scope After Clarifying the Requirements

Once you’ve clarified these key points, you’ll have a much better understanding of what the system needs to do. Here's a summary of what you might gather after asking these questions:

Generate unique short URLs: The core functionality remains to shorten URLs into a compact, user-friendly format.
Custom short URLs: Users may want to create personalized URLs, which will require additional logic to handle potential conflicts.
Expiration: URLs might expire after a certain time period, requiring a system for managing expiring links.
Traffic expectations: The system needs to handle millions of requests efficiently, with load balancing, caching, and database optimization to ensure smooth performance.
Security: We may need to implement some level of security to protect against malicious URL submissions.

With the scope well-defined, you’re now in a much stronger position to design a URL shortener that meets the specific needs of the system. The next steps will involve translating these requirements into a high-level system design and then diving into the components.

Step 2: High-Level Design

Next, you can discuss the high-level architecture of the URL shortener system. A URL shortener consists of a few key components:

User Interface: Where users can enter a long URL and get a short URL.
Backend Service: This handles generating the short URL, storing it, and managing the redirection when the short URL is accessed.
Database: Where the mappings between short URLs and long URLs are stored.

Here’s a high-level flow:

The user submits a long URL through the user interface.
The backend generates a unique short URL.
The mapping of the short URL to the long URL is stored in the database.
When the short URL is accessed, the system fetches the original long URL from the database and redirects the user.

At this stage, you can also explain the choice of technologies. For example, the backend might be implemented using a framework like Node.js or Python’s Flask. The database could be an SQL database or a NoSQL option like MongoDB, depending on how you anticipate storing and retrieving the URLs.

Step 3: Deep Dive into Key Components

1. URL Generation Logic

The heart of a URL shortener is generating unique, short URLs. One approach is to use Base62 encoding. Base62 uses a combination of lowercase letters, uppercase letters, and digits, giving us 62 possible characters for each position in the URL. This makes the short URL compact while still allowing for a large number of possible combinations.

You can also mention hashing algorithms like MD5 or SHA-256, but the problem with hashes is they create long strings, which defeats the purpose of creating a "short" URL. Another option is using an auto-incrementing ID, which is then encoded into Base62 for the short URL.

2. Database Design

In the database, we’ll need a simple table to store the mappings between short URLs and long URLs. Here's a basic schema:

Short URL | Long URL                             | Creation Date | Expiry Date
----------|--------------------------------------|---------------|--------------
abc123    | http://example.com/verylongurl...    | 2023-09-30    | NULL

The Short URL column will contain the generated short string, and the Long URL will store the original URL. If short URLs are meant to expire, you can include an Expiry Date column to automatically remove old URLs.

3. Redirection Service

The redirection logic is simple: when a user accesses a short URL (like short.ly/abc123), the system looks up the short URL in the database, retrieves the corresponding long URL, and redirects the user.

It’s important to mention that the lookup should be optimized for fast retrieval, as this will be a common operation. Using caching (e.g., Redis) for frequently accessed URLs can speed this up significantly.

Step 4: QPS Calculation

Let’s now calculate the expected QPS (Queries Per Second) to understand how to scale the system. We’ll use three different examples to illustrate this process.

URL Shortener with Consistent Traffic (24-Hour Traffic)

Assume the system handles 1 million requests daily, including both URL creation and redirection.

Total Requests: 1,000,000 requests per day.
Time Period: 86,400 seconds (24 hours).
QPS: 1,000,000 ÷ 86,400 = ~12 QPS.

The system should be designed to handle an average of 12 QPS, but it’s important to consider occasional spikes.

To calculate the QPS for peak hours in the same example, we need to assume a shorter, busier period when traffic is concentrated. Let’s say peak traffic occurs over 4 hours during the day, which is a common assumption for systems that see higher usage during certain periods.

Peak Hour Calculation

Total Requests: 1,000,000 requests per day.
Peak Traffic Period: 4 hours (which equals 14,400 seconds).
Let’s assume that 50% of the daily traffic occurs during these 4 peak hours. This is often a reasonable assumption for systems with busy periods.

Steps

Peak Requests: 50% of 1,000,000 requests = 500,000 requests during peak hours.
Time Period: 4 hours = 14,400 seconds.
QPS during Peak Hours: 500,000 ÷ 14,400 = ~34.7 QPS.

During peak hours, the system should be designed to handle approximately 35 QPS to manage the concentrated traffic load. This is significantly higher than the 12 QPS average over 24 hours, which highlights the importance of designing for peak traffic to prevent bottlenecks.

Step 5: Scaling the System

To handle higher traffic and ensure the system scales, you can discuss the following strategies:

Load Balancing: Distribute incoming traffic across multiple servers to avoid overloading any single server.
Caching: Use a caching layer (e.g., Redis) to store frequently accessed URLs in memory, reducing database queries.
Database Sharding: As the number of stored URLs grows, sharding the database can help manage larger datasets by partitioning them into smaller, more manageable chunks.

Step 6: Handling Edge Cases

1. Collision Handling

When using algorithms like Base62, there is a chance of generating duplicate short URLs. You should discuss strategies to handle collisions, such as checking for duplicates in the database before finalizing the short URL.

2. Expired URLs

If the system supports expiring URLs, you need a mechanism to clean up expired URLs. You can run periodic background jobs to delete expired URLs from the database.

Step 7: Trade-offs and Improvements

In every system design, there are trade-offs to consider:

Storage vs. Speed: A NoSQL database (like MongoDB) might offer faster writes at the expense of slightly less consistency compared to SQL databases.
URL Length: Shorter URLs are more user-friendly but allow for fewer possible combinations. Base62 provides a good balance, but longer URL strings can increase scalability.

You can also suggest potential improvements, such as adding analytics to track how many times each short URL is clicked.

Conclusion

In this tutorial, we walked through the steps of designing a URL shortener, from clarifying the requirements to estimating QPS and handling scaling issues. By breaking the problem into smaller parts, understanding the trade-offs, and preparing for edge cases, you can demonstrate a thoughtful and complete design approach.

Clap here if you liked the blog