Pastebin

Seite 1

Functional Requirements:

Users should be able to upload or “paste” their data and get a unique URL to access it.

Users will only be able to upload text.

Data and links will expire after a specific timespan automatically; users should also be able to specify expiration time.

Users should optionally be able to pick a custom alias for their paste.

Non-Functional Requirements:

The system should be highly reliable, any data uploaded should not be lost.

The system should be highly available. This is required because if our service is down, users will not be able to access their Pastes.

Users should be able to access their Pastes in real-time with minimum latency.

Paste links should not be guessable (not predictable).

Extended Requirements:

Analytics, e.g., how many times a times a paste was accessed?

Our service should also be accessible through REST APIs by other services.

Some Design Considerations

What should be the limit on the amount of text user can paste at a time? 10mb?

Should we impose size limits on custom URLs? Consistent url db.

Capacity Estimation and Constraints

We can assume 5:1 ratio between read and write.

let’s assume here that we get one million new pastes added to our system every day. This leaves us with five million reads per day.

Traffic estimatesNew Pastes per second:

1M / (24 hours * 3600 seconds) ~= 12 pastes/sec

Paste reads per second:

5M / (24 hours * 3600 seconds) ~= 58 reads/sec

Storage estimates:

At this rate, we will be storing 10GB of data per day.

1M * 10KB => 10 GB/day

If we want to store this data for ten years, we would need the total storage capacity of 36TB.

With 1M pastes every day we will have 3.6 billion Pastes in 10 years. We need to generate and store keys to uniquely identify these pastes. If we use base36 encoding ([A-Z, a-z, 0-9, ., -]) we would need six letters strings:

64^6 ~= 68.7 billion unique strings

If it takes one byte to store one character, the total size required to store 3.6B keys would be:

3.6B * 6 => 22 GB

Bandwidth estimates:

For write requests, we expect 12 new pastes per second, resulting in 120KB of ingress per second.

12 * 10KB => 120 KB/s

As for read request, we expect 58 requests per second. Therefore, total data egress (sent to users) will be 0.6 MB/s.

58 * 10KB => 0.6 MB/s

Memory:

Since we have 5M read requests per day, to cache 20% of these requests, we would need:

0.2 * 5M * 10KB ~= 10 GB

5. System APIs

addPaste(api_dev_key, paste_data, custom_url=None user_name=None, paste_name=None, expire_date=None)

paste_key is the url generated

getPaste(api_dev_key, api_paste_key)

deletePaste(api_dev_key, api_paste_key)

Database Design

A few observations about nature of the data we are going to store:

We need to store billions of records.

Each metadata object we are going to store would be small (less than 100 bytes).

Each paste object we are storing can be of medium size (it can be a few MB).

There are no relationships between records, except if we want to store which user created what Paste.

Our service is read heavy.

High Level Design

See diagram

Component Design

Application Layer:

Same key generation which will be from to Key Generation Service: which will be stored in db once url is generated. Refer TinyUrl for more info(Basic system design and Algo)

DataStore Layer:

We can divide our datastore layer into two:

Metadata database: We can use a relational database like MySQL or a Distributed Key-Value store like Dynamo or Cassandra.

Block storage: We can store our contents in a block storage that could be a distributed file storage or an SQL-like database. Whenever we feel like hitting our full capacity on content storage, we can easily increase it by adding more servers.

See diagram