Please enable JavaScript.

Coggle requires JavaScript to display documents.

Designing Pastebin (1.5 Hours) - Coggle Diagram

- - - - Attacker can simply copy and paste malicious scripts as url for victims to click. And they can even just given a fake shortened url for CSRF or XSS.
- - - - User can paste specific among of pastes in our web app: for guests without an account, they can paste 10 files per 24 hours; for user with account they can paste 25 files per 24 hours; and for members we allow them to paste 250/day. For more pastes, they need to contact us.
      - For normal user, we only allow them to paste say 500KB (or 500K characters), but for member we allow them to paste 10 MB (or 10M characters).
    - - User should be able to set expiration date for the pastes.
    - - User should be able to set public, private, and unlisted status for the pastes based on their user status.
    - - Users should be able to highlight the kind of programming language they are using (or we can use some detection technique to do it automatically)
  - - - We hope that the reading speed is fast, and the submitting time can be a little bit slower if the file is large.
    - - We want to prevent any kind of data loss, since user might have sensitive or important note that are not baked up when typing.
    - - We want to make sure that our service is consistently available, and have least downtime possible.
      - :warning: Paste links should not be predictable
- - - - :red_cross: User amount (Included in the traffic estimates)
        
        Total user amount
        
        300 million users
        
        Say only 50% of them are active, which means only 150 million active users that creates write.
        
        Normal user
        
        30% of them are normal users: 45 millions
        
        Members user
        
        20% of them will be member: 30 million users
        
        Guest users (without account, just read)
        
        Say 50% are guest users: 150 million * 0.5 = 75 million users
      - :red_cross: Contents (Storage Estimates)
        
        Say each active user, if they are guest, they create 10 pastes per day, and user will be 25 per day, and member will be 250 per day.
        
        For guest and normal user, they can create at most 500KB/file, and for members, they can create 10MB/file
        
        Total content size will be
        
        Total = 75 million * 10 files/day * 500KB/file + 45 million * 25 files/day * 500 KB/file + 30 million * 250 files/day * 10,000KB/file
        
        Total = 375 billion + 563 billion + 75,000 billion = 76 trillion byte ≈ 75TB/day
        
        But since we only expect 1% of the maximum size is used (most of the people won't fill their paste with text, and not everyone will use all their limits everyday)
        
        We have 75 TB/day * 0.01 = 750 GB /day
      - :red_cross: Throughputs and Bandwidth (Traffic & Bandwidth Estimates)
        Bandwidth is basically the size of the traffic.
        Traffic is the amount of transaction in the system for the use case.
        
        Since we have a clear per day limit, we can calculate the average QPS (query per second) for all active users
        
        Average QPS = (10 * 75 million + 25 * 45 million + 250 * 30 million) / (75 + 45 + 30 million) = 62.5 million QPS
        Average size per second = `62.5 million QPS * 500 KB/file = 31 billion = 31 GB
        
        If we only consider 10% of this size, then it will be 3.1 GB/second for read.
  - - - New paste per day is about 1 million (write), and access to these pastes will be about 5 times the size of the write, thus 5 millions (read).
        
        This is because paste bine won't be as popular as Facebook or Twitter.
      - QPS for Write
        
        1 million/day / 86400 s/day ~= 12 queries/s
      - QPS for Read
        
        12 read QPS x 5 = 60 write QPS
    - - Single paste
        
        Each file can go up to 10MB, but most of them are just code and snippets. Thus we assuming on average each file is just 10 KB (10,000 characters).
      - Storage usage per day
        
        We will write 1 million pastes per day, and each paste is 10 KB on average. Thus the total storage need for a day is about 1 million pastes/day * 10 KG/pastes ~= 10 million KB = 10 GB/day
      - Storage usage for 10 years
        
        10 GB/day x 365 days/year x 10 years ~= 37 TB
      - Storage for paste key
        
        Number of pastes in 10 years
        
        1 million/day 365 days/year 10 years ~= 3.7 billion pastes
        
        Number of characters needed for 64 base encoding ([A-Z, a-z, 0-9, ., -])
        
        About 6 characters, since 64^4 ~= 68 billions, which is enough for identifying 3.7 billion pastes in 10 years
        
        Given that we will have 3.7 billion pastes, and for each paste the key size is about 6 bytes. The total size of key in 10 years will be about 3.7 billion keys * 6 bytes/key = 22GB.
      - :warning: Using 70% capacity model (we don't let our data to exceed 70% of the total storage)
        
        37 TB / 0.7 ~= 52 TB
    - - Ingress (write)
        
        We have 12 writes/s, and each write is about 10KB. So the size of writing per second will be 120 KB/s
      - Egress (read)
        
        We have 12 * 5 = 60 read/s, and each read is about 10KB. Thus the size of read QPS is about 600 KB/s = 0.6 MB/s
    - - We use the 20/80 rule and assuming we only need to save 20% of the total read size everyday for hot spots
        
        5 million read query/day 10 KB/query 0.2 = 10 billion bytes = 10 GB
- - - - Key: pasteID: varchar(16)
        Values:
        
        OwnerUserID: varchar(8)
        
        PrivacyLevel: enum
        
        CreationDate: datetime
        
        ExpirationDate: datetime
        
        LanguageCode: varchar(1) for popular ones
        
        :red_cross:PasteText: varchar(500) or varchar(10000) [:warning: we don't store text here]
        
        :warning: PasteKey: varchar(512) this just refers to an external object storing the contents of the paste
    - - Key: userID: varchar(8)
        Values:
        
        DateJoined: datetime
        
        TotalPaste: int
        
        TotalView: int
        
        TotalPasteViewed: int
        
        PrivatePaste: int
        
        UnlistedPaste: int
        
        PublicPaste: int
        
        :warning: Name: varchar(20)
        
        :warning: Email: varchar(32)
        
        :warning: LastLogin: datetime
- - - - First we generate a six-letter random string, which would serve as the key of the paste. (we will use KGS to generate all the keys beforehand, so we can have a Key-DB and a replica for backup or fault-tolerance).
        
        Then we can insert the paste to the datastore, and metadata of paste and user to datastore too.
        
        Error handling: if the key is duplicated for some reason (with KGS this will not happen by server), such as user entered duplicated key, we should return an error. Or we can generate new key.
        
        PS: KGS design: we save all the key in two tables: used and not-used table. If we want to save some key to the memory cache of the application servers, then we move the allocated key to used table immediately after we share the key.
        Key lost by server down can be ignored: because we have more than 68 billion options, which is much more than we need.
    - - We can retrieve the information of pastes based on the URL or paste ID from the datastore. If it is found, it returns the paste's contents. Otherwise, an error code is returned.
  - - - :warning: We can divide the datastore layer into two:
        
        Metadata database
        
        Using relational DB like MySQL or a Distributed Key-Value store like Dynamo or Cassandra.
        
        Object storage
        
        Using AWS S3 Standard class is good, and we can have Infrequent Access for backup (or even Glaciers class for even lower frequently access data or archieves)