-1

I have been learning a lot of new things lately, DevOps, Cloud Computing, Monitoring, and Security. I have been facing my problems dead on, but System Design seems to be a bit complicated. I have watched YouTube Videos and Read Articles online. Just so it is clear, for example, I know that for System Design one should:

  1. Gather all Functional Requirements and Non-Functional Requirements
  2. Figure out Capacity Estimates
  3. Design Core Entities or in other words Database Design
  4. API Design
  5. Create High Level Overview
  6. Deep Dive into the actual Architecture

That was a checklist from a YouTube Teacher (Ex Meta Staff Engineer).

There is probably other ways to unravel the core parts of a Web System, all the Functional Requirements and there specific needs, like a Database Cache, or a Queue, or Load Balancer, or a Replication for Databases... I have learned there are two ways of seeing this

  1. Big Design Up Front (BDUF)
  2. YAGNI

Meaning either you create a Perfect System from the start = BDUF!

Or you start with the basics, MVP features, less System Design = YAGNI

But my question is, even if you choose which ever philosophy you choose (in my case I choose YAGNI, but leaning towards a hybrid approach) how do you implement the more complicated patterns, or a better way to ask what I am asking is, when to choose a Complicated pattern like Queue Based Load Leveling or Sequential Convoy or Leader Election or CQRS? Should I keep iterating over a list like this for each Web Service (Timeline Service, Message Service, Post Service as examples) -> https://roadmap.sh/system-design until the Web Service handles all the assumptions like will this Web Service need a Load Balancer starting with N amount of users, for example, or should I implement a Sequential Convoy Pattern since my Messages need to all be in order for some reason. My point I am trying to make, is should I know all of these patterns by heart and know when to use it when the time arises? Or do I look up information when a the System runs into a bottleneck or when I am planning everything up front.

That is my first question... Should I keep iterating over a list of System Design Patterns -> https://roadmap.sh/system-design for a specific Web Service say the Timeline Service for a Social Network, until every corner has been thought of, and then move on to the next Web Service.

My other question to this first question is, should I memorize all System Design patterns so I can implement the Pattern on demand in any specific context, either in BDUF meaning I can work like a machine as I plan the System and throw every System Design Pattern at the Web Service there is until satisfied... or either in YAGNI when a New Relic Alert pops up I know what to do. Or is looking up information okay to do, like on here or Googling stuff up, in a professional setting.

Another question is should I start to add System Design patterns Up Front following the BDUF Philosophy... or should I use the YAGNI Philosophy. Or a Hybrid Approach.

Another question is even with either or or the Hybrid Approach, the correct way to know when to add another System Design Pattern say the Competing Consumers Pattern is by first receiving an alert from an Instrumentation Tool let's say New Relic, and solve that bottleneck or failure from New Relic by using a System Design Pattern. Is that accurate? Or why else would there be a need for implementing a new System Design Pattern let's say 2 years into production if not by the Observability System like New Relic complaining about high Disk I/O... Is that correct or am I mistaken?

I understand that things like DevOps (Automation Servers, Configuration Management Tools, Unit Testing Tools, Code Coverage Tools, Static Code Analysis Tools, Docker, Cluster Management Systems) are a requirement and not something to add when needed. I understand that most Cloud Services (like AWSs EC2, Cloud Formation, ELB, Auto-Scaling, ECS, EKS, IAM, RDS, S3 if using this Storage, etc...), most of these Services are a requirement not added when needed. I understand Monitoring Tools like New Relic or Data Dog and ELK are a requirement. I know about Security which is a topic in itself, but also a requirement. And my point to this is, the tools listed above, besides maybe some Cloud Components like Storage or Load Balancers, etc..., aren't apart of System Design right? They are all required in modern Software Development and should not be "chosen" for.

Improve this question
16
  • 1
    Getting us to vet your AI answer is just not an appropriate question. There is an appropriate question in here but it should be concretely addressed. I have voted to close it for lack of clarity but will vote to reopen if the question body was fixed. Commented Nov 17 at 4:46
  • 1
    @Arthur that does sound very interesting. As for your last question: no. You pick new pattern only when you've exhausted all other options. For example typically it is a lot easier (and cheaper) to fix a performance problem by simply adding hardware. I would never start a project by designing say AWS infrastructure. This may become necessary later however, so your project should be prepared for an infrastructure change (which not always is obvious). Commented Nov 17 at 8:51
  • 2
    You generally start simple, and add complexity and patterns when and where needed. With experience you can start anticipate problems, and apply solutions proactively. If you lack the experience or are unsure if you can reach the required scale you might need to do some testing and/or prototyping to gain some of that experience. It is also important to understand the real problem, if the CPU load is high, what is it used for? In my experience it is common that resources are wasted doing redundant work. And once you understand the problem, the fix is often straightforward. Commented Nov 17 at 9:37
  • 6
    @Arthur, I think you're approaching this the wrong way. You seem to be collecting patterns and all kinds of disparate technologies. This leads to a solutions-in-search-of-problems approach. The better way is to get a job where you will be presented with specific problems to find solutions for, and the potential to work adjacent to experienced staff. YouTubers talking about their experience with large multinational online technology vendors like Meta, are not really comparable to writing everyday business applications and similar Commented Nov 17 at 10:44
  • 1
    Such tools may be of great help if you have the problems the tools (or patterns) are intended to solve. If you have already built a system you probably know what problems you have, but if you are still designing the system it can be difficult to anticipate problems without experience of similar systems. Tools like CDN, load balancers, monitoring etc are probably not very useful if you can serve all requests from a single server with sufficient margin. So how many requests will you have? How many requests can one server handle? Experience will help avoid both under and over designing. Commented Nov 17 at 14:05

1 Answer 1

Reset to default
0

When to chose a pattern in YAGNI? Well, essentially, when you know that you'll need it.

Basic example. I regularly bootstrap projects for my personal usage. As I can't afford them to become unnecessarily complex, I try to make them as simple as possible from scratch.

If, for instance, I need to store data, but at an early stage, I have no idea if it's better to use a relational database or a document database or something completely different or a mix of several types, I just don't make the decision and go for the simplest approach, which is usually a bunch of plain JSON files. As soon as the project grows and this original solution becomes untenable, I move to something else, usually with (1) a clearer idea of the target and (2) a design that makes it easy to swap one implementation for another—since I knew I would have to switch eventually.

If, on the other hand, I'm absolutely sure that PostgreSQL is the thing, and by the way I also need to have an Apache Kafka, and I can justify that, well, no reason to wait—it's much simpler to put those ones from the beginning. Which doesn't mean that I don't need to design a system on a way that makes it still possible to swap a given implementation for something else, by the way.

I have learned there are two ways of seeing this [...] BDUF [and] YAGNI.

Unless you work on very specific projects in very specific domains (such as aerospace), forget about BDUF. Designing a perfect system from scratch for a piece of average enterprise software is a fallacy—and usually a sign of complete incompetence.

My point I am trying to make, is should I know all of these patterns by heart and know when to use it when the time arises?

In general, try to be aware that such things exist. But you don't have to remember the details, as you can always skim through the documentation when you need them.

Should I keep iterating over a list of System Design Patterns [...] until every corner has been thought of, and then move on to the next Web Service.

This is not how software development usually works. More often than not, you'll find yourself sharing limited time between several projects. Few chances to encounter a situation where you can afford to work on something “until every corner has been thought of.”

I understand that things like DevOps [...] are a requirement and not something to add when needed. I understand that most Cloud Services (like AWSs EC2, Cloud Formation, ELB, Auto-Scaling, ECS, EKS, IAM, RDS, S3 if using this Storage, etc...), most of these Services are a requirement not added when needed. I understand Monitoring Tools like New Relic or Data Dog and ELK are a requirement.

That's a lot of technologies you list here. Let me share a secret with you. Just don't tell anyone, okay?

There is not a single technology that is mandatory for all your projects. No matter what the salespersons tell you. And yes, the salespersons have a tendency to say that if you don't use their new shiny thing in your project, you have zero chances of succeeding. Guess what? There are plenty of projects that, by the past, succeeded without using the new fashionable thing. And there are plenty others that use it and fail.

If your project has tests that actively prevent regressions, if you have automated build and deployment, if you can access logs when things go wrong and if you can scale easily when you encounter an increase in system use, you're doing absolutely great.

Even when your build is a series of Bash scripts, the solution gets deployed on a bunch of Raspberry Pi devices, and the logs are plain text files to load with scp.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.