Introduction
My story
It all started when my interviewer asked me, "How would you design your business-logic layer?"
I'm sitting there in the middle of a dimly lit room, in front of three senior software developers, (none of them looking very pleasant I might add), hoping that not a single one of them catches a glimpse of the bead of sweat I just felt roll down the side of my head.
Business-logic layer... That's practically another language to me. I hadn't the slightest idea of how to answer it.
After a few seconds of deliberation, I started to say words. I probably said something about MVC (model-view-controller), how I organize my folders, and maybe something about how I structure my controllers in a Node.js & Express.js project.
Every word leaving my mouth was an extra bit of dirt flung into the hole I was digging for myself. As I was speaking, I was realizing more and more that nothing I said had anything to do with "business-logic".
Eventually, the sense came over me to stop speaking.
I was then met with a very uncomfortable couple of seconds of silence. After a head nod from one of the fellows and some glances at each other, one interviewer said "OK, I think that'll be about it. Thank you, do you have any questions for us?"
Aaaaaaab-solutely not, I'll see myself out — said my brain. Trying to erase that moment from my memory as soon as possible, I probably asked "So, what's the culture like here?"
Safe to say, I didn’t get the job. My recruiter even dropped me. Thinking back to what I did to prepare myself for the interview, I remember studying:
- Whatever I found off of the "how to prepare for a full-stack web developer interview" Google search
- The algorithms in Cracking the Coding Interview
- Javascript and all its silly language quirks like closures, IIFEs, and passing by reference vs. value.
Not only that, but I had been working on building my own startup company during the time I was in school, and I thought that all that code I wrote would have translated into some serious experience.
I may have completely bombed that interview, but looking back today — that was one of the best things that could have happened to me. It was that moment that I realized there was an entire world of software design and architecture that I needed to teach myself. I became incredibly interested in this topic. I was going to learn. And I wasn't going to fail the next interview.
My early research
Mid-2018, in the middle of my research, I really needed to pay the bills, so I ended up landing a job as a Frontend Consultant. I was mostly looking for a low-stress job I could perform to hack my career by purchasing and studying as many books on software design, patterns, and architecture as possible and applying everything I learned from those books to improve my cumbersome ~300k-line Node.js startup code.
Every evening after work for 8 months, I read books and wrote code. I reviewed everything they taught me in school, but this time, learning from the experts' books.
I started with Object-Oriented Programming. What were the 4 principles of OOP again? What does an abstract class do anyway? Why would you ever want to use a static method? I heard that inheritance was a bad thing, right? Why should I avoid that?
As I was learning, I wrote down all of the concepts that I’ve heard of but never really cared to try to understand like POJOs, dependency injection, dependency inversion, inversion of control, concrete classes, design patterns, and principles, etc.
I revisited ideas that I knew about but never fully understood in depth like coupling, cohesion, managing dependencies, and separation of concerns. Soon, I was learning a lot of new things like the hexagonal architecture, Conway’s law, Use-case driven development, TDD, and the SOLID principles.
The point where it all really felt like it paid off was when I discovered Domain-Driven Design. My learning approach was to learn by doing in addition to teaching others. I quit my job, refactored Univjobs' codebase using Domain-Driven Design practices, and began to regularly share what I was learning about software design and architecture with my peers online on my personal blog.
Today, thousands of readers every month are learning how to write testable, flexible, and maintainable code @ khalilstemmler.com.
Why I wrote this
Two years later, after about seventy-or-so blog posts, ten thousand newsletter subscribers, and the dissolution of Univjobs, I wanted to work on building a Domain-Driven Design course. It was the most useful topic I discovered on the blog, and it also seemed to be pretty popular.
As I started to learn more about who was interested in this course, I realized that the skills gap in software design skills was deep.
Maybe there were bigger fish to fry.
Practical design skills aren't being taught
The truth is, not a lot has changed about the fundamentals of software design over the past 60 years, but there's a huge lack of training on it.
By and large — junior and intermediate developers are still left confused and with a lack of guidance on how to confidently and consistently develop high-quality software.
From simple stuff like how to structure your project or name things well so that things can be understood, found and changed, to how to decide on an architectural style and organize your business logic, we're leaving developers out to dry.
No one is teaching developers essential software design and architecture fundamentals
I tend to agree with Eric Elliot, who says that "99% of working developers lack solid training in software design and architecture fundamentals. 3/4 of developers are self-trained, and 1/4 of devs are poorly trained by a dysfunctional CS curriculum. And almost zero companies make up for those deficiencies with in-house training and mentorship. In other words, if you simply accept the status quo and refuse to offer training in-house, your team will be the blind leading the blind".
Bad design is extremely expensive
Bad design has some seriously costly consequences. According to CNBC, in September 2018, companies spend an approximation of ”$85 billion yearly dealing with bad code”.
That blew me away.
Why is it happening?
There are so many reasons, but an executive summary is:
- Developers aren’t being taught essential software design skills
- Most companies say they practice Agile
- Actually practicing Agile means tight feedback loops and the ability to change and refactor code
- To refactor code, we need tests
- To write tests, we need to know how to write testable code
- Most developers can't write testable code
- Therefore, productivity plummets over time
How projects with good tests, bad tests, and no tests progress over time. Without tests (or with bad tests), it's hard to continue to safely make progress because new changes introduce regressions. Refactoring without tests is a great way to introduce regressions.
There's a lot to learn from the past
Every so few years, a new technology comes out that builds on things we’ve learned and rely upon from the past. Redux is the observer pattern. GraphQL is architectural dependency inversion.
Any JavaScript developer from the 2010s knows how frustrating it can be to try to keep up with the current way to handle state, organize components, etc — only to have new “best practices” and tools emerge a few months later.
As an industry, we tend to be very quick to forget our roots and quick to reinvent the wheel.
Eventually, this got to a point where I just decided I wasn't going to chase the newest tools and technologies anymore. I wanted to look backwards first before looking forwards.
I decided I was going to learn the practices, patterns, and techniques that influence all our tools and innovations.
It's Santayana's curse after all:
"and those who know not of history are also doomed to repeat their mistakes".
A philosophy of mastery
A few years into my study, I became very interested in philosophy. While I was reading all kinds of books and diverging around, it became clear that this process could go on forever.
I was fortunate to have met a wise soul who would then introduce me to Aristotle’s philosophy of wisdom and mastery.
Differentiating between two types of wisdom: Sophia & Phronesis
Aristotle was one of the most influential teachers in history. He wrote about physics, logic, metaphysics, ethics, music, politics, linguistics and more. Aristotle exerted a unique influence on almost every form of knowledge in the West.
In his book, The Nichomachean Ethics, he distinguishes between two different types of wisdom: sophia, meaning "factual wisdom" *and *phronesis, meaning "practical wisdom".
I found it helpful to distinguish the two.
Sophia
The word philosophy comes from the originating Greek word, philo-sophia, the "love of wisdom".
It’s knowledge concerned with the structurally teachable, composable, logical, factual knowledge that we often equate to science.
For example, a book titled "1000 Facts About Sharks" is likely going to be a sophia-sided book.
Sophia-sided programming books are good because they often introduce you to the vocabulary and ideas in a new domain.
And that’s important when we start out learning anything.
We want to start with an ontology.
What’s an ontology? It’s a map.
Ontologies
For example, when I later started essentialist.dev, I had to learn marketing and sales.
What did I know about marketing and sales?
Nothing.
So, I both threw myself in with coaches and bought a number books to load the new domain into my head.
This helped me create a map of concepts — a graph of relationships. A way to understand the world I just walked into, how things relate, and why we do what we do.
Thats what an ontology is, and that is very much what you are reading right now, this wiki.
Phronesis
Now, ontologies are important, but they’re only the first step of learning.
Phronesis, then, is practical wisdom.
This is about practical action. It’s about how best to act in scenarios that call for it, it’s much more difficult to teach, because books can’t account for the quantum nature of reality and all the various potential decisions and actions you can take at that time.
Sure, they can provide guidance, and sure, while there are influential, timeless, classic programming books that withstand software trends — books that you would definitely benefit from reading — I learned that you still need direct experience.
You need to learn through practice how to take Right Action.
But how?
The key to master anything: Aristotle’s process of Mastery
Over 2000 years ago, Aristotle identified a process for becoming successful at anything (and everything) in life.
It goes like this.
Step 1: Identify the purpose of the thing we want to use. Knowing the purpose of something allows us to evaluate the ways we can use it.
- e.g. knives are for cutting
Step 2: Identify the qualities that make it accomplish its purpose well.
- e.g. if knives are for cutting, a sharp knife is a good knife, because it cuts well.
Step 3: Mimic others who accomplish the goal well
- e.g. we find people who are good at cutting and observe them.
Step 4: Develop our own principles through practice
We figure out the principles guiding their success.
- e.g. we learn what makes them good at cutting.
This might be the halfway point between two extremes.
- e.g. to cut something, the knife must be neither too long nor too short. (middle point)
It might also be a minimum or a maximum.
- e.g. to cut something, the knife must be sharp enough. (minimum)
- e.g. to cut something, the knife must not be too *heavy. (maximum*)
After enough experience, we build habits and we master what we're practicing.
- e.g. we practice cutting until cutting well is easy and pleasurable.
That's the approach.
Essentially, the key is to model what the pros are already doing and use that direct experience to gather your own set of principles.
So that’s what I did.
Applying the process
I started with the question: “What is the purpose of code?”
Now, I’ve heard all sorts of arguments for code being an art, science, math, and so on.
It certainly depends on who we ask, doesn't it? People write code for different reasons. Interaction designers make generative art, data scientists practice exploratory coding, and I'm sure mathematicians have their own use cases as well. Even musicians code!
Let's consider the purpose of code as an art for a moment.
- e.g. code is for creating art
If that's the case, what qualities would be necessary for code to be considered good art? Perhaps code needs to be clever, interesting, engaging, unique, inspiring, polarizing, or evokes an emotion (sadness, nostalgia, love, angst, anguish). These are all potential qualities for good art (well, at least the weird stuff that I'm into).
It's clear that we could go down this route (and that would definitely be fun), but it's not in line with what the majority of us are using code for.
For the majority of us that work as skilled tradespeople for hire, the purpose of code is to create products. For who? Well, for users & customers.
- e.g code is for creating products for users & customers
Next: Identify the qualities of code that make for having done it well
Well, if it were done well, it would serve the needs of all that use it.
Customers pay us so that we help them make money or save money.
And users use the products so that they get their needs met, typically they get back time or they have their time enriched.
But wait, code actually has three consumers.
Users, customers and the developers that maintain it.
Therefore, some of the essential qualities to accomplish its purpose well are:
- It serves the needs of the customer: The product has requirements — functional and non-functional. Are we meeting them?
- It serves the need of the users. Are the users getting what they want out of the application?
And finally, the needs of the developers. In the past, I said that those needs can be summarized as:
- Flexibility: We want to be able to add new features by merely adding new code, and changing as little existing code as possible.
- Testability: When requirements change and we have to add new features, can we continue to do so without breaking existing features? To stay productive, we need tests. Tests have a lot of benefits, but the important ones to list here are that they prove business correctness, prevent regressions, and give us the confidence to safely refactor.
- Maintainability: In actuality, there are many more code qualities that we could list, like consistency, number of elements, clarity, simplicity — but ultimately, it is our ability to understand code that most effects its maintainability. Why is this so important? Well, imagine you're a customer with a budget. Now imagine that you're paying for time and materials (meaning — you're paying for the spent by the developers working on the project). How long does it take the developers to produce a new feature that adds value to the project? A week? Two weeks? A month? Two months? That's a lot of time. And time is money. Code needs to be designed in a way such that it can be cost-effectively changed.
But today, I simply say that it is code which helps developers achieve their Key Developer Use Cases: discover, understand, add, change, remove, test, debug, deploy, and monitor features.
Therefore,
💡 The goal of code: Code is to create products for users & customers. Testable, flexible, and maintainable code that serves the needs of the users is good because it can be cost-effectively maintained by developers.
Step 3: Master the techniques of those who are doing it well
If phronesis is "practical wisdom", then the phronimos are the wise developers; the ones with practical experience building software with the qualities that we want.
Back when I first wrote this, among my personal phronimos-list are the following developers:
Martin Fowler, Robert C. Martin, Kent Beck, Rebecca Wirfs-Brock, John Ousterhout, Scott Wlaschin, Vaughn Vernon, Steve Freeman, Matthias Noback, Greg Young, Eric Evans, Dan North, Gregor Hohpe, Udi Dahan, Gerard Meszaros, Kevin Henney, Michael Feathers, Pedro Moreira Santos, Vladimir Khorikov, Marco Consolaro, Alessandro Di Gioia, Thomas Pierrain, Adam Dymitruk, and Kent C. Dodds.
The next step was to read their books and blog posts, watch their talks, take their courses, hands-on trainings, and chat with them on Twitter to learn their techniques and philosophies for software design.
Step 4: Develop our own principles by practicing and building experience
Lastly, practice. Follow the techniques that the pros use and master them. Come to your own conclusions and develop principles on when to use them based on your own experience.
Decide to break the rules once you've mastered them.
Personally, after having finished this wiki and having practiced, taught, and implemented the most important parts of this material myself in the real world, I’ve come to believe the most important thing is to master a system or a workflow.
I believe that your ultimate goal is to become a systems thinking software developer, one that can use consistently go from problem to working software on any side of the stack.
I also believe that the most important skills are what I call The 12 Essentials, which cover the 80-20 of the vast ontology of topics we cover in this wiki.
I believe when you understand the major topics (covered in this wikibook), plus when you build the core skills (The 12 Essentials), plus when you master a workflow to put it all into practice (such as XP, or the personal workflow I teach called FA²STR), you are officially set.
While I teach the 12 Essentials and FA²STR in The Software Essentialist trainings, but I’ve left them out of this wiki since my methods are subject to change.
Regardless, that first step is to build an ontology.
That’s what we’ll do here with this wiki.
How this wiki will benefit you
This wikibook is for any developer that feels like their code gets worse instead of better over time.
What you're reading is a wiki that will show professional software developers the essential software design and architecture best practices, topics & ideas they didn’t teach you in school.
This book is for:
- Bootcamp grads, junior, intermediate, and even senior developers — basically, any developer that wants to learn to master software design.
- Developers frustrated with writing buggy code and having things break over time.
- Developers interested in the battle-tested techniques to confidently, repeatedly, and reliably ship code without introducing regressions.
- Developers who want to learn how to design large-scale applications.
- Developers that want to learn how to write clean, flexible, testable, and maintainable software.
- Developers who care about shipping quality code and want to learn how to write code that can actually be tested.
This book has one key goal.
That is to help you build awareness and destroy the problem of the unknown unknowns of software design and architecture. That means everything from clean code to micro-services; and from object design to Domain-Driven Design.
We will be looking at the most influential ideas and wisdom from the phronimos developers — our teachers— on how write testable, flexible, maintainable code.
Depending on your experience, some of these ideas might be new to you, but in reality, very little of these topics and techniques are new. The vast majority of what you'll read are techniques that have been around for a long time. It is, however, likely you haven't been exposed to some of them or merely haven't been shown how to do them thoroughly.
What I'm offering is my personal ontology of techniques from a number of developers much wiser than I. If you find something particularly clever or handy, I have to credit the experience of my predecessors. If you find an error, you can assume that I'm at fault.
How this wiki is organized
When I first started this, I wanted it to be a short handbook.
Ha. Funny!
There is a lot of content here, let me tell you. It’s an entire world we’re building, after all.
So allow me to explain how I've broken this wiki up into several parts.
Parts I through III gets you up to speed on the unknown unknowns of software design, teaches you how to make more human-friendly design decisions, and summarizes all the habits and techniques that experts are using to take a software project from inside the customers' heads to deployed in production.
Parts IV to VII focuses on object-oriented programming mechanics, design principles, patterns, and how to incrementally ship simple, testable code using Test-Driven Development.
In VIII and IX, we shift our focus to architecture, learning different architectures to help better inform the walking skeleton we decide on. It's here that we learn how to construct a bullet-proof web application architecture using Domain-Driven Design, Hexagonal Architecture, and CQRS.
In the final chapters of the book, we discuss advanced testing topics from a systems thinking point of view, and look at techniques like the High Value Integration Tests that can help you get the best return on investment from your testing architecture.
Some chapters contains exercises which will test your understanding and so you can build experience with the techniques.
At the end of many chapters, I also include links to references so that you can deeper by reading the books, blogs, papers, and watching the videos that informed the chapter.
Let’s get started
Now, with all that said, let’s dig in!
Resources
Articles
- https://howtobeastoic.wordpress.com/2016/09/20/sophia-vs-phronesis-two-conceptions-of-wisdom/
- https://en.wikipedia.org/wiki/Phronesis
- https://medium.com/@_ericelliott/if-training-is-not-realistic-youre-in-the-wrong-industry-32e488b864ad
Continue reading
- Introduction
- My early research
- Why I wrote this
- Practical design skills aren't being taught
- Bad design is extremely expensive
- There's a lot to learn from the past
- A philosophy of mastery
- Differentiating between two types of wisdom: Sophia & Phronesis
- Sophia
- Ontologies
- Phronesis
- The key to master anything: Aristotle’s process of Mastery
- Applying the process
- Step 3: Master the techniques of those who are doing it well
- Step 4: Develop our own principles by practicing and building experience
- How this wiki will benefit you
- How this wiki is organized
- Let’s get started
- Resources
- Articles
- Continue reading