Introduction

The "Design" stage is all about producing a solution to a problem. If you are designing in the absence of a problem you are not designing. The previous article on "Analysis" covers the problem identification pre-requisite. This article covers the solution identification pre-requisite to building meaningful programs.

Having agreed on a common understanding of a problem does not mean we have solved the problem of mutual understanding. The defects of human communication only become more intense as we begin exploring the solution space of a given problem. Design, therefore, is a strategy for minimizing communication failures when discussing solutions to a mutually understood problem. Rich Hickey in "Hammock Driven Development" rightfully described problems of misconception as the driven of most large problems in software projects. This article presents a methodology for solving problems within the web application development domain.

Each topic I will discuss is optional for you to pursue. If your requirement is to print "hello, world" you don't need to design anything. Use your best judgement to determine when certain aspects of the design process can be safely ignored.

Design is adjacent to planning. There are many points in this blog where I could have veered off into a discussion of planning doctrines and the science of project management. I won't do that here. However, I will say a well-designed system is not the only outcome of the design process. Another outcome is a plan. There are logistics associated with accomplishing any task. Design informs you of the task and the plan informs stakeholders of resource requirements.

This blog is informed by my experiences as a back-end engineer for business applications. This advice may or may not apply to other domains of software engineering.

Core Ideas

Gather Functional Requirements

In software engineering and systems engineering, a functional requirement defines a function of a system or its component, where a function is described as a summary (or specification or statement) of behavior between inputs and outputs.

Wikipedia

Functional requirements are what a program does. They are not a description of how a program should accomplish it. For example, given the task to create an identity system some functional requirements might be: user registration, reset passwords, and login users.

When designing any software project you should collect these requirements and document them (if they aren't already). Collecting them, in text, allows others to check your work. It ensures that your understanding of the project aligns with other stakeholders in the business. They might chime in and say "don't forget about logging out users" or they might say "we don't need user registration" or they might ask a question "why do we need to allow users to reset their password".

It's rare you'll hear a project proposal and understand immediately every intention that the requester had. So don't assume. Spend a few minutes writing down the requirements as your understand them and share it around.

There's a hidden benefit here besides mutual understanding. You may, after analyzing the requirements, realize the organization does not have the capability to execute on the requirements. Or you may realize the requirements are mutually contradictory. These are really great things to find out early. You either fix them early at very little cost or you abandon the project at very little cost.

Gather Non-Functional Requirements

In software engineering and systems engineering, a non-functional requirement (NFR) is a requirement that specifies criteria that can be used to judge the operation of a system, rather than specific behaviors. The plan for implementing non-functional requirements is detailed in the system architecture, because they are usually architecturally significant requirements.

Wikipedia

Non-functional requirements are more technical in nature than functional requirements. When we document non-functional requirements what we're trying to do is prepare ourselves for system design work. If we know, for example, that our project requires low latency to be competitive then we know a cron job is not sufficient to solve the problem. Documenting non-functional requirements acts as a funnel for our architectural design. It progressively cleaves off large portions of the software world as we add more requirements.

Some questions you can ask to find non-functional requirements are:

1. What are my latency targets?
2. What is my expected average message volume?
3. What is my expected average message size?
4. What is my consistency model?
5. What are my availability requirements?
6. Can we tolerate network partitions?
7. What are the SLAs on offer?
8. Do I have any legal or data-residency requirements?

Here we hit more exit opportunities. Can we fulfill our non-functional requirements? For example, if data-residency is a legal requirement but we can't deliver it then we can exit the project early. If requirements have been imposed on us which are impossible to accommodate (for example satisfying all three properties of the CAP theorem) then we can exit the project early or otherwise request a loosening of the requirements.

Documenting requirements is a tactic for minimizing information loss. When we communicate informally (verbally or otherwise) we don't provide the full resolution a given problem requires to be solved. Its the engineers job to document, to the best of their abilities, the project's requirements. The most successful projects have a foundation of mutual understanding between all interested parties.

After gathering your requirements you can start thinking about the data you need to send and receive. Your inputs and outputs. At this stage we're not trying to design by contract. We're trying to understand wha

Define a High Level System Architecture

Define Your Core Types

Before drawing our system's architecture diagram, we want to have an approximate idea of what our data looks like. It doesn't really matter how much effort you put into this step or what level of detail you provide. The main questions we're trying to answer at this stage is, roughly, how large are the messages I'll send and receive (to the nearest order of magnitude) and who has what data.

For example, if you expect to receive a message with a user-id but you need a username to complete your message processing logic then you'll need to fetch that data from somewhere else. That's a connection on our architecture diagram. If you expect to receive very small messages you might structure your architecture differently than if you receive very large messages.

Types will be defined in greater detail in future sections.

Draw an Architecture Diagram

Using your knowledge of the problem and with an understanding of your organization's capabilities in mind, draw an architecture diagram which will satisfy the problem. You can use excalidraw or mermaid or whatever you're comfortable with. Drawing out the architecture provides a great visual summary of your system and allows other parts of your brain to contribute the design process (particularly the visual areas).

Once you have a diagram, share it around with a short explanation of the problem you're solving. Consider including your functional and non-functional requirements in bulleted form to quickly bring others up to speed. If you're not the subject matter expert in your organization, consider asking one to review your work. If you are the subject matter expert in your organization consider letting more junior employees review as a mentorship opportunity.

We've found another exit opportunity. Is our architecture diagram utterly insane? Does it look like a nightmare to maintain? Is it going to consume huge amounts of already scarce engineering resources? Now is the time to bring these concerns up. The opportunity cost of a project is not arbitrary. You and your organization only has so much time in the day. You have to be certain you're spending it effectively. A visual architecture diagram is a great way to bring non-technical people into the cost analysis process.

Define Your Wire Protocol Format

At this stage we need to start thinking about how clients and servers are going to interact with one another. They'll communicate across a network boundary and this is colloquially referred to as "the wire". When communicating across the wire we lose a lot of the benefits programming languages provide (such as types and type checkers and compilers). To compensate we need to be thoughtful in our designs and we need to adopt a few techniques for standardizing this inter-machine communication pattern.

It's here that we'll form our first "contracts". "I promise to return X if you promise to send Y". It doesn't make sense to talk about designing wire formats in the absence of any standards so I will now introduce two standards which I prefer to use. API Blueprint and JSONAPI. Both of which I will discuss below.

API Blueprints

An API Blueprint is a high-level, human-readable documentation and design format for describing RESTful APIs. It uses a Markdown-based syntax to define the structure, endpoints, requests, and responses of an API. Because of its plain-text nature, API blueprints can easily accommodate contributions from non-technical individuals. It enables non-technical stakeholders to meaningfully contribute much later in the design process.

An API blueprint embodies the 'design by contract' philosophy. It describes what requests can be made, what parameters resources accept, and what responses to expect. Crucially this contract is defined prior to writing any code. This early design step makes the later development processes 'non-blocking' since each engineer can reference the blueprint independently without coordinating with engineers working in other domains. Without a blueprint, engineers often communicate API details through informal channels, such as a Slack message. These informal channels are error-prone, go out of date quickly, and leave a lot of room for interpretation. A blueprint, on the contrary, provides a precise, structured specification that answers every question an engineer might have: "what is the exact URL?", "what HTTP method does it use?", "what fields are required in the request body?", "what does the response look like on success?", "what does it look like on error?", "what are the possible status codes?". Because this is written in a structured format there is no ambiguity. Every engineer is reading the same document and the same definitions, which dramatically reduces communication requirements.

Finally, once an API blueprint has outlived its life as a development aid it continues its life as documentation. Future consumers will have every detail you had when you initially developed the API.

It doesn't matter how strictly you adhere to the API blueprint specification. What matters is that some useful fraction of it is included in your design process.

An example API blueprint is pasted below for reference.

FORMAT: 1A
HOST: https://api.example.com

# My API

## Users [/users]

### List All Users [GET]

+ Response 200 (application/json)

        [
            { "id": 1, "name": "Foo" },
            { "id": 2, "name": "Bar" }
        ]

### Create a User [POST]

+ Request (application/json)

        { "name": "Baz" }

+ Response 201 (application/json)

        { "id": 3, "name": "Baz" }

JSON API

JSONAPI is a specification for how APIs should structure their JSON requests and responses. It's a standard that aims to reduce cross-talk during API design by defining conventions for how resources, relationships, pagination, filtering, sorting, and error handling should work.

By following shared conventions, you can increase productivity, take advantage of generalized tooling and best practices.

jsonapi.org

It doesn't matter how strictly you adhere to the specification. What matters is that you are consistent in the portions you do follow. Ambiguity is complexity and it will consume a significant amount of engineering resources if you don't make an effort to reduce it. Have a standard and follow it.

An example of a JSONAPI formatted response is pasted below for reference.

{
  "data": {
    "type": "articles",
    "id": "1",
    "attributes": {
      "title": "My Post"
    },
    "relationships": {
      "author": {
        "data": { "type": "people", "id": "42" }
      }
    }
  },
  "included": [
    {
      "type": "people",
      "id": "42",
      "attributes": { "name": "Jane" }
    }
  ]
}

Define Your Logic

In the previous section I discussed the process of defining the communication protocol your system will use. This is a great pattern for system design because it improves communication between human stakeholders. However, some units of work are not worth communicating the details of to humans. As we've worked our way through the design process we've gradually increased the resolution at which we solve a given problem. From a plain, human-language explanation of what we're trying to accomplish to now describing our solution in a machine readable format. Its in this section we abandon communication with other humans and focus on communication with the machine.

It's rare, I think, for software engineers to categorize writing code as a stage in the design process. Programming languages are extremely precise in their description of a program and require more mental effort on behalf of the engineer than a human language, such as English, will demand. English, and other human languages, lack the specificity and feedback that a compiler can provide. Bret Victor describes the lack of feedback as "the thoughts we can not think".

[...] To be able to try ideas as you think of them. If there is any delay in that feedback loop, between thinking of something and seeing it, and building on it, then there is this whole world of ideas which will just never be. These are thoughts that we can't think.

Bret Victor - Inventing on Principle

When we write our design in code (rather than in our mind or on paper) we introduce assistants into our thinking process. The compiler, the machine, your development environment will all give you immediate feedback when your ideas conflict with what reality can actually accommodate. They will tell you: you're wrong and you need to change your approach. Seeing written code on a screen will give you a sense of the cost your architecture imposes in terms of lines of code written and in terms of the level of expertise required to understand it.

This feedback is critical because it informs our design decisions. But writing the software as a component of designing the software seems to invert the principle of design before building. However, the process of building is more significant than writing code. Ask any engineer who has finished the first "90%" in a few days and the last "10%" in a few weeks. Even so, many common behaviors programmers express when writing code will conflict with the "design first" philosophy. To accommodate this step we need to change the way we write software. We need to be able to iterate on the logic of a program with high precision, at speed, and with minimal interference from the outside world.

To accomplish this we can lean on two techniques of software engineering: dependency injection and design by contract. Dependency injections absolves us of all responsibility to interact with the outside world. We can focus purely on the logic of our program. Design by contract enables the hyper-specificity required to model the program. Consider this function:

def create_opaque_token[UserId](
    grant_type: str,
    username: str,
    password: str,
    scope: str,
    fetch_user_from_username: Callable[[str], UserLoginMeta[UserId] | None],
    store_token: Callable[[BearerTokenRow[UserId]], None],
    test_password: TestPassword = test_password,
    access_token_generator: Callable[[], str] = DEFAULT_ACCESS_TOKEN_GENERATOR,
    refresh_token_generator: Callable[[], str] = DEFAULT_REFRESH_TOKEN_GENERATOR,
    get_current_datetime: Callable[[], datetime] = DEFAULT_DATETIME_GETTER,
    access_expires_after: timedelta = DEFAULT_ACCESS_TOKEN_EXPIRES_AFTER,
    refresh_expires_after: timedelta = DEFAULT_REFRESH_TOKEN_EXPIRES_AFTER,
) -> BearerToken:
    if grant_type != "password":
        raise IdentityError(code="invalid_grant")

    user = fetch_user_from_username(username)
    if not user:
        raise IdentityError(code="token_user_not_found")
    if not test_password(password, user["hashed_password"]):
        raise IdentityError(code="token_passwords_did_not_match")

    access_token = access_token_generator()
    refresh_token = refresh_token_generator()

    iat = get_current_datetime()

    store_token(
        {
            "access_token": hashlib.sha256(access_token.encode()).digest(),
            "access_expires_at": iat + access_expires_after,
            "refresh_token": hashlib.sha256(refresh_token.encode()).digest(),
            "refresh_expires_at": iat + refresh_expires_after,
            "issued_at": iat,
            "is_revoked": False,
            "scopes": scope.split(" "),
            "user_id": user["id"],
        }
    )

    return {
        "access_token": access_token,
        "token_type": "Bearer",
        "expires_in": access_expires_after.seconds,
        "refresh_token": refresh_token,
        "scope": scope,
    }

Above I've defined a token issuance program for OAuth2. It's a fairly complex piece of logic. There are a lot of things that need to happen to ensure a user is properly authenticated and a few more things to make sure that the authentication process is secure. This complexity is inherent to the problem. We can't avoid it. During the writing of this function we'll find many places where our initial assumptions were wrong or failed to capture all the possible states this program needs to model. For example, when designing an authentication function you might conceive that a password can be incorrect but not that a user could not be found.

What's absent is the incidental complexity a problem like this might introduce. Note the absence of dependencies. There is no web-framework, there is no database, there is no cryptography library, and there is no token generation library. We don't even make an assumption about what type a user's ID would be. But the logic of the program has been implemented in its entirety.

With the program in its present state we can fully simulate the token issuance process in our test-suite and in practice. We do not need a database or any other dependency. If we want to test the logic's interaction in a client-server context then this function could be exposed on a temporary (or permanent) endpoint.

This particular program didn't call for it but what about logs, metrics, and other maintenance-critical but not design-critical properties a program can have? Your non-functional requirements are still requirements. You should write all of it. Accept a metric emitting higher-order function, a log emitting higher-order function, and so on. There's no difficulty in doing it and omitting it means returning to this program later after your mind has already context shifted. Always complete the work that's in front of you in its entirety. Here we're implementing the logic of the program so implement the logic of emitting a log and the logic of emitting a metric.

This is great but in the introduction I promised speed. Is this faster to produce than winging it? Yes, its faster. Complexity is multiplicative. Solving the inherent and incidental complexity at once is significantly more difficult than solving them separately. There are many more lines of code to keep track of, many more problems to consider, and many more context switches from logic to implementation details. Often the speed improvements from winging-it come from an accidental relaxation of the requirements stemming from cognitive overload. Not considered by the winging-it style, is the fact that it takes more effort to see a completed program and therefore you can not bail out as early if the program can not be made to work.

Okay but what about other design systems? In the absence of a specific example I can't tell you. But what I can tell you is if your design is not getting immediate feedback from a trusted, accurate source it will take you significantly longer to find those issues on your own. Type systems, borrow checkers, compilers, and other tools enhance the design process. Designing your programs in Notion forgoes over fifty years of tooling computer scientists have produced for us.

Conclusion

If you learned nothing else from this blog learn this. Resolve ambiguity early. Between yourself and others and yourself and the machine. Do not wait for ambiguity to become a problem. Squash it early.