Production ready service for face blurring with Go, Python and React. Part 1: System design

Let's design and build a service for blurring faces like this one, with Go and React, using a model provided by Tensorflow. The service should be production ready, with ability to scale the amount of replicas of different components, and should support observability and logging.

This is going to be a good exercise that will cover a lot of things, including, but not limited to queue processing, websockets, AI, observability, etc.

# Acceptance criteria

Before committing to the system design itself, let's outline the acceptance criteria and main features.

The service should support uploading photos to a bucket and blurring human faces if detected
The list of uploaded photos must be stored on the backend for a user and shown in the UI
Processed photos must be stored next to the original ones
Shortly after the photo is processed, it must be sent back to the front-end using websockets
The deployment should be containerised, not using lambdas.
The service should support horizontal scaling of each of the components, in order to potentially handle millions of requests
There must be metrics exported to analyze
- The average task duration
- Amount of tasks executed per a day
- Amount of failed tasks

# High level design

On a very high level the system is represented as a dashboard talking to the API backend using REST.

However, as we need to accommodate the criteria listed above, we can extend the design by introducing smaller components:

Here,

The application has two tiers: Public tier (untrusted) and VPC tier (trusted).
The process of finding and blurring faces may take certain amount of time, therefore cannot be done synchronously. It must be asynchronous, which means there must be a job created and put into a process queue. For that a separate service is needed - the Worker service.
The API service provides REST endpoints for obtaining the list of uploaded images. Since the jobs are executed asynchronously, there should be a way to communicate the status update back to the client in real time. For that, Websockets could be utilised.
The Dashboard is a web application where all the uploaded images are displayed. It also allows uploading more images.
The communication between the Public and VPC tiers must be secured. There are many ways to achieve this, but one of the options could be JWT and OAuth0. In order to do that, we need a dedicated Auth service.
The files obviously may not be stored in the container file system. An external storage is needed.
The Database is used to store a list of uploaded files and the job queue. Having the database shared between several microservices is okay here, because both services belong to one domain and share the complete set of data.
The Event bus allows the API service and the Worker service communicate with each other.

# Scaling

For a cloud native application scaling is an important topic. In order for the system to be able to handle millions of requests per a second, it must:

move all heavy jobs to the background and thus delay the execution and
effectively scale horizontally, so the work load can be distributed and done faster.

So.

All Public tier communications between the services are a subject to regular Internet network lag.
The VPC tier communications are covered by the Cloud provider using low latency network, and we have only limited influence on that.
The Dashboard runs on the client side and obviously doesn't need to be scaled.
The Auth service, should a third party service be chosen, is scaled by the infrastructure of the vendor.
The Storage is managed by the Cloud provider, and it's scalability is not our primary concern.
The event bus, if not hosted on premises, doesn't require any manual measures for scaling.
The database can be scaled in two ways:
- adding read replicas,
- introducing sharding, for example, by the user id.
The Worker service we scale manually. Multiple replicas can be created, but a there is a question: "How to balance the work load between them?"
- We can introduce the partition key, and distribute the load based on its value. It can be any value that works the best.
  - The main idea here is for every replica to know exactly which range of the messages it's responsible for.
  - Each replica knows nothing about the other replicas, and unaware of their existence (unless we have a cluster, which would be over-engineering for this case).
- Another option would be to have a load balancer - an external entity that will decide which replica takes the next job.
  - The round-robin load distribution then may be used, as well as any other smart algorithm that picks the right worker for the job.
- With or without the load balancer, we should avoid overheating replicas in cases when certain partition keys are updated way more frequently than the other.
- In our simple case we can start with two replicas, so the partition key can be a serial integer and one node would serve even keys, while the other - odd keys.
- Each replica can run threads to parallelize the process even further. Typically, the amount of threads equals to the amount of CPU cores available.
The API service we must scale by setting the amount of replicas for the deployment. The Kubernetes internal load balancer will take care of balancing traffic between the replicas.

# Diving deeper

Let's dig one level deeper and see what's inside every component of the system.

# Database

First let's identify the list of entities and how they relate to each other.

✅ The Image entity holds information about images uploaded by users.

✅ The User entity holds information about users.

✅ The ProcessingQueue entity represents an entry of the processing queue. The Image entity itself can't play this role, because one image could be processed multiple times if necessary.

All datetime-s must be stored as TIMESTAMP WITH TIMEZONE and in the UTC timezone (no timezone essentially) to remove any possible ambiguity. If the backend wants to modify datetime-s internally, it must first convert the datetime-s to the business entity timezone and then back to UTC for saving.

# Dashboard

The Dashboard is quite a minimalistic SPA that provides:

A single view for a user to interact with - the home page listing all images ever uploaded, with pagination.
A widget for the file upload, using either drag-n-drop or choosing from the file picker.
An authentication widget that will allow users to sign-in.

Anonymous access isn't allowed, that's why until the user signs-in the image uploading is not possible.

As soon as an image is uploaded, the processing starts. However, it won't happen immediately. The Dashboard must keep the websocket connection open to get notified when the backend is done processing an image. When that happens, the Dashboard reloads the image with faces blurred.

# API

The API service basically implements the backend-for-frontend pattern. It implements both REST and WebSocket transport. Going WebSockets-only is technically possible, but it adds additional overhead and stretching points in the Dashboard logic, that we probably don't want to have. When a Dashboard wants to request some data, it makes use of REST endpoints, and the Dashboard also listens to the async event coming from the server.

Ideally, both REST and WebSocket response body should have equivalent structure, to make the transport layer easily interchangeable without needs for introducing drastic logic alterations.

When it comes to REST, there are two ways to have it.

Classic REST with different verbs and structure. This approach is recommended when the API is publicly available.
RPC-style endpoints. In this case only POST is used, and each endpoint bears the name of the function it executes, e.g. /book/create will execute Books.Create() method. Many fullstack frameworks benefit from this approach, as POST is rather reliable and don't suffer from limitations of other verbs such as GET or DELETE.

Since we don't build public API, the second way is preferred.

# The API contract

Let's now cover the API contract.

✅ The content type of both requests and responses shall be application/json.

✅ All datetime-s coming from or going to REST and WebSockets must be in the ISO 8601 format and be in the UTC timezone (no timezone essentially). This will help to remove ambiguity and give the Dashboard an option to choose what timezone to display to a user: the local browser timezone, the business entity timezone or something else.

✅ Every time a call happens, a random UUID is chosen by the server. This ID is returned among the response headers, and later can be used as a tracking ID to search for log entries associated with this call:

X-Operation-Id: 93ccfa49-4780-4081-8232-8e67c50a5358

✅ The API should support versioning. The version number will be included into the path as the leading element.

So, we need to support the following endpoints:

👉 POST /v1/upload/url/get gets a new signed URL for file upload.

// request:
{}

// response:
{
  version: "v1",
  url: "https://our-storage/uploadhere"
}