Managing time in distributed systems
A while ago I faced timezone issues in my job project... again. This was a bitter consequence of proper datetime management not being in place at the stage of system design, and now everyone suffer because of it: our users, managers and we - engineers.
When engineers face issues with datetime, they sometimes start building weird things, like avoiding the usual datetime format and sticking to objects like
I assume it happens because people don't understand what the real cause of their troubles is.
This is the reason why I once again went over the main rules of dealing with time, and finally decided to cement it in a form of an article in my blog.
So,
Types of time in your system
The first thing to understand is that in every distributed system there are always several times, such as:
- The database time - the timezone your database operates in, comes from the locale of the computer where the database is hosted.
- The pod time - this usually comes from the cluster settings and propagated to go/node/python/etc via the system locale.
- The API contract time - the datetime communicated via API.
- The browser time - this is usually the locale of the client system. The business can operate in one country, and the operator may be travelling or operating from the other country. This must be taken into account.
- The business entity time - this time depends on where the business is physically located and operates. For example, if the company operates in Argentina, the time will be of Buenos Aires.
The most challenging question to answer usually is: "If the operator and the business are located in different timezones, what time the operator should see in the dashboard? The browser time, or the business entity time?"
Datetime isn't only a timestamp
When I see a date in the UI, such as "04.07.2019", my first question would be: "This is the fourth of July 2019 where?"
The "04.07.2019" in Seoul and "04.07.2019" in Buenos Aires are two totally different things. Furthermore, at every given point of time, there may be not the same day on the planet: when it's midnight in Greenwitch, to the east of it the new day comes, while to the west it's still the previous day. This must be taken into account when, for example, a sequence of dates is generated on the server.
I like the approach the creators of Golang took: every date object is a combination of a timestamp and location:
location, _ := time.LoadLocation("America/New_York")specificTime := time.Date(2024, time.June, 15, 12, 30, 0, 0, location)
In Node there is no built-in functionality that allows setting the location, as the standard Date object operates time zone offsets, which is incorrect and error-prone (see the explanation below). That's why with Node a thid-party library must be used:
const moment = require('moment-timezone');const specificTime = moment.tz({ year: 2024, month: 5, day: 15, hour: 12, minute: 30, second: 0 }, 'America/New_York');
Don't work directly with time zone offsets
Imagine you have a timestamp, but instead of a location you want to use a time zone offset. Say, you want to calculate it for New York, you google the timezone offset and it shows you UTC-4. But wait... It's UTC-4 only in summer, during the daylight saving time. In winter it is actually UTC-5, so you need to determine whether you timestamp is in or out of the DST for New York. Ooops, smells like trouble already, doesn't it? :)
Operate always with locations, not timezones, and let the underlying library do all the dirty work.
Explicitly communicate the location in the UI
It usually helps to make it clear what location is assumed, when displaying datetime. There could be two options:
- have the location stated next to the datetime itself, e.g. "04.07.2019 10:00 in Buenos Aires",
- or, if it feels like too much, the location can be stated in the application settings.
In the second case, this location can even be a configurable setting, to add even more flexibility.
Daylight saving time and calendar generation
You ever wondered how a datepicker calendar matrix is generated? There is no rocket science behind it, because for that internal logic of the JavaScript Date object is utilised: just create a date with the day 1 and desired month and year, and add one day offset in a cycle, until the month is changed.
However, there is a gotcha some people may overlook. The Date object also holds the information about all occasions when and where the daylight saving time was ever applied. The daylight saving time is basically when there are two special day in every year: a day with 25 hours and another with 23 hours. This is applicable for the most of locations. The only location where it's not that way is Greenwitch. That is why, if someone needs a calendar, they must not iterate a local datetime object, but rather the one in the UTC. Otherwise, there could be day duplication or day skip in the matrix for certain months.
const value = new Date(Date.UTC(2019, 6, 4, 11, 0, 0));// iterate over the value here ...
Use UTC for API and database
It makes sense to keep the timezone consistent for API contracts. Every client should be able to get the datetime in a unified format and make its own conversion if necessary. For that reason, all datetime is typically presented in UTC for REST, Grpc, GraphQL and so on.
Furthermore, storing UTC datetime in a database also typically makes things easier for everyone. That is, when declaring a table, never use the DATE type, use the TIMESTAMP or TIMESTAMPTZ with values in UTC.
Constant time format
Always aim for using consistent time format everywhere in the UI. In my project we have a design flaw: data export returns datetime in the MM/DD/YYYY format, whereas the imports accept the DD/MM/YYYY format. This is such a hassle now, because I've seen what our users have to do: they download a file, then swap the day and month in every row, and then have it re-imported. This issue is so easy to fix code-wise, but so hard to do it from the user perspective, because the users have already produced certain habits, they've built tooling on-top of that bug, they've written documentation, playbooks and howto-s.
Pity.
I have covered the most common mistakes and helpful tips when dealing with time in a distributed system. I hope you've found it useful, and learned something new today. Till the next time!
Sergei Gannochenko
Golang, React, TypeScript, Docker, AWS, Jamstack.
19+ years in dev.