TL;DR: due to a software update error and lack of proper error checking 33 users had their dashboards exposed to one another.
There’s no other way to say this: we messed up.
What happened?
At approximately 22:42 CEST (20:42 UTC) June 21st 2022 a routine automated software update changed the server configuration in a way that rendered storage of active user sessions inaccessible. User sessions are what we use to store information about the currently logged in users.
This caused users signing in in during the interval from the update happened to when we temporarily disabled sign in to share the same session. This meant that a user visiting the site would see a partial or full dashboard belonging to one of the other users. Most things simply didn’t work.
When we noticed this around midnight CEST, we quickly revoked all active sessions and disabled sign in while investigating the incident.
Sin in was not enabled again until we were 100% confident that we had figured out what happened, why, and that we had fixed it. That happened around 01:05 CEST (23:05 UTC).
Why?
This happened because an automated routine software update changed the way the server treated the configuration, which then combined with weak error checking in the application code resulted in the same invalid session being shared between the active users.
What we are doing to fix this
We are hardening the piece of the application that handles user sessions and will add checks to ensure invalid sessions are rejected. This will prevent this from happening again if another update should fail.
This is the first time an update has caused issues like this, so we don’t expect it to happen again. But we didn’t expect it to happen this time either. This was our fault, and we’ll strive to do better.
In addition to publishing this blog post, we will also make an effort to alert the affected users directly.
This is unacceptable, and we sincerely apologize for letting you down.
