Tweet
Bug fix: the long server freezes (and a couple of client crashes)
First, the good news: the server stayed up for five days with no crashes. The server never fell over.
But some of you hit something worse than lag, and we found it.
What we found
From around 8 October the game world would occasionally freeze solid for 30 seconds to 2 minutes. Monsters stopped, chat stopped, and some of you were disconnected. It happened about 60 times, mostly in clusters:
(All times UK.)
The cause was two database lookups that were far slower than they should have been:
Both were reading the entire items table every time instead of jumping straight to the rows they needed. If that happened while the server was saving, the two got stuck waiting on each other, and the whole game loop waited with them until the database gave up 30 seconds later.
What we fixed
The client fixes arrive through the patcher; just open the launcher as normal.
If you were caught in one of the freezes
First, the good news: the server stayed up for five days with no crashes. The server never fell over.
But some of you hit something worse than lag, and we found it.
What we found
From around 8 October the game world would occasionally freeze solid for 30 seconds to 2 minutes. Monsters stopped, chat stopped, and some of you were disconnected. It happened about 60 times, mostly in clusters:
- 10 Oct, about 19:13 – 19:22
- 11 Oct, about 03:23 – 05:23, 09:39 – 10:13 and 19:04 – 19:50
- 12 Oct, about 12:44 – 14:14 and 15:59 – 16:41
- 13 Oct, about 02:25 – 03:05
(All times UK.)
The cause was two database lookups that were far slower than they should have been:
- The mount upkeep check, which looks at what is stored on each mount
- Loading a character's items when someone logs in
Both were reading the entire items table every time instead of jumping straight to the rows they needed. If that happened while the server was saving, the two got stuck waiting on each other, and the whole game loop waited with them until the database gave up 30 seconds later.
What we fixed
- Database: added the missing indexes, so both lookups now go straight to the right items. This went live at 09:08 this morning. Since then the longest hitch has been 1.4 seconds, down from 1–2 minutes.
- Server: the mount upkeep check now gives up after 5 seconds instead of 30, so even if something like this ever comes back it cannot hold the server hostage. The server was restarted with this at 10:26 today.
- Client crash: Ctrl-dragging an item onto another to combine them, then clicking Yes after the selection had been cleared, crashed the game. Fixed.
- Client logging: one error that sometimes appears while entering the world now records exactly what failed, so we can track it down. You won't notice anything, but it helps us.
- Patcher: a half-finished download left on the patch server could show "Patch failed". The patcher now ignores leftover partial files.
- Microsoft Store version: it was skipping the patcher and going straight to login, so Store players never got updates. The new Store build opens the patcher like the website version. It is waiting for Microsoft to approve it. Until then, use the website download if you want the latest build.
The client fixes arrive through the patcher; just open the launcher as normal.
If you were caught in one of the freezes