A Note from Jenn Chan

A Note from Jenn Chan
Posted by: JChan on 06-09-2020 06:30 PM

Hail Adventurers,

(TL;DR version at the bottom.)

I'm writing to you today to apologize for all the issues that we've been experiencing with EQII these past few months. We've been moving at a breakneck speed to grow into our newly formed Darkpaw studio and ensure that Norrath is around for many more years to come. Unfortunately, in our haste our "raid wiped". We've failed ourselves and we've sadly failed you, our dear friends, but we've reviewed our parses and we have new strats.

The following are not excuses. We know that we have lost some trust and that we need to work to regain that. As some of you have noticed, we're trying to be much more transparent and in doing that, let me give you some information about a few issues that we have encountered in the past few months so we can move forward together on the path to great things.

First off, I'd like to address the unexpected down time in the last week of May. There was a major outage that affected all of our servers, our forums, and numerous other systems in our infrastructure. In accommodating our growing population and addressing current bug issues, we have been incorporating some new hardware. Unfortunately, the transition with the new hardware did not go as smoothly as planned, which brought on many unforeseen difficulties that we have overcome and a few we are still battling. This took out most of our ability to diagnose the core issue and also took out our ability to use our regular channels to communicate the outage. Once the issue was found and resolved, we focused our efforts to bring up the game servers at the time cost of fully bringing up our diagnostic tools. By not having these internal tools up, we failed to see another growing issue that later caused continued login issues throughout the rest of the night. We have taken measures to ensure the initial issue will not happen again.

In the matter of our recent individual world stability and lag issues, we are currently in the midst of refreshing a number of our systems and are working with our partners to build out more regular maintenance plans so that issues like these are minimized in the future. Earlier this year we had completed a number of migrations ranging from as large as total data center moves to as small as replacing memory on a number of our systems. These conversions and moves are critical to the future health of our infrastructure and unfortunately some of these migrations caused unexpected outages. Specifically, for performance, we have already migrated our database to a faster solution, improved the way we spread out loads, and upgraded some of the underlying operating environments. Continuing our lag mitigation plans, we are working on upgrading the underlying tool set and are identifying areas where we can upgrade the hardware. Additionally, we have been adding more diagnostic systems to better analyze where our actual bottlenecks are. We've also pulled earlier into the schedule the adoption of some performance profiling tools that were originally set for late this year / early next year. As you can see here, we have completed a number of lag mitigation projects and still have a number of projects that are going to take some time to complete.

Regarding our overall quality levels, this has been a regular battle that we've had to fight many times over. Thank you to the players that have so graciously volunteered their time to assist us over the years on our test server, in betas, and in reporting issues on our forums and in /bug. With the conversion to the bimonthly update schedule we hoped to stem the more reactionary style changes and get the time to put in more complete fixes to issues. We're still learning how to work at that cadence and will be implementing more changes internally that can better take advantage of the higher thresholds for release. We are also looking into ways we can get content, performance improvements, etc. onto the Test Server earlier and reduce changes going to live in a broken state. This is in order to keep quality at the high standards we as developers and you as players expect, while also adhering to the new work environment of social distancing or working from home. Many of our hardworking developers and support staff (ex: QA, Operations, Platform) have been voluntarily working an increased number of hours to make sure our schedules don?t slip and that most bugs don't reach live. These are strange times and we are prone to human error. Frankly put, we've been making silly mistakes that we wouldn't have been making a matter of months ago. Not to fret, as I am confident in my team who have been very agile in doing everything they can to fix problems as they arise.

So going forward, we need time to work on the solutions that I've touched on above. I'm asking you to continue to have patience with us and to continue to work with us in these trying times. We're going to complete and release some projects that we feel are almost done and pause for a bit to catch our breath and analyze the pain points from a holistic view. I've asked the team to double down on our current efforts in stability, performance, and quality. As a result, some of you may notice a lack of new things in-game for a bit while we focus on these issues. This doesn't mean that our normally scheduled annual plans at the end of the year are at risk, just that there will be a pause for a bit. In the meantime, please accept our apology and this Samenda Honorhoof (mount) which you can get by logging into the game between June 9 at 12 PM PDT to June 24 at 12 PM PDT.. I believe that with your help we can make it out of these challenging times and can once again enter a mythical era.

Never give up!
Jenn Chan

TL;DR:
I apologize for the issues that we've been experiencing. We've encountered large and unexpected problems and we've fixed some of them. We need more time to work on fixing the rest. Please be patient. Things will get better. We're giving you a mount. Never give up!

View on EQ2players.com