Blog Product

Crushing Bugs & Looking to the Future

wrote this on

Summary: We’ve made fixes to the app that address some issues that users were reporting, including updates to the new settings dashboard and folders. We also enhanced HipChat integration, and took big steps to avoid another outage like we had in February.

Hiccups with Settings

Some customers had reported issues with our new settings dashboard. The dashboard itself  appeared to be working, but the settings weren't integrated into the rest of the app correctly.  While we missed this in staging, our customers started seeing the effects immediately.

Fortunately we were able to squash the bulk of the bugs in a few days, and everything is back up and running smoothly.

HipChat Updates

After only two weeks, the new HipChat app is already #1 in our App Store. When we removed email body content from HipChat, users demanded it back. We’re happy to say that you can now select whether or not to include ticket bodies in your HipChat notifications.  

Those bodies are now easier to read too, after a few new formatting changes for HTML emails.  

Folder Count Issue

Internally, we're most excited about this fix, as it has been our most-reported bug from customers..

The elusive "Folder Count Issue" has dogged us for months, but we finally found the culprit this week.  It turned out to be caused by four actions, which had to be taken by two separate users in an exact order.

It took eight hours to debug, but the issue is now fixed.

Upcoming Infrastructure Changes

With February’s outage burning in our minds, we spent a lot of time thinking about recovering and ensuring that it won’t repeat itself.  It's not something we ever want to put our customers (or ourselves) through again, so we now have a backup plan for all our key points of failure. While next week's post will go into more detail, some of the biggest pieces:

We're moving up to nine EC2 instances in total( from five), and have completely automated provisioning our cluster from scratch. Next week we'll share the final plan, and what it took to switch over.

← Older Post Newer Post →