Crushing Bugs & Looking to the FutureAdded on Mar 3, 2014
Summary: We’ve made fixes to the app that address some issues that users were reporting, including updates to the new settings dashboard and folders. We also enhanced HipChat integration, and took big steps to avoid another outage like we had in February.
Hiccups with Settings
Some customers had reported issues with our new settings dashboard. The dashboard itself appeared to be working, but the settings weren't integrated into the rest of the app correctly. While we missed this in staging, our customers started seeing the effects immediately.
- The "My New & Open" folder would show tickets for the last user to edit it in settings
- Rules would break because they were looking for numeric ID’s instead of email addresses
- Templates for automatic responses were saved with the wrong variables names
Fortunately we were able to squash the bulk of the bugs in a few days, and everything is back up and running smoothly.
After only two weeks, the new HipChat app is already #1 in our App Store. When we removed email body content from HipChat, users demanded it back. We’re happy to say that you can now select whether or not to include ticket bodies in your HipChat notifications.
Those bodies are now easier to read too, after a few new formatting changes for HTML emails.
Folder Count Issue
Internally, we're most excited about this fix, as it has been our most-reported bug from customers..
The elusive "Folder Count Issue" has dogged us for months, but we finally found the culprit this week. It turned out to be caused by four actions, which had to be taken by two separate users in an exact order.
It took eight hours to debug, but the issue is now fixed.
Upcoming Infrastructure Changes
With February’s outage burning in our minds, we spent a lot of time thinking about recovering and ensuring that it won’t repeat itself. It's not something we ever want to put our customers (or ourselves) through again, so we now have a backup plan for all our key points of failure. While next week's post will go into more detail, some of the biggest pieces:
- Dedicated resources for outgoing email
- Backup database server ready to takeover at a moment's notice
- Backup chat server so your customers are never left hanging
- Splitting resources between Amazon availability zones in case of system wide outages
We're moving up to nine EC2 instances in total( from five), and have completely automated provisioning our cluster from scratch. Next week we'll share the final plan, and what it took to switch over.
Today’s Outage - What Happened,
Feb 21, 2014
What We Did, and What We’re Doing
As many of you are undoubtedly aware, Groove experienced a serious server issue last night and this morning that caused the app to go down for all users.
We’ve said it a number of times today, but it bears repeating: while the outage was out of our control, we were unprepared to quickly and effectively deal with it, and the ultimate responsibility for that lies with us.
We undermined the trust that our customers have in us, and for that, we can’t apologize enough. We’ve recommitted ourselves to regaining and maintaining your trust by taking steps to ensure that nothing like this ever happens again.
Now that we’ve restored our servers and brought the app back to full functionality, we’d like to offer a rundown of the technical details behind what happened.
This morning at 8:51AM EST we received a notice from Engine Yard that one of our cloud servers is scheduled to be retired by Amazon on Feb 25th. To our shock, Amazon retired it late last night with no previous warning.
The server in question was our master database instance. Because of this, the entire cluster had to be shut down and recreated, and previous manual tweaks to the configuration had to be located and recreated as well.
Some of the issues we ran into while rebuilding the cluster:
- Hardcoded ec2 hostnames prevented server instances from talking to each other
- SSL certificates for the server running Live Chat and real-time in-app updates were lost, preventing it from running
- Lost link to search configuration
Additionally, our previous public IP address was lost and reused by Amazon, so we had to acquire a new IP address from Amazon and update our DNS records.
One of the biggest fails on our end which delayed the resolution was related to monitoring. We have server monitoring set up, but because the server failed during the night, our team didn’t see the alert emails until they woke up in the morning. We’ve already switched our systems to call our personal cell phones in the event of an outage, 24 hours a day.
As much as today drained our team as we scrambled to put out this fire, it was worse for the customers who rely on us to provide a stable app. And for that, again, we apologize.
There’s a lot more to say about the events that occurred today, and we’ll be doing that on our blog next week. But in the meantime, we wanted to let you know exactly what happened from a technical standpoint.
In the meantime, we’re grateful for your patience, your words of support and understanding throughout today, and most of all, your trust in Groove.
An Update HipChat Users Will LoveAdded on Feb 19, 2014
A few months ago, we ran a survey to learn more about what our customers wanted from Groove.
One question asked: Which apps would you like to see Groove integrate with?
While the question allowed for multiple responses, we were blown away to see that 65% of respondents said HipChat. That was the first-place winner by a factor of three.
And in our one-on-one conversations with many of you, we’ve heard the same.
You use HipChat. You love it. And you’d love to use it with Groove.
Starting Today, You Can
If you didn’t know, HipChat is an awesome real-time group chat and instant messaging app. You can have group conversations, one-on-one chats, share files and a whole lot more. It's available via a web app, desktop client and native mobile apps. All in all, it's a really fantastic way for small businesses to communicate. We use it all day at Groove.
Our new integration bridges your Groove account with HipChat, giving you a seamless connection between customer support and your team’s favorite communication app.
By enabling the free HipChat integration from the Groove App Store, you’ll be able to get notified in a HipChat room whenever there’s activity you need to know about on Groove.
It’s totally customizable, and you can pick which mailboxes and events you want to get notifications for:
Steve at StatusPage.io, one of our early testers, noted:
"Integrating our Groove account with HipChat helps us respond to support requests almost immediately. This was an easy win for us in the Customer Happiness department."
The whole process is faster and easier, for you and for your customers.
And of course, social channels are integrated too, so if you’ve got Groove set up with your Twitter account, you’ll see new Tweets right in your HipChat window:
Give the new HipChat integration a try, and let us know what you think!
Improved SettingsAdded on Feb 13, 2014
Since launching Groove, our product enhancements have focused on the ways in which you interact with your customers: organization, automation and communication.
Today, we’re excited to announce an update that’s a little bit different, but every bit as valuable.
Until now, your account settings had to be updated in a pop-up modal that forced you to navigate through tabs, and wasn’t always the most intuitive.
And while it always worked, we had the opportunity to make it better, faster and simpler.
So that’s what we did.
Today when you log in to Groove and click on Settings, you’ll see your new Settings Dashboard:
Everything, from your profile and preferences, from billing info to automation settings is in one easy-to-navigate place.
We added categories and icons to help you find what you’re looking for without digging, and sidebar navigation to help you switch between settings pages without using your back button:
Getting rid of the popup gave us more room to make some of the in-app actions easier, and you’ll also notice an improved interface for things like creating labels and rules:
We hope you’ll enjoy these changes as much as our testers have, and as always, we’d love to hear what you think.
Send a Tweet to @GrooveHQ with your feedback!