When enterprise architects consider mobile application integration, the focus is typically on the link between...
the cloud and the consumer. A new generation of mobile DevOps integration tools promises to improve the connection between application infrastructure, operations personnel, QA and developers. This promises to streamline the application development lifecycle, improve velocity, and reduce the duration and severity of bugs and security vulnerabilities, said Hav Mustamandy, director of systems operations at eHarmony.
Leading mobile DevOps integration tools services are provided by PagerDuty, BigPanda and VictorOps. These tools make it easier to create customized monitoring views of any slice of the IT infrastructure by application, teams, cloud, customer, data center or other logical groupings of the IT environment. As a result, an enterprise can respond to and resolve critical issues faster. "It's like a spam filter for IT alerts," said Assaf Resnick, CEO of BigPanda. "By filtering out irrelevant noise, teams can spot and resolve critical issues 90% faster."
Furthermore, these tools make it easier to maintain an authoritative system of record that can make it easier for less skilled staff to resolve similar issues when they recur, said Todd Vernon, CEO of VictorOps.
These tools orchestrate development information such that information can be quickly passed along via a text message, mobile, app or Web client to IT staff with a particular expertise. Application development teams see only alerts that are relevant to their applications. DBA teams are exposed only to database-related alerts across clouds and data centers.
Sifting the wheat from the chaff
Enterprises may use multiple monitoring tools for applications, servers, databases and networks. This results in a hodgepodge of alerts. Many of these tools contain the ability to send text messages to mobile devices, but no intelligence on who is the best person to resolve a problem.
As a result the IT team may get thousands of daily IT alerts and must hire dedicated operations personnel who end up being glorified gatekeepers for identifying the best person to solve a particular problem. IT teams can waste critical time attempting to identify which alerts belong to which environment, determining their impact and deciding which action needs to be taken. This is a time-consuming process that can slow detection and recovery times.
These new mobile DevOps integration tools use machine learning, pattern analysis and knowledge management techniques to pass along the appropriate information. They also can integrate with other chat applications to create a system of record for tracking mobile alerts.
Keeping love on track
The dating site eHarmony has been working with PagerDuty to streamline the alerting process for its private cloud infrastructure. During peak usage, eHarmony and other data sites experience a 25% to 30% increase in activity between December and Valentine's Day.
Working with the PagerDuty alerting service allowed eHarmony to decrease incident resolution time by 80% and mean time to alert by 56% during this critical window, said eHarmony's Mustamandy. "Millions of people rely on eHarmony's online compatibility matching system, and any downtime that occurs can result in a decrease in revenue and customer loyalty," he noted.
EHarmony realized it was more cost-effective to rely on an outside alerting consolidation service to automate all incident resolution processes and decrease resolution downtime. The use of PagerDuty's advanced analytics capabilities also helped eHarmony gain a better understanding of problems so that it could restructure response processes and improve IT staff training to resolve problems more efficiently.
Architect an efficient alerting flow
Mustamandy also said that it is good practice to leverage this operations incident data to improve operations processes and development. EHarmony uses the PagerDuty service to set up groups, and the tool allows them to summarize the number of different types of alerts sent to each group, as well as the types of issues that occur. He said this makes it easy to see how many alerts go to each group so that eHarmony can decide what its DevOps teams need to focus on.
Another good practice is to look at ways of moving beyond email and text messages by themselves, which make it difficult to integrate in a coherent way. Mustamandy said, "The biggest issue with [Short Message Service] was that many of the applications we were monitoring had no direct SMS integration. In addition, if we just relied on SMS, it was difficult to focus on the reporting and identify key metrics to improve upon."
EHarmony has been experimenting with integrating Shinken for distributed monitoring combined with Nagios as the main collection point for data. Shinken is a monitoring aggregation point, but it does not have a built-in paging system. In contrast, eHarmony's current solution sends SMS texts, provides more comprehensive alerting information via an app, and then stores a record of issues for subsequent analysis. Having a dedicated mobile DevOps integration tool like PagerDuty helps streamline the collection and distribution of IT data more efficiently than Shinken and Nagios, said Mustamandy.
Automate the fixes when possible
Going forward, eHarmony would like to further automate the issue resolution process. Mustamandy recommends that enterprises consider leveraging cloud platforms like OpenStack for automatically spinning up new virtual machines. He believes that the right integration strategy would allow continuous integration servers like Puppet and Chef to automatically resolve issues without having to bother a live person at three in the morning.
"If someone can't act on a problem, there is no need to alert them," Mustamandy said. "But the enterprise should still log these issues so that underlying problems can be resolved during working hours. At the end of the day this will lead to happier programmers and Ops teams and more reliable enterprise architectures."
DevOps and the cloud-oriented world
How to establish a DevOps team structure
Learn about the connection between AWS, DevOps