Cloud Native Monitoring

Neel Shah, VP Software Development
June 6, 2024

This blog marks the conclusion of our series on the AICPA Exams Team’s transition to the Azure cloud. Throughout this series, we have outlined our strategic approach to modernizing our core systems using service-oriented architecture principles and transitioning legacy applications to the cloud. We have detailed our carefully planned migration strategies, discussed the deliberate selection of databases, and explored enhancements in data warehousing and business intelligence. Additionally, we examined the crucial role of Platform-as-a-Service (PaaS) and serverless technologies, such as Azure Functions, Azure App Service, Azure Cognitive Search, and Azure Service Bus, in supporting our distributed system.

In this final installment, we turn our focus to Application Performance Monitoring (APM) and message workflow monitoring. These essential components ensure our newly migrated applications perform optimally in the cloud. We will explore how APM and message workflow monitoring have been seamlessly integrated into the AICPA Exam Team’s cloud environment, the insights they have provided, and their ongoing roles in tuning and maintaining system performance.

What is Application Performance Monitoring (APM)?

Application Performance Monitoring (APM) involves employing software tools and gathering telemetry data to oversee and optimize the performance of critical business applications. By deploying APM tools, enterprises gain real-time insights into how their applications perform, ensuring that service levels are maintained and customer experiences remain positive. This technology empowers IT, DevOps, and site reliability engineering teams to rapidly identify and fix any issues within applications.

Key Terms Explained

• Telemetry: The automatic collection and transmission of data from remote or inaccessible sources to an IT system for monitoring and analysis.
• Distributed Tracing: A method used to track requests as they travel across various services in a microservices architecture. This helps in understanding the interaction and dependencies between services.
• Synthetic Traffic: Simulated user traffic generated to test and monitor the performance of an application under controlled conditions.

Core Functions of APM

APM is multifaceted, involving several critical functions that ensure applications perform optimally and reliably:

• Performance Monitoring: Continuously tracks application performance to ensure they meet predefined thresholds and standards, assessing metrics such as response times, system throughput, and transaction volumes.
• Anomaly Detection: Utilizes baseline performance metrics to detect anomalies or performance outliers, crucial for preemptive problem-solving and maintaining system stability.
• Root Cause Analysis: Provides detailed diagnostics to pinpoint the exact sources of performance issues, whether they originate within the code, infrastructure, or external dependencies.
• Capacity Planning: APM data aids in understanding the resource demands of applications, facilitating effective capacity planning to ensure infrastructure scales appropriately without resource wastage.
• User Experience Monitoring: Assesses technical metrics and end-user experiences, tracking user interactions and the impact of application performance on user satisfaction and engagement.

Importance of APM in Modern IT Infrastructure

The role of APM in modern IT infrastructure is critical, driven by the following needs:

• Enhanced User Experience: In the competitive landscape of digital services, user experience is paramount. APM ensures applications are responsive and reliable, directly influencing customer satisfaction and retention.
• Operational Efficiency: By automating performance monitoring and issue detection, APM reduces the need for manual oversight, allowing IT teams to focus on strategic initiatives rather than firefighting.
• Cost Management: Efficient resource utilization and capacity planning reduce operational costs. APM helps in optimizing infrastructure expenditure, thus ensuring financial efficiency.
• Business Continuity: With continuous monitoring and quick troubleshooting capabilities, APM minimizes downtime and ensures business operations are not adversely affected by unforeseen IT disruptions.
• Compliance and SLA Adherence: APM tools monitor applications to ensure they meet compliance standards and the performance metrics outlined in Service Level Agreements (SLAs), maintaining customer trust and contractual obligations.
• Enhanced Product Development: By monitoring and analyzing synthetic traffic in test environments, APM uncovers limitations and identifies errors, allowing development teams to resolve potential issues before applications go live.
• Support for Distributed Systems: APM manages complex interactions in microservices or distributed architectures, ensuring that system-wide performance meets expectations.
• Facilitating Business Collaboration: APM improves communication and collaboration across business units by sharing metrics and analyses, which helps break down silos and boost productivity.

Key Components of a Robust APM System

Here are the key components that make up the foundation of a robust APM system:

• Data Collection: Uses agents installed in applications or on host machines, instrumentation for in-depth performance data, and APIs to collect metrics from non-instrumented components.
• Metrics and Telemetry: Collects essential performance metrics such as CPU usage, memory, disk I/O, and network latency, alongside broader telemetry data from applications and infrastructure.
• Event Logs: Records detailed events within an application, including system calls, user actions, errors, and transactions, which are essential for post-occurrence diagnostics.
• Tracing: Includes distributed tracing for tracking requests across services in microservices architectures and code-level tracing to pinpoint issues down to specific lines of code.
• Error Tracking and Crash Reports: Automatically detects, logs, and alerts teams to application errors and crashes, providing critical details like stack traces.
• Analytics and Visualization: Features dashboards for real-time performance monitoring and reporting tools for analyzing historical data to identify trends and improvement areas.
• Alerting and Notification Systems: Employs automated alerts to notify IT staff about performance anomalies or failures based on set thresholds.
• Root Cause Analysis: Utilizes tools and methodologies, often enhanced by AI and ML, to determine the underlying causes of performance issues.
• Capacity Planning and Forecasting: Analyzes historical performance data to predict future resource needs, aiding efficient infrastructure planning.
• Integration with Other IT Management Tools: Ensures seamless operation with other IT management tools like incident management systems and CI/CD pipelines for a holistic approach to IT operations.

APM on Azure

Azure equips users with robust APM tools like Azure Monitor and Application Insights, designed to effectively collect, analyze, and manage telemetry data across your applications and infrastructure. Central to Azure’s APM approach is a structured data collection, aggregation, analysis, visualization, and automation process. This section outlines each process step, showcasing how Azure’s technologies enhance APM capabilities.

Diagram Source: Microsoft, Azure Monitor data platform

Data Collection

Data collection is the first step in the APM process, involving several key Azure components:

• Application Insights Agents: SDKs installed into your application codebase that automatically capture performance metrics, exceptions, and usage data. They support various programming environments like .NET, Java, JavaScript, and more.
• Azure Monitor Agents: For broader system metrics and log data, Azure Monitor agents are deployed on virtual machines and containers. They capture system-level metrics such as CPU usage, memory usage, network I/O, and more.
• Diagnostics Settings: Azure services have built-in diagnostic capabilities that can be configured to send logs and metrics to Azure Monitor. This includes data from Azure SQL Database, Azure Blob Storage, and other services.

Data Aggregation and Storage

Once data is collected, it needs to be aggregated and stored effectively:

• Azure Monitor Metrics Database: This is a fully managed, time-series database optimized for fast retrieval of monitoring data. It stores performance metrics collected by both Application Insights and Azure Monitor.
• Log Analytics Workspace: This is where logs are aggregated and stored. It provides a query engine and data exploration tools to analyze large volumes of log data across different sources.

Data Analysis and Visualization

The core value of APM is derived from analyzing the collected data:

• Application Insights Analytics: Provides powerful querying capabilities to explore and analyze telemetry data. It uses the Kusto Query Language (KQL), which is designed for fast data exploration and complex analytics.
• Azure Dashboards: Allows users to create custom visualizations using data from the Metrics Database and Log Analytics. These dashboards can be shared across teams and offer real-time operational insights.

Alerting and Automation

Azure Monitor and Application Insights can automatically trigger alerts based on predefined conditions or anomalies detected in the data. These alerts can, in turn, trigger automated actions like scaling resources or invoking Azure Functions to handle complex tasks.

Benefits Realized with Azure APM

• Seamless Integration with Azure Services: Azure Application Insights integrates deeply with various Azure services, enabling comprehensive monitoring of applications, virtual machines, databases, and more. This integration provides a holistic view of the entire ecosystem’s health. For instance, Azure Service Health provided real-time insights into the health of our services.
• Distributed Tracing: A major advantage we experienced with Azure Application Insights was its distributed tracing and the different dashboards that made finding issues across our complex microservices architecture easier. By leveraging distributed tracing, we could trace requests across various services and quickly identify performance bottlenecks. The dashboards provided a comprehensive view of the system, allowing us to drill down into specific issues for efficient resolution.
• Real User Monitoring (RUM): Real User Monitoring (RUM) has been instrumental in detecting trends underlying performance issues and conducting root cause analysis. This capability allows us to analyze real user interactions and derive actionable insights. RUM helped us detect patterns of slow response times for specific user interactions. Analyzing this data enabled us to identify and address underlying issues, enhancing the overall user experience.
• Advanced Data Visualization: Azure Application Insights offers powerful data visualization tools, including interactive charts, graphs, and customizable dashboards. These visuals enable developers and stakeholders to gain insights quickly and make informed decisions without sifting through raw data. Interactive dashboards created using Power BI provided comprehensive views of system performance, facilitating data-driven decisions and continuous improvement.
• Intelligent Analytics and Diagnostics: Azure Application Insights provides intelligent analytics and diagnostics capabilities, allowing developers to efficiently identify performance issues, exceptions, and failures. Using Application Insights’ analytics, we could preemptively resolve potential performance issues, reducing downtime and improving system reliability.

Challenges to Using Azure APM

• Signal-to-Noise Ratio: While Azure APM offers extensive monitoring capabilities, the volume of telemetry data can be overwhelming, making it challenging to identify actionable insights. Configuring dashboards and alerts highlighting significant deviations from expected performance helped reduce noise and focus on actionable data. However, finding the right balance to filter out unnecessary alerts without missing critical issues required continuous fine-tuning.
• Cost Management: Azure APM can become costly, particularly as usage scales or if detailed telemetry is needed. Managing costs while maintaining essential monitoring capabilities requires careful planning. Setting up proper filters and thresholds allowed us to control the volume of telemetry data collected, optimizing costs. Despite this, we had to monitor quotas closely to avoid unexpected expenses.
• Complexity in Setup and Management: Azure APM’s comprehensive features are powerful but can also be complex to configure and manage, even for teams with significant Azure expertise. The breadth of features requires thoughtful configuration to ensure they meet specific needs and integrate smoothly with existing workflows. To fully leverage Azure APM, our IT staff utilized extensive training, ultimately leading to smoother operations and more effective monitoring.
• Dependency on Azure Environment: Azure APM’s deep integration works best when fully embedded within the Azure ecosystem, which might not suit all business models. Our full commitment to Azure allowed us to maximize the benefits of Azure APM without compatibility issues, streamlining operations.

Choosing the Right APM Solution

Azure offers powerful native APM tools like Application Insights and Azure Monitor. Still, third-party solutions such as NewRelic, DataDog, and Dynatrace also provide unique features that may better meet specific requirements.

When integrating an APM platform, consider the following:

• Stakeholder Buy-In: Ensure all teams understand and support the adoption of APM.
• Feature Set: Choose an APM tool that aligns with your application needs and supports your technology stack.
• Budgeting: Account for the computing resources needed for extensive monitoring alongside the cost of the APM tools.
• Ownership and Maintenance: Define which teams will be responsible for the APM’s oversight and upkeep.

Monitoring Messaging Workflows

NServiceBus acts as the backbone of our messaging infrastructure, facilitating communication between microservices with features like automatic retries, message forwarding, and error handling. Given the critical nature of our applications, ensuring the health and performance of NServiceBus is vital.

• ServicePulse: For real-time monitoring, we rely on ServicePulse, which provides real-time alerts about issues with message processing. For instance, ServicePulse alerted us during our migration to a spike in failed messages due to a configuration error, enabling us to resolve the issue and minimize disruption promptly. The dashboard views in ServicePulse offer a detailed look at the health of the messaging system, including metrics like message throughput and error rates.
• ServiceInsight: ServiceInsight complements ServicePulse by offering advanced debugging and diagnostic capabilities. It visualizes the flow of messages through the system, helping us trace and diagnose issues effectively. For example, during a critical update, ServiceInsight helped trace failed messages to a service incorrectly handling certain message types, enabling a swift fix and minimal downtime.

Conclusion

The AICPA Exams Team’s internal systems transition to the Azure cloud has been a complex but rewarding journey. Application Performance Monitoring (APM) and message workflow monitoring have been crucial in ensuring the success of this migration. By leveraging tools like Azure Monitor, Application Insights, and NServiceBus monitoring utilities, we have not only maintained operational excellence but also set a foundation for future improvements.

As we move forward, continuous improvement through effective APM and workflow monitoring remains a priority. Our systems are performing well today and poised for future enhancements, ensuring that we continue to meet and exceed the expectations of our users and stakeholders.

About the Author

Neel Shah is a seasoned technology leader passionate about driving innovation and delivering exceptional software solutions. As the Vice President of Software Development at Strasz Assessment Systems, Neel brings a wealth of experience in leading high-performing teams, shaping strategic initiatives, and fostering a culture of excellence. With more than two decades immersed in the technology sector, Neel has developed a deep understanding of distributed systems and Service-Oriented Architecture. This expertise enables him to spearhead the creation of scalable, resilient software solutions tailored to meet the dynamic needs of clients and stakeholders, underpinned by pragmatic technology choices. By fostering a culture of innovation and continuous improvement, Neel empowers teams to push the boundaries of what’s possible and deliver impactful results that drive business growth. In his leisure time, Neel enjoys quality moments with his wife and two boys, playing cricket, watching movies, and staying updated on the latest developments in science and technology.