Elevate from monitoring to observability

What is monitoring?

Monitoring involves collecting and analyzing data from your systems to ensure that they are healthy and running. It answers questions like:

• Is my server up?
• How much is the CPU usage?
• Are there any errors in the logs?

Monitoring usually revolves around three concepts:

• Metrics: Quantitative data points, such as availability and performance, include CPU usage, memory consumption, and latency.
• Logs: Textual records of every event that occurred in the system.
• Alerts: Notifications triggered based on specific thresholds that define the quality of performance.

Limitations of monitoring

While monitoring is critical, it has certain limitations:

• Reactive approach: Responds to issues but does not prevent them.
• Limited visibility: Focuses on predefined metrics, potentially missing hidden problems.
• Siloed approach: When it comes to root cause analysis, it identifies the issues in silos but does not correlate them.
• Alert fatigue: Too many alerts can lead to desensitization and missed critical issues.

Why observability?

The Rumsfeld Matrix, also known as the Johari Window, is a conceptual model used to illustrate different levels of awareness about knowledge and uncertainty. This includes four quadrants:

Known Knowns

These are the things you know, understand, and are fully aware of. They are the metrics and logs you are fully aware of and monitor regularly. They give clear visibility into system performance and point out obvious issues.

Example: CPU usage, memory consumption, and network latency.

Monitoring tools track these metrics regularly, project them on dashboards, and generate alerts for any threshold violations.

Known Unknowns

These are the things you know that you don't know and are areas where you are aware of your gaps in knowledge. You are aware that you need to monitor certain metrics but are unsure how to do so. Your monitoring tools lack the capacity to monitor them, and there could be potential threats.

Example: Monitoring third-party service dependencies where you have limited visibility.

You can address this by expanding your monitoring capabilities or integrating additional tools to fill the gaps in your current setup.

Unknown Knowns

These are the things you don't know that you know. The data and insights are available within your monitoring tool, but you don't use them effectively. These could be overlooked risks.

Example: Logs that are generated but not analyzed for trends or anomalies.

To leverage unknown knowns, you need to improve visibility and collaboration within your team with better data integration, ensuring that all relevant stakeholders have access to and understand the data being collected.

Unknown Unknowns

These are the things you don't know that you don't know. This category represents complete uncertainty and unpredictability, covering unforeseen challenges and risks that could arise. These are the areas that could be black swans.

Examples: Unexpected system behaviors or bugs that surface under rare conditions or security vulnerabilities that have not been identified.

All you need is a robust obervability strategy to address the unknown unknowns.

What is observability?

Observability goes beyond monitoring by providing a holistic view of your system's health and performance. It enables you to understand not only what is happening but also why it is happening. Observability is the ability to understand the internal state of a system by analyzing logs, metrics, and traces. It provides a holistic view of system behavior, enabling root cause analysis of issues, and is an active approach to identifying and troubleshooting problems.

While the Known Knowns and Known Unknowns can be solved with monitoring, you need observability for Unknown Knowns and Unknown Unknowns.

Observability involves the following:

• Metrics: Quantitative data points, such as availability and performance that include CPU usage, memory consumption, and latency.
• Events: Discrete occurrences that represent significant changes or actions within a system. They capture state changes and notable activities like user logins and deployment of a new application version.
• Traces: The end-to-end journey of a request through a distributed system, including the interactions between different services and components. This can be distributed tracing or transaction tracing.
• Logs: Textual records of every event that occurred in the system.