That dynamic is shifting in an interesting way. As organizations start investing in agentic AI workflows for software development, there's a new top-down pressure to produce exactly the kind of context-rich documentation that engineering managers have been quietly advocating for years. Agentic systems need well-defined boundaries, clear interface contracts, and accurate architectural references to make good decisions autonomously. Leadership is now asking for these artifacts because AI needs them — and that's creating buy-in that was historically very difficult to generate.

The irony isn't lost on me. Human engineers have always needed this context too. The difference is that "invest in documentation" never made it past a quarterly planning meeting, while "enable our AI agents to work effectively" apparently does. I'm not complaining... I'll take the win! But it's worth naming: if your organization is finally building out these maintainability artifacts for AI, make sure your engineers benefit from them just as much.

As product-market fit is found, companies often choose to focus on growth, expanding their customer base and acquiring as many as possible within their market. This is the "scale-up" mode, and this is where HOW becomes significantly important: how to consistently deliver quality service as customers and usage volume grows exponentially.

It's the intentional tackling of the HOW problem that is the focus of "Operations". This is the work of going from ad-hoc processes and reactive firefighting to a structured, process-driven operation that handles scale efficiently. Here's what operationalizing a reliability engineering team typically involves:

• Playbooks - Creating standard operating procedures for common incidents, outages, and maintenance tasks
• Controls, Automations and self-service - Building tools that enable better control of the systems, or to automate repetitive tasks and allow other teams to help themselves
• Observability and monitoring - Implementation of metrics, logging, and alerting systems that provide visibility into system health and simplify troubleshooting
• Service Level Objectives (SLOs) - Identifying the critical user journeys, establishing clear reliability targets and tracking performance against them
• Refactoring legacy systems - Systematically identifying, updating or replacing early "quick and dirty" solutions with more scalable architectures that can handle increased load and complexity
• Incident management framework - Structured approach to handling incidents with clear roles, communication channels, and post-mortems
• Capacity planning - Regular forecasting of resource needs based on growth projections
• Knowledge management - Documentation systems that capture institutional knowledge and reduce dependency on specific team members
• Cross-training and hiring - Ensuring the team has redundancy in skills and is staffed appropriately for scale
• On-call rotation - Establishing a fair, sustainable schedule for engineers to handle off-hours incidents with clear escalation paths and support systems to prevent burnout

The term "operations" has roots in both military and business process management, where it means to convert strategic goals into day-to-day operational processes. In software development, the language really gained traction with the rise of cloud operations in the 2010s, influenced by Google's approach to running large-scale SRE teams. It boils down to putting something into operation in a systematic, repeatable way.

Operationalizing is building systems, processes, and team structures that can handle a 10x scale without requiring a 10x headcount. While vertical scaling helps with raw capacity, it doesn't address processes that won't scale, or the exponential spikes or system patterns that can lead to cascading failures and erratic behavior.

In addition to efficiencies, a focus on operations is also highly rewarding for both engineers and end-users. In both cases we're investing in predictability, meaning a better and more consistent experience with fewer surprises. For engineers in particular, this can drastically improve work-life balance and increase flow time, reducing burnout.

Of course, it's not as easy as it seems. Especially when it comes to reworking legacy tech while scaling, or identifying and tuning SLOs. These are areas often underestimated, where experienced senior and staff engineers can make a huge difference.

For my next post, I'll tackle the "HOW" of operations – very meta! By that, I mean the heuristics I use to identify priorities and sequencing. I've found mapping activities to be super-useful in surfacing certain insights, such as value-stream mapping, and cognitive load/complexity mapping, and I think it's worthwhile of a post. I might also dip into how qualitative/anecdotal data is another useful driver, like the good work being done by the folks over at getdx.com.

Before I begin, I'd like to note that I won't address the manager's responsibilities, which are frequently framed as delivering value to the organization, often by achieving operational objectives while retaining and growing reports. Instead, I'll focus on the qualities that make for a great manager within the context of a team.

Consistent Practices and Behaviours

A manager creates a stable and productive environment by establishing a repeatable set of practices and behaviours. When team members know what to expect, they can focus their energy on solving problems rather than worrying about unpredictability. Practices can (and should) evolve, but there ought to be a core stability that reinforces a psychologically safe environment for work to get done.

For example, I have a weekly team meeting with an agenda that acts as a checklist of our most important responsibilities and allows the team to discuss significant issues that may have come up. I also have 1:1 meetings every two weeks and a monthly retrospective to uncover better ways of working.

I also have a standard method for feedback (both praise and constructive) that I incorporate into my onboarding of any new direct report.

• Mark Horstman covers this topic in his book The Effective Manager and the Manager Tools Podcast.
• Will Larsen has a blog post that covers how to systematize just enough for teams to be effective: https://lethain.com/work-policy-not-exceptions/
• Claire Hughes Johnson also recently published a book on this topic, which describes the repeatable set of practices as the manager's "operating system."

Regulated and Open

Software Development can evoke contrasting opinions, perspectives, and dynamics. Managers who successfully manage their own temperaments are far more effective when collaborating with others during times of increased tension.

Managers must effectively work with people not just on their team but throughout the organization. No matter the context, an effective manager leads from a place of responsibility, curiosity, and openness. The Conscious Leadership Group refers to this place as "above the line."   Conversely, being "below the line" means being closed off, defensive, and reactive.

• The book The Fifteen Commitments of Conscious Leadership describes the qualities of conscious leadership in useful detail and provides methods for moving from "below the line" to "above the line."

Reliable and Effective

A manager is responsible for the standard of quality across their teams. This means modelling professionalism and follow-through. It doesn't mean that you need to be stuffy or uptight; it just means that you know which qualities are most important to build trust and an effective team and work to push that standard higher. Sometimes, I refer to this as "setting the tone."

For example, if I say I will do something, I do it and report back to the team. I use proper spelling, grammar, and full sentences, and I expect that folks come to meetings prepared. I use agendas for meetings, keep them on topic, and end them early instead of running down the clock.

• In his book The Art of Leadership: Small Things, Done Well, Michael Lopp highlights that managers must set the highest standard for follow-through, which is crucial in building a high-trust team.
• The central lesson of the book The Score Takes Care of Itself is that leaders who set, model, and maintain a high standard will create an environment where success is a natural outcome.
• Another fun read that covers this ground is Turn the Ship Around! by David Marquet. I liked this one, too. It goes a little deeper into how shifting language can be a powerful tool in shifting behaviours.

Ambitious

A line manager often links the CEO or VP and the staff, ensuring effective alignment of organizational goals. With clear understanding and articulation, aligning the organizational vision with the personal drivers of your direct reports is possible, which can result in a powerful source of motivation.

In addition to responding to top-down objectives, a great manager will also work from the bottom up, looking for opportunities or innovations that could generate new revenue streams or unlock untapped organizational value.

Lastly, a great manager understands that engineers love hard problems and that setting high targets is a precursor to achieving exceptional outcomes. But just as important is understanding that making unachievable commitments on someone else's behalf is a source of demotivation. In this regard, it's important to maintain awareness of the technical environment in which your team is operating and to collaborate with your team to identify ambitious targets that are also achievable.  

• The book Drive: The Surprising Truth About What Motivates Us does a great job of unpacking how different people are motivated by different factors. It's worthwhile to check in regularly with your direct reports on the most important factors and then find ways to align those drivers with operational objectives or growth goals.  

Risk-Adverse

Lastly, I don't often see this aspect come up in discussions as frequently—perhaps because it's often relegated to project management, a separate area. But from my own experience, I've noticed that a process that identifies and manages risk early enables far greater delivery and fewer surprises.

I like starting with a risk mitigation activity with my team that uncovers the assumptions, dependencies, and bets. Then, to raise our confidence, we validate assumptions and spike in the areas where we are least confident. We then move to system design diagramming and technical designs for significant areas.

If you can do this quickly and repeatably, you will often avoid working on the wrong thing for too long or seeing a project go drastically off-track. A direct report might occasionally feel unfamiliar with the "design up front" approach. They come around when they see that the entire thing can be accomplished within hours, that it generally yields better results, and that it's fun, too.

• In the book "Righting Software," Juval Löwy emphasizes the importance of addressing areas of volatility in a codebase and the critical role of project design in software architecture. In particular, he states that complex software benefits from considered design, scoping, and sequencing. The design involves defining the architecture, components and interactions; the scoping ensures the functional and nonfunctional requirements are met and communicated, and the sequencing ensures that the development workflow is optimal for the team, which I often interpret as "allowing for the maximum number of iterations on running code".
• The Pragmatic Engineer has lots of content about how engineering managers should be familiar with project management and risk management.

At this point, I'm convinced this is a book that I've been missing. Of course every leader likes to think that they're "conscious" and with a high degree of self-awareness, but this is one of those things that can be perfectly described as a blind-spot. Emotions are tricky to navigate, and I'm eager for more management tricks and tools. I'm excited to dedicate my morning time over the next couple of weeks in the hopes of learning new such tricks and tools from this book. I'll come back to this post with updates as I go.

Update: I've read this book, and it would have been a lot more eye-opening to me when I started my leadership journey. Although I appreciated the articulation and writing style, I had already come to many of the insights on my own. I will recommend this book to new managers or folks leaning in that direction.

Notes from the introduction:

• You need process, and you need it sooner than you realize
• A company will not get far without "core processes": strong management and sound operating systems.
• Companies and teams must establish a playing field where everyone participates and marks progress.

You know why playing a game is fun? Because it has rules, and you have a way to win. Picture a bunch of people showing up at an athletic field with random equipmnt and no rules. Someone is going to get hurt. You don't know how to play, you don't know how to score, and you don't know how to win.

• Research has found that people who outperform in their fields employ strategies that move them past the autonomous stage of learning, like athletes who use speed workouts to improve their performance.
• A combination of core frameworks, such as hiring and planning practices and underlying leadership principles, can help scale an organization.

Workbooks: http://press.stripe.com/scaling-people/workbooks

Consider reading: https://www.amazon.ca/Working-Backwards-Insights-Stories-Secrets

Two more books on Stripe Press that I can thoroughly recommend:

• High Growth Handbook by Elad Gil
• An Elegant Puzzle by Will Larson

Crowdbotics was founded in 2017 by Y-combinator alum and Forbes Magazine "30 under 30" Anand Kulkarni. Since then, Crowdbotics has paved the way for low-code software development, enabling anyone to generate production-grade applications in minutes using AI-assisted product requirement analysis, visual building tools and code generation.

Over 20,000 apps have been launched through Crowdbotics, including mission-critical healthcare applications, venture-backed software products earning millions in revenue, learning management platforms, and government tools.

I'll work closely with Product Management to develop and scale the architecture and engineering teams necessary to enable a growth vector in an exciting new domain.

Although Crowdbotics has a small headquarters in Berkeley, CA, it's a globally distributed remote-first company with members across all time zones. My team is primarily situated in PDT, although there are a few in EST, as well as members in Nairobi, Dubai and Nepal!

Build Versus Buy and Return on Investment

This chapter provides solid advice for those who are unfamiliar with the "not invented here" syndrome.

When considering building with open source tools, weigh the full impact of hidden costs like recruiting, hiring, and training to develop and maintain custom solutions and the opportunity costs of not delivering core business value.

Efficient Data Storage

There are many challenges when it comes to storing but especially querying observability data, which has real-time requirements on billions of rows of ultrawide events of high-dimensionality and high-cardinality data. This chapter uses Honeycomb's Retriever implementation to elucidate the various tradeoffs. Other publicly available data stores up to the challenge include Google Cloud Big Query, ClickHouse, and Apache Druid.

Cheap and Accurate Enough: Sampling

Your team is probably more concerned about traces that contain errors or poor performance. Sampling is an excellent technique for improving the signal-to-noise ratio on events you care about, drastically reducing complexity and costs when considering storage and query requirements.

Because sampling is so valuable when handling observability data at scale, it's becoming increasingly common for open-source instrumentation libraries such as OTel to provide sampling logic capabilities.

There are two different sampling strategies to use tactically:

• Head-based: The decision is made immediately and is propagated downstream via headers. Pro: reduces the overhead of collecting and storing unnecessary traces right at the source. Con: Potentially significant or anomalous traces may be missed or incomplete if only some services in a distributed system decide to sample a request.
• Tail-based: The sampling decision occurs at the end of a transaction or request. The system collects all spans related to a trace and then decides whether or not to keep it based on various criteria. Pro: All meaningful traces are retained, leading to better insights. Con: More resource-intensive; implementation is more complex.

Telemetry Management with Pipelines

More to come.

Unconstrainted complexity inevitably results in suboptimal outcomes, such as:

• A slowdown in a team's ability to accurately assess a problem or deliver a solution
• A slowdown in the time it takes to onboard new engineers
• An increase in incidents and rework
• A drop in morale or an increase in attrition

It's impossible to eliminate complexity: some problem spaces are just naturally complex, and there's no way around it. The trick for an engineering manager is to manage complexity, which means identifying, measuring, and simplifying where possible.

Talking about complexity

Complexity can be found in many forms. It's helpful to be able to identify which aspect of complexity we intend to simplify.

Aspects and measurements

• Cognitive load: characterized by an overwhelming diversity of detailed tasks that also increases context switching and few opportunities for deep work or system optimization, resulting in generally slow delivery and low morale. Measure with developer surveys, 1:1s, and onboarding metrics for new hires.
• Process complexity: characterized by too many cooks in the kitchen, too many meetings, too many required signoffs, slow decision-making, unmet requirements, and frequent change orders.
• Codebase complexity: characterized by slow builds, slow tests, reduced cycle time on code changes, reduced cycle time on code reviews, and increased rework or deployment rollback. Measure using maintainability index scoring using an index that works for the stage and engineering goals of the organization.
• System complexity: characterized by no single person knowing how the system works, no known success metrics, or low-quality metrics; quality or performance feedback often comes from end-users, and debugging or root cause analysis requires multiple people and a significant investment. Measure with build times, time it takes from code written to code in production, time for functional tests to run, rework ratios, and other DORA-inspired metrics.

Assessment and Approaches

The Cynafin framework is helpful for leaders to assess the operating context to take appropriate actions quickly. The four quadrants are Simple, Complicated, Complex, and Chaotic, where simple is characterized as apparent cause-and-effect relationships where correct answers are based on facts and easily verified, and things become much less linear from there. I won't go into the details of each quadrant's characteristics (feel free to read the link or check out the wiki page – it's excellent). But mapping complexities with this framework has often led me to the following actionable behaviours:

• Reduce Chaotic domains to Complex by drawing out a signal from noise by introducing Observability and collaborative Event Storming.
• Reduce Complex domains to Complicated by introducing abstractions, boundaries, patterns, and workflows.
• Make Complicated domains Simple by introducing or improving tooling, access to specialists, or load-shedding via a specialized team to handle inherently complex systems that are core to the organization's revenue streams.
• Eliminate Simple tasks via automation or outsourcing.

Conclusion

• Divide and Conquer: Complexity in software is typically managed by a "divide and conquer" approach, which can be applied at any level of granularity when considering software systems.
• Abstractions, Interfaces and Boundaries: introduce smaller cognitive loads with specialization and bounded contexts modelled on supporting the business's current and future revenue streams. Move teams to support the bounded contexts, and have those teams own the architecture that they depend on and document with C4 patterns.
• Refactoring to Design Patterns: Refactor complicated codebase towards named design patterns to increase understandability, maintainability, and flexibility.
• Introductions of Frameworks: If you're noticing the same sorts of use cases come up frequently, introducing a framework can make a huge impact, although it might require an up-front investment, so be prepared with a considered plan when proposing to your team.  
• Enabling Teams: A temporary enabling team to aid in strategic refactoring or short-term organizational projects such as GDPR, security improvements, etc. Enabling teams are also great for tackling technical debt or building technical surplus to unlock additional product velocity. They can also be transitioned easily to permanent developer experience teams if desired.
• Complex Subsystem Teams: Load-shed core complexity from stream-aligned product teams with the introduction of a team designed to handle a particularly complicated area.

Bonus points:

• Establish baseline performance measurements with SLAs, SLOs, and SLIs for services in production: Identifying baseline performance metrics enables a clear focus, especially when things are overwhelming. Knowing the performance and expectations of production services results in understanding the capacity for taking on new objectives and making wise decisions. For example, reducing complexity might be more critical when SLAs are unmet.
• Establish high-level goals with Objectives and Quantitative Key Results: Identify big goals and ensure they can be achieved incrementally (i.e. a mix of leading and lagging metrics enables progress can be measured throughout development and not just at the very end). Working with Product Stakeholders to identify the right success metrics early results in a much simpler development process and managed expectations.

But in retrospect, it's clearer to say that I use the GROW model. It's become widespread in tech, and it's generally the answer that folks want to hear.

GROW Model of Coaching

So what is the GROW model? It's a way of speaking that is intended to result in better nondirective coaching. Why non-directive coaching? Because most leaders naturally employ a directed style of leadership which takes place primarily through "telling" and leans heavily on authority. This style is prevalent in training a report for a specific job that is likely low-variability.

The challenge of modern teams is a fast-moving world: objectives change, technologies evolve, and it's no longer feasible to expect a manager or leader always to know the best way forward. Directed leadership also has the unfortunate property of stifling ownership and does not build organizational capacity well.

A GROW approach to coaching requires patience to learn and apply but is rewarding and energizing when used successfully. Avoiding leading questions means that the direct report always owns the problem to be solved, and the coaching method is specifically tailored to open perspective and draw out insight.

• GOAL: Asking what the person wants to achieve from the session and in the immediate future, such as "What do you want when you walk out the door that you don't have now?"
• Reality: This means asking probing questions about the current situation in detail. The trick here is to inquire about facts describing reality, such as what, who, when, and where – but explicitly not why, because it is tied to motivations and justifications, which border on judgment and can raise defences, which is counter-productive. A good question here is, "What are the key things we need to know?"
• Options: This step is to help broaden the perspective. Often people who are seeking help feel like they have limited options. By opening the floor to creative thinking, more possibilities and perspectives emerge. I learned a trick from Product Management: Diverge for Options, Converge for Decisions. A good question here is, "If you had a magic wand, what would you do?"
• Will: Asking "What will you do?" encourages detailing a specific plan. Sometimes the plan might just be to learn more about the situation, to show appreciative inquiry in learning more about a problem or someone's perspective or concerns. Sometimes, it's about preparing for a conversation or drafting a proposal. Another trick here is to ask how confident the person is in their plan and how likely they will be to act on it. Investing in helping someone through a problem, only for them not to feel confident enough to action on it, can be demotivating for all parties. My job as a people manager is to validate that my coaching serves the intended function.

The Effective Manager

This is the book I come back to more than anything. According to Mark Horstman, successful coaching is simply about asking for more. Supporting, mentoring, and coaching is not an end in itself. People managers are intended to drive value for the company. Effective managing is giving frequent feedback on performance, and effective coaching is continuously asking for higher levels of performance.

This may sound aggressive or counter-intuitive, but in a high-trust environment, it's a superpower. Too many managers feel like coaching is something to apply to low-performers, but coaching high-performers is satisfying and can yield much more.

Either way, every direct report should know their performance expectations in their role, and a regular review on the topic should occur. Not every direct will be rushing to exceed, and that's okay. They know what it is and are directly responsible for their career development. They also know that their manager will work with them to achieve their performance goals.

These are the steps for effective coaching that MT recommends:

1. Collaborate to Set a Goal: Describe a behaviour or result to achieve by a date. For example: by (four months from now), you will deliver admin features within one week on average without introducing any regressions. (This is an example of the MT goal structure DBQ – Date Behaviour Quality). MT specifies that coaching goals are long-term behavioural goals – if they can be achieved in less than four months, a frequent feedback model will suffice.  
2. Collaborate to Brainstorm Resources: There are no silver bullets. Go for volume, not accuracy.
3. Collaborate to Create a Plan: Each step in the plan contains a deadline and behaviour and is completed when the reporting happens to the manager – effectively, the plan is an accountability system that you are directly investing in for the sake of your report. The plan is only for the first few weeks – it's not worth planning four months if they can't make it past the first week.
4. The Direct Acts and Reports on the Plan: If the system has been set up correctly, we should receive regular updates in the form of task completion emails, and we then are briefly discussing the progress each week in 1:1s.

Steps 3 and 4 become iterative toward achieving the goal from step 1. If the direct report fails to accomplish a coaching task, we give the direct report negative feedback. MT gives an example: "When you miss your coaching deadlines, that's more work for later. Can you change that?". Similarly, positive feedback can be given when the direct report completes a coaching deliverable.

I love that Manager Tools willingly leverages what we know about organizational behaviour to set short deadlines on doable tasks to increase the chance of completion. Whenever possible, look for opportunities to observe the direct engaging in the behaviour we want, to provide the direct with feedback on what we observe, and make that a regular, very short-scope task. If coaching is behavioural, then the best way to achieve it is to enable and incentivize frequent practice.

The Coaching Habit

This book claims that seven questions and a coaching habit will foster team autonomy and empowerment. The central argument of this book is that successful coaching is through asking questions rather than providing answers. I found this book to be too long for its little substance, but if you are looking for a way to refresh your 1:1 questions, you can look to the seven questions for inspiration.

Applying Observability Practices in Your Team

Start with the most significant pain points, and then flesh out your instrumentation iteratively.

Observability-Driven Development

• A key finding of Accelerate: Building and Scaling High Performing Technology Organizations was that the inverse relationship between speed and quality is a myth: high-performing teams can release high-quality code quickly, and these two qualities are correlated and reinforce each other. Conversely, failures tend to happen more often for teams that move slowly and take substantially longer to recover.
• The key metric for the health and effectiveness of an engineering team is the time elapsed from when code is written to when it is in production. Every team should be tracking this metric and working to improve it.
• Isolated test-driven development does not reveal whether customers are having a good experience with your service.
• Observability should be used early in the software development life cycle, during the development process, to help catch defects earlier and reduce the cost of fixing them later. This is what is meant by "Shifting Observability Left."

Using Service-Level Objectives for Reliability

• Threshold alerting is for known unknowns only. This isn't sustainable; distributed systems' failures are inevitable and unpredictable.
• A good alert must reflect immediate user impact, be actionable, be novel, and require investigation rather than rote action.
• SLOs decouple the "what" and "why" behind incident alerting.
• SLOs are excellent at communicating how to prioritize reliability vis-a-vis with feature development. If we aren't hitting SLOs, the focus ought to be reliability.
• Two types of SLOs: time-based measures (99th percentile latency less than 300ms over each 5-minute window) and event-based measures (proportion of events that took less than 300 ms during a given rolling time window).
• For time-based: 99p as the target; for every 100 minutes, I'm allowed 1 bad minute. For event-based: for 100 events, I'm allowed one bad event.
• Use event-based because they provide a more reliable and granular way to quantify the state of a service. They are more precise. They measure brownouts better, like when more events fail but not all of them. You can more reasonably measure an SLO with event-based availability targets.
• If SLOs are not being met, but customers are also not complaining, then perhaps it's okay to reduce the SLOs, if that could enable product development elsewhere.
• If customers complain, it might be a poor leadership decision to reduce the SLOs further.

Debugging SLO-Based Alerts

• Stop relying on experience to guess what is happening in a system. It's unreliable and unsustainable.
• Observability is not specific to debugging. Debugging's concern is to remove a bug, but it says little about the overall state of a given system. Observability will tell you which systems are good candidates to improve, which may involve debugging but could also involve performance improvements, refactoring or redesigning to achieve a target SLO.

Observability and the Software Supply Chain

• Slack implemented Observability in the software supply chain, instrumenting the CI pipeline to solve complex problems throughout the CI workflow that were previously invisible or undetected.

We can’t understand a complex system if it’s a black box.

Observability aims to understand and explain your system's internal state from its outputs, ideally without adding new metrics.

Structured events are the building blocks of observability. High cardinality, high dimensionality, and context-rich events facilitate discoverability, enabling a movement away from reactive, iterative debugging to an approach where curiosity is immediately rewarded.

To answer all possible questions with metrics, all metrics would have to be captured at all levels of granularity, which is unrealistic and prohibitively expensive. You'd spend more time with metrics than with the actual software. Furthermore, domain expertise is still necessary to contextualize the metrics and make sense of them concerning the question at hand.

When emitting high-context events, we still get all metrics related to the event, but only those metrics, which is much more reasonable. We only require metrics relevant to the event's context, and the events can be compared for outliers in an existing data set. This is useful for performance analysis.

Chapter Five concludes that metrics are too low-level and isolated to serve as a building block for true software observability. They should instead be relegated to where they are efficient in monitoring low-variability infrastructure and system-level concerns.

Glossary

• Metric: a pre-aggregated measurement, as a scalar value, collected to represent system state, with optional tags used for grouping and searching.
• Structured Event: a record of everything that occurred while one particular request interacted with your service, organized and formatted as key-value pairs so it's easily searchable.
• Distributed Trace: the tracking of interrelated events that occur throughout a distributed backend, usually in the service of a single request.
• Trace Span: the segments that comprise each part of a distributed trace. These might correlate to network jumps between services, or particular areas of measurement,  are typically differentiated as root span and parent-child spans, and contain specific data used to enable the stitching of the trace: the trace ID, span ID, parent ID, Timestamp, and Duration. In addition, additional data can be added to a span as a series of tags to be leveraged in custom queries and sampling rules.

OpenTelemetry

OpenTelemetry (OTel) is a Cloud Native Computing Foundation incubating project formed by merging the OpenTracing and OpenCensus projects. This happened in 2019, so as a whole OTel is still relatively new. Despite its age, it has seen rapid adoption in the industry, with technical committees composed of representatives from Google, LighStep, Microsoft and Uber. Some benefits to using OpenTelemetry:

• Vendor-agnostic and community-supported means you only have to instrument once to send telemetry data to different products.
• Consistency in language and established semantic conventions help with alignment and ensure that everyone is on the same page.
• Ample availability of libraries with broad language support

OTel provides libraries, agents, tooling, and other things designed for capturing and managing telemetry data across your services.  

OpenTelementry Concepts

• API: OTel libraries have a specific interface that developers use to interact with the OTel system
• SDK: The concrete implementation component of OTel that tracks state and batches data for transmission
• Tracer: A component in the SDK that tracks which span is currently active in a system process. It also enables adding attributes or events to the span or modifying its state.
• Meter: A component responsible for creating instruments used for reporting measurements in your process and the ability to access and modify measurements, such as by adding or retrieving values at periodic intervals.
• Context propagation: The current inbound request contains headers that the SDK deserializes to specify the present context for the process and also serializes it to pass downstream.
• Exporter: A plug-in for the SDK that translates OTel in-memory objects into the appropriate format required by a specific destination, such as stdout, a lot file, Zipkin, Jaeger, Lightstep or Honeycomb.  
• Collector: a standalone binary process that receives telemetry data in OTLP format, processes it, and sends it to one or more configured destinations.

The next meetup is July 20th, where we cover Part III: Observability For Teams.

At a high level, it's to deliver value, keep the engineering team engaged, and retain and grow direct reports. But let's unpack this a little further.

This means ruthlessly prioritizing while simplifying complexity and promoting deep work. Tactically, an effective EM knows which architectural and project considerations can be made to increase the throughput of delivered value as the organization pivots and scales.

As a leader, the EM sets the pace for others to follow and models the organization's values. The EM also must win the team's and their peers' trust and be trusted to make wise and quick decisions when needed.

An empathic EM cultivates a high-trust environment and knows how to get the most from their team. They are practiced in delegation and coaching, and they give quick and effective feedback. They promote deep work, and they appreciate the role of developer experience in a highly effective team.  

In addition, the engineering manager is also the primary point of contact with many stakeholders across the organization, working collaboratively with product managers, other engineering managers, and domain specialists. The engineering manager presents frequent and disciplined communication with stakeholders, directors, and executives, keeping confidence high and surprises low.

How do you measure your engineering team?

Maybe more than anything else, the meat of being an effective EM is a healthy obsession with metrics – leveraging data for measurements in all areas. For example, here are just a few areas that jump to mind:

1. Establishing quantitative criteria for business objectives and the continued measurement of progress towards said goals, such as via KPIs or OKRs.
2. Service-level indicators to form service-level objectives, such as expected response times or cycle times across a range of percentiles, to know a service is performing as expected and without degradation.
3. Internal measurements correlated to team velocity, perhaps measure improvements around previously-identified bottlenecks such as code review, the time it takes to run a test suite or the number of specific incidents per week.
4. Weighted decision matrixes for capturing backlog items, technical debt, code quality and design complexity. For example, quantifying how decoupled a module is from the rest of the system or how many dependencies a given module has.
5. Individual performance measurements to facilitate promotions and growth goals for direct reports. For example, being able to draw attention to an individual's contributions in crucial areas, the quality of the contributions, and the correlated impact on the business.
6. Capacity estimates that influence architectural and infrastructure decisions. For example, being able to accurately estimate the load and utilization of a system so that it's used optimally under regular conditions while also scaling to peak conditions.

At a business objective level, I use the OKR framework to ensure that my team delivers the right value at the right time and is moving in the same direction as the organization. At this level, I tend to use a mix of leading and lagging indicators. If a team is having difficulting hitting leading indicators, the lagging indicators will be even more challenging. More importantly, leading indicators enable faster iteration.  

I work more tactically at the team level, employing a "theory of constraints" approach. Instead of specific generic engineering metrics (number of commits, DORA metrics), I look for bottlenecks, then figure out a metric that correlates with unblocking, and work toward that. For example, in my previous role, a team was bottlenecked on incidents causing rework and context switching. I identified "incidents per month" as a metric and strategized with the team for solutions, considering process change, architecture change, and additional testing. Later, the bottleneck moved to code reviews, and the focus became reducing the cycle time on a code review.

Throughout my experience, I've found using a mix of high-level and low-level objectives, and a mix of leading and lagging indicators, enables a predictability in software management that I really enjoy.

Notes that I've found interesting from the first section of the book published by O'Reilly. I'm reading it as part of the Honeycomb-hosted book club.

What's a metric?

A metric was introduced in 1988 as the foundational substrate of monitoring. It's a single number with optional tags for grouping and searching. They are cheap and straightforward, enabling tooling and optimizations around collecting, storing, shipping, and analyzing. They are easy to aggregate in time-series buckets.

What's Observability?

Originally coined in 1960, it was a characterization to describe mathematical control systems, defined as a measure of how well the internal states of a system can be inferred from knowledge of its external outputs. Extrapolate that to today, and the idea is that if structured events are emitted from throughout the system, they can be dynamically explored with granular control to give a much faster and more accurate diagnosis of the system state.

The book presents observability in software systems as "understand any system state your application may have gotten itself into, even new states you couldn't have predicted, without shipping custom code to handle it."

The pillars of observability are described as:

• Structured events
• Hypothesis-driven debugging
• Tooling that supports high cardinality, high dimensionality, and explorability

How does it differ from Monitoring?

Monitoring has been conventionally expressed as using logs, metrics, and traces to approximate overall system health. The telemetry is set up ahead of time based on assumptions of how a well-understood system will operate. As such, monitoring is well-suited for well-understood and less volatile systems, such as infrastructure-level analysis.

But with shifts towards continuous delivery and cloud native practices, software grows in complexity. New software systems can have a varied and emergent state space, contingent on invariable factors. Traditional monitoring and it's assumptions might no longer be the best tool for the job.

Debugging with monitoring requires certain assumptions, and to get value from metrics requires a level of knowledge of the system and how it is supposed to function. The upper limit on effective troubleshooting is constrained by your ability to pre-declare conditions that describe what you might be looking for. The role of intuition requires extensive experience.

Monitoring is great for infrastructure, Kafka and messaging queues. These are "known territory", where you know what to look for. Observability is ideal for "unknown territory", where you might not know what to be looking for. It's hard to find what you don't know what to look for.

What does Observability enable?

Observability through high-cardinality events enables an exploration of the "state space of system behavior" as a manner of investigation. This changes the classic troubleshooting workflow from a reactive model that is heavily reliant on institutional data, experience, and intuition, to a proactive model that rewards curiosity.

It's basically a "back-foot vs front-foot" distinction. Troubleshooting with monitoring is reactive, and requires work up front to set the conditions. Observability enables investigation in the moment, to explore conditions with existing data.

How does it work?

Structured events are key-value pairs, at an arbitrary length. The data should be high-cardinality, because this will be most useful in identifying data during debugging a system. For example, you will benefit from being able to tie data back to users, timeframes, nodes, processes, batches, etc.

Ideally the events are "wide" enough to carry all significant context variable that could influence the state space of system behavior. This can mean hundred or even thousands of key-value pairs, enabling drilling down on any combination of those keys.

Next Book Club meeting is scheduled for June 16th and will cover Part II: Fundamentals of Observability

1. The Goal: A Process of Ongoing Improvement
   Eliyahu M. Goldratt
   The classic pulpy novel about a manufacturing plant in crisis that introduces the "Theory Of Constraints" is also a great introduction to business in general, with a plethora of principles, stereotypes, and insights still relevant today.
2. The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win
   Gene Kim
   The same winning formula as "The Goal", but completely modernized for an organization headed towards a digital transformation; Introduction to DevOps and agile principles in action.
3. The Unicorn Project: A Novel about Developers, Digital Disruption, and Thriving in the Age of Data
   Gene Kim
   A follow-up to the Phoenix Project with emphasis on Developers and the role of Developer Experience in organizations achieving product success.
4. Measure What Matters: How Google, Bono, and the Gates Foundation Rock the World with OKRs
   John Doerr
   How to choose measurable objectives that drive behaviour and result in outcomes; How this system scales across an organization to ensure all teams are are working towards the same goal.
5. Drive: The Surprising Truth About What Motivates Us
   Daniel H. Pink
   How to attract and motivate knowledge workers though Purpose, Mastery and Autonomy.
6. The Five Dysfunctions of a Team: A Leadership Fable
   Patrick Lencioni
   People issues are messy; what to look for and what can be done.
7. Good To Great: Why Some Companies Make the Leap... and Others Don't.
   Jim Collins
   Where to focus, where to invest, and what to avoid; The importance of disciplined thought and behaviour.
8. Team Topologies: Organizing Business and Technology Teams for Fast Flow
   Matthew Skelton, Manuel Pais
   What sociology and science tells us are the most important factors for high-functioning teams in areas of high complexity; How to organize teams to achieve goals while adapting to changing conditions; If architecture of the system and architecture of the organization are at odds, the architecture of the organization wins;
9. Sooner Safer Happier: Antipatterns and Patterns for Business Agility
   Jonathan Smart
   Flow efficiency, Lead time and Throughput; Identifying types of work with Cynefin framework; Leveraging Lean and Agile when it makes sense.
10. Sense & Respond: How Successful Organizations Listen to Customers and Create New Products Continuously.
    Jeff Gothelf & Josh Seiden
    Introduction to concepts of Lean, Agile, and a Test-And-Learn approach; How a Sense & Respond model can be used throughout an organization.

Notable Mentions

1. Peopleware: Productive Projects and Teams
   DeMarco Tom
   Another classic book that by today's standards might appear a little less relevant than it did when it was published. Many scenarios may not appear directly applicable to modern situations. But I still love this book. The principles still stand today, and I enjoy it from the historical perspective, too.