Free ThinBook/Whitepaper on Feature Toggles, by John Kodumal founder of LaunchDarkly (feature mgmt app https://learn.launchdarkly.com/effective-feature-management/

Excerpts

Introduction

Software development professionals need to recognize this shift and appreciate that achieving true continuous integration and continuous delivery (CI/CD) puts software development teams in a fundamentally different world. This new world brings with it enormous benefits but also new challenges. Delivering more changes more frequently can also introduce increased chaos and risk, especially when our toolkits for controlling risk have not adapted as quickly to this new world

a world in which dozens of changes are released each day

In this book, I define a new category of software called feature management and discuss the underlying technologies, use cases, and best practices for adoption.

feature management service

CHAPTER 1 Feature Management

What Is a Feature Flag?

In the early days of software development, this might have been a command-line argument, an undocumented variable in a configuration file, or a hidden value in a registry

There are two parts to the seemingly simple enableFeature call highlighted in the preceding example that make it special

The second parameter to the function (elided for simplicity in our code snippet) is a context. It typically identifies a specific user, but it can contain any arbitrary data. This is important because it makes the behavior of the function context sensitive.

Second, the enableFeature function calls out to the feature management platform, which takes the context and a set of rules defined in the feature management system to determine which code path is executed.

Feature flags can either be temporary or permanent. Temporary flags are often used to safely deploy changes to your application or to test new behaviors against old ones. After a new behavior is being used by 100% of your users, the flag is intended to be removed. Permanent flags give you a way to control the behavior of your application at any time. You might use a permanent flag to create a killswitch or to reveal functionality only to specific users.

Ship When You’re Ready

Deploy != Release

Feature flags allow your team to create an agreed-upon process for deployment with success criteria that you can define at each stage. The flags give you the ability to move forward, or backward, based on the success at each critical decision point.

The software industry has moved toward distributed microservices and an ever-expanding roster of external service providers

Unlike 10 years ago, your software can’t be tested realistically on a developer’s laptop. And even your testing and staging environments are unlikely to be true representations of your service.

Testing with real, live users on your production environment will always yield better data than any test or staging system

The question is how to do this testing safely. With a feature management platform, teams use powerful targeting rules to control access to new features and can instantly turn access to any feature off, bringing back the old behavior instantly

Canary releases, ring deployments, and percentage rollouts are different patterns for safely testing in production by progressively increasing the exposure of any new change

Percentage Deployments

Ring Deployments

The difference is that the groups are selected specifically to manage the risk of deployment. Microsoft deployments often start with small groups of low-risk users and expand through larger, higher-risk populations.

A typical ring deployment begins by releasing first to internal users. After you have verified that the change has been successful for that set of users, you can expose the next set of users: the canary group.

Then, you can move on to the third ring: beta testers or early adopters

Engaged and happy customers who had the chance to see new features first felt like they were special and in-the-know

This company created its canary group by integrating its Net Promoter Score (NPS) survey solution with its feature management system

Stay in Control

A well-wrapped feature means that you can quickly turn it off if it is performing poorly

Safety valves are permanent feature flags that you can use to quickly disable or limit nonessential parts of your application to keep the rest of the application healthy

Simple safety valves are first managed manually, but the next logical step is to automate them, triggering changes to the flag’s rollout rules based on error monitoring or other metrics. This kind of automation is powerful, but you must use it with caution—a false positive caused by faulty error monitoring could cause unnecessary customer impact

Infrastructure Migrations

Infrastructure migrations, like database schema changes, are some of the riskiest changes in software systems

The key idea is to decompose the problem into multiple steps, with each step controlled by a feature flag that you can roll back

The key idea is to decompose the problem into multiple steps, with each step controlled by a feature flag that you can roll back. At each step, you use correctness checks to validate that it’s safe to proceed to the next step.

Experiment with Everything

product improvements

The ability to release changes to a limited set of users makes it much easier to gather feedback about your product

Customize on the Fly

CHAPTER 2 Advice from the Front Lines

You can begin with feature flags now, no matter what your architecture or deployment cadence is.

Your First Feature Flag

Kill switches are the simplest use case for feature flags and make for a great first flag in your software. I’ve found that teams derive immediate value by introducing kill switches around a new feature

It’s also important to follow these first flags through the entire life cycle, including removing them after they’re no longer needed

Feature management is valuable to many teams in an organization, not just the development team. Depending on the need, features might be managed by product managers, marketing, support, or DevOps teams.

They are an important step in moving toward trunk-based development because the ability to flag code and keep it hidden allows you to merge aggressively, even before the branched code is ready for release.

it’s important to think through the impact of feature flags on your testing strategy

It isn’t necessary (or even possible) to test every combination of feature flags.

flags are an application-level concern, not a library concern

The easiest way to write unit tests for code containing feature flags is to mock-out the feature management service. Mocking is a simple way to avoid interactions with the external feature management service, which is important for unit testing.

Managing Technical Debt

Temporary feature flags introduce technical debt

There’s no royal road to flag cleanup, but there are some processes that make it manageable.

The developer who introduces a flag should be responsible for cleaning it up. To enforce this, appoint a maintainer for the flag.

I usually have some idea of how long it’s intended to remain in the code

create a branch and pull request (PR) to remove the flag at the same time the flag is introduced. This extra branch or PR is kept unmerged until it is time to delete the flag

if feature flags always have the sentinel prefix fflag

Name feature flags in commit messages. When you introduce a flag, add a commit message like “Introduces feature flag flagkey.” Similarly, when you delete a flag, add a commit message like “Removes feature flag flag-key.”

Scaling to Large Teams

hundreds or thousands of developers

Feature Flags Versus Blue/Green Deploys

Blue/green deployments are a technique that involves running two identical production environments (“blue” and “green”). At any point in time, one environment is “live” (e.g., blue), while the other (green) is “idle.” To prepare a release, new code is pushed to the green environment. Gradually, traffic is pointed to the green environment until eventually the blue environment becomes idle

Although there are areas of overlap, each approach has distinct benefits, and the best strategy is to use both

Feature Flags Versus Configuration Management

Configuration parameters are typically stored in files, environment variables, or services like Consul or Redis

For teams using feature flags successfully at scale, it’s tempting to think about moving all of this configuration data into a feature management platform. Feature management platforms solve many of these change management problems, but I still do not recommend moving configuration data into feature flags. Because feature flags are dynamic (they can change at runtime) and context sensitive (they can change depending on the user context), it is more difficult to get a single, simple view of the configuration data.

Summary

CHAPTER 3 Selecting a Feature Management Platform

Feature Management... often go through similar stages of development

Stage 0: Conig ile

Stage 1: Database

Stage 2: Database with context

open source tools

Stage 3: Feature management service

Feature Management Is Mission Critical

In any networked system there are two methods to distribute information. Polling is the method by which the endpoints (clients or servers) periodically ask for updates. Streaming, the second method, is when the central authority pushes the new values to all the endpoints as they change.

Design for Failure

If it fails (for whatever reason), your application should be designed such that it continues to function

Design for Collaboration

just as it is helpful to separate large codebases into smaller units, you can separate your flags into different projects and help the developers avoid the mental overhead of considering hundreds of flags that they don’t need to care about.

important to keep an audit trail of changes made to each flag

Most modern Software as a Service (SaaS) applications are composed of many different programming technologies, and multiple languages are used

it’s important that your feature flags work consistently across all of your applications. When a feature is enabled for a user it must be available across platforms, whether that’s in a browser or a native mobile app.