Feature Flags Are Not Configuration

Note: AI helped put these thoughts into words; the experiences and perspectives are my own.

Feature Flags Are Not Configuration

When production experimentation is necessary, and when a feature flag is the wrong tool.

Production Is Part of the Test Environment

We like to believe that a change can be fully validated before it reaches production. A good test suite, a staging environment, and a careful review are essential, but they cannot reproduce every condition a production system will encounter.

Production has a different scale, workload, traffic distribution, hardware profile, and collection of strange historical states. Some problems only appear when a change meets those conditions. The question is not whether we should test in production. The question is whether we can do it deliberately, with limited exposure and a safe way back.

This is where feature flags are useful. A feature flag gives a system a controlled decision point: use the new behavior for this request, customer, host, or percentage of traffic, and use the old behavior everywhere else. It turns a large, irreversible launch into a sequence of smaller experiments.

The Kernel Upgrade Problem

At one point in my work, we needed to upgrade the guest kernel used by a compute platform powered by Firecracker. The existing guests ran Linux 5.10, and the target was Linux 6.12.

That sounds like a version change, but a kernel is not an isolated application dependency. Between those versions, behavior can change across drivers, system calls, networking, memory management, security settings, device models, and kernel configuration options. A guest can boot successfully and still fail under a workload that exercises a particular path. A configuration that works for one workload may expose a problem for another.

It is difficult to know in advance that such an upgrade will not break something. We can build test images, run integration tests, exercise representative workloads, and inspect the release notes. We should do all of those things. But no test environment perfectly represents every customer workload and every production condition.

In a situation like this, a feature flag lets us introduce the new kernel gradually. We can keep Linux 5.10 as the default, enable Linux 6.12 for an internal workload or a carefully selected slice of capacity, observe boot success, networking, performance, errors, and customer-visible behavior, and then expand the rollout. If the new kernel causes a problem, disabling the flag returns new workloads to the known path without reverting the entire platform release.

The flag does not make the kernel upgrade safe by itself. It creates the control needed to learn safely.

What Makes a Good Feature Flag?

A useful flag has a narrow purpose and a clear owner. It controls a meaningful behavioral choice, such as which implementation handles a request or which version of a platform component is selected.

It should also have:

A measurable hypothesis: We know what we are trying to learn and which metrics or alerts will tell us whether the new behavior is healthy.
A bounded blast radius: The first rollout targets a small and understood portion of traffic, users, or infrastructure.
A safe fallback: Turning the flag off returns the system to a previously deployed and operational path.
An expiry plan: The flag is temporary unless there is a strong reason for it to remain as a permanent product decision.
An owner: Someone is responsible for expanding, disabling, or removing it.

The operational details matter as much as the boolean. A flag that can be changed but is not observable is not a reliable experiment. A flag that can be disabled but leaves partially migrated state behind is not a genuine rollback. Before enabling a risky path, we need to understand what disabling it actually does.

Flags Are Not a Substitute for Review

Feature flags are sometimes described as a way to separate deployment from release. That separation is valuable, but it can be misunderstood. A flag does not eliminate the need for code review, testing, or an operational runbook. It only changes when a deployed code path becomes active.

The code behind the flag still needs to be reviewed. Its data migrations, compatibility assumptions, resource usage, and failure modes still need to be understood. The flag should make an uncertain change easier to release gradually, not make an unreviewed change acceptable.

It is also important to distinguish a flag from a kill switch. A kill switch may exist permanently to disable an emergency-prone capability. Most rollout flags should have a shorter life: enable the new path, gather evidence, remove the old path and the flag. Otherwise every request eventually has to account for both branches, and the system accumulates permanent conditional complexity.

When Not to Use Feature Flags

The fact that a feature flag can hold a value does not mean it should hold every value that changes at runtime. A flag system is usually optimized for making behavioral decisions quickly. It is not automatically a good system for managing configuration or recording durable data.

Configuration Management

Imagine a service has a simple configuration value that determines which resources it is allowed to use. That value may include CPU limits, memory limits, network permissions, or other operational boundaries. It is tempting to put the configuration inside a feature flag because the flag provider makes it easy to edit without deploying.

That is usually the wrong trade-off.

Configuration changes should ideally be protected by the same review and ownership model as other production changes. A configuration repository can have strict code owners, pull-request approval, a clear Git diff, automated validation, and a history that explains exactly what changed and why. Those controls make a sensitive change deliberate.

Some feature-flag providers offer revision history, but a history of values is not the same as a useful change review. It may show the value after each edit without presenting the surrounding context or a Git-style diff. Access controls may also be broader or less strict than the ownership rules protecting production configuration repositories.

Configuration should be managed as configuration. If it needs a review, an audit trail, validation, and a predictable rollback, use a configuration-management workflow that provides those properties directly.

Change Management

A feature flag should never be used to disguise a change that requires a formal operational process. Changes to permissions, resource limits, network policy, data retention, or security posture often need explicit approval, communication, and evidence of who authorized them.

Putting such a change behind a flag can make it look reversible while bypassing the controls that should surround it. The ease of changing a flag is precisely why it may be the wrong tool for a high-consequence configuration change.

The rollback question is a useful test: if switching the flag off does not completely and safely undo the change, the change probably does not belong behind a feature flag. A flag cannot undo data that was deleted, resources that were permanently reconfigured, credentials that were exposed, or an incompatible migration that has already happened.

Datastore Behavior

Do not use a feature-flag service as a datastore where every flag is equivalent to a database record. Once flags become entries such as customer-123-limit, customer-124-limit, and customer-125-limit, the system has lost the semantics of feature rollout and gained an opaque database with poor query, validation, and lifecycle properties.

This pattern makes it difficult to answer basic questions: Which records exist? Who owns them? What is their schema? How do we validate them in bulk? How do we migrate them? How do we back them up and restore them? A feature-flag provider may technically store the values, but technical possibility is not a sound data model.

Use a database for durable application state, a configuration system for operational configuration, and a feature-flag system for controlled changes in application behavior.

A Practical Decision Rule

Before creating a flag, ask what kind of change it represents:

If it changes executable behavior and we need to learn from a limited production rollout, a feature flag may be appropriate.
If it defines the desired operating state of a service, use configuration management.
If it changes security, permissions, resources, or other controlled operational policy, use the relevant change-management process.
If it represents durable business or application state, use a datastore.

Then ask whether the change has a real fallback, whether its effects are observable, and who will remove the flag. If those questions do not have clear answers, the flag is probably creating uncertainty rather than containing it.

Conclusion: Temporary Control, Not Permanent Infrastructure

Feature flags are one of the most useful tools for dealing with uncertainty. They let us test changes against reality without exposing everyone to the change at once. For a risky platform change such as moving Firecracker guests from Linux 5.10 to 6.12, that controlled exposure can be the difference between a measured rollout and a platform-wide incident.

But feature flags are not a replacement for testing, code review, configuration management, change management, or a database. Their value comes from being narrow, observable, reversible, and temporary.

Use a feature flag when you need a controlled experiment in behavior. Do not use one merely because it is the easiest place to store a value. The right tool is not the one that can hold the change. It is the one that gives the change the guarantees it actually needs.