Support multi-environment deployment to a single APIM instance

[petehauge]

Summary

Support deploying multiple environments (e.g., dev, staging, prod) to the same Azure API Management instance, with clear separation between environments via naming conventions or prefixes.

Problem

Currently, apiops-cli assumes a 1:1 mapping between environments and APIM instances. Many organizations want to consolidate multiple environments into a single APIM instance for cost or operational reasons, but need clear separation between environment-specific APIs.

Proposed Solution

Extract

• Support extracting APIs from a single APIM instance while tagging/grouping them by environment (e.g., via API name prefix, path prefix, or tags).
• Allow filter configuration to scope extraction to a specific environment subset.

Publish

• When publishing to a target environment (e.g., "Dev"), automatically apply an environment-specific prefix or suffix to API names/paths (e.g., dev-my-api or /dev/my-api).
• Support configuration to define the naming/prefixing strategy per environment.
• Ensure environment-specific named values, backends, and other resources are correctly scoped.

Configuration

• Extend the configuration system to support an environment concept with per-environment prefixing rules.
• Allow users to define environment mappings in their configuration files (e.g., configuration.dev.yaml targets the same APIM instance but with a dev- prefix).

Acceptance Criteria

• Extract supports filtering by environment within a single APIM instance
• Publish supports applying environment-specific prefixes/suffixes to APIs
• Configuration schema supports environment-to-prefix mappings
• Documentation covers multi-environment setup on a single APIM instance
• Tests cover extract and publish with environment prefixing

[jxyiliu1]

Hope we could have this feature implemented soon, our organization is waiting for this feature for long....

[petehauge]

Thanks jxyiliu1 for the heads-up! I also saw htwashere was interested in this feature too.

I'm working on a design now, but there are some unexpected complexities. Could you review the rough design & plans below and let me know if you have any concerns so we can refine the design?

1. The current design is that there would either be a 'source of truth' APIM instance, or artifacts in code, which would be the 'base' for all the environments. That way we can do a clean 'extract' without any environment-specific naming. Is that reasonable?
2. There will probably be some 'global' resources (like policy fragments or loggers) that are specific to the instance and not associated with a particular environment (like dev, qa, stage, prod). Would it be reasonable to expect users to filter those out when publishing? There could be a separate pipeline for publishing those if changes are needed.
3. The way we expect it to work is that there's a new property in the environment overrides file that has a 'NamePrefix' for that environment. Then when we publish, we would prefix all the names that we publish with that name prefix. (like "dev_" or "prod_").
4. We would have to do deep re-writes of names as we publish, for example, if a Policy XML uses a named value, we'd need to deeply rewrite the names to point to the environment specific names. Does that sound reasonable?

Thanks!

[htwashere]

Thanks @jxyiliu1 for the heads-up! I also saw @htwashere was interested in this feature too.

I'm working on a design now, but there are some unexpected complexities. Could you review the rough design & plans below and let me know if you have any concerns so we can refine the design?

1. The current design is that there would either be a 'source of truth' APIM instance, or artifacts in code, which would be the 'base' for all the environments. That way we can do a clean 'extract' without any environment-specific naming. Is that reasonable?
2. There will probably be some 'global' resources (like policy fragments or loggers) that are specific to the instance and not associated with a particular environment (like dev, qa, stage, prod). Would it be reasonable to expect users to filter those out when publishing? There could be a separate pipeline for publishing those if changes are needed.
3. The way we expect it to work is that there's a new property in the environment overrides file that has a 'NamePrefix' for that environment. Then when we publish, we would prefix all the names that we publish with that name prefix. (like "dev_" or "prod_").
4. We would have to do deep re-writes of names as we publish, for example, if a Policy XML uses a named value, we'd need to deeply rewrite the names to point to the environment specific names. Does that sound reasonable?

Thanks!

Hi Peter Hauge (@petehauge), thanks for your attention to this feature. Here are my thoughts to your design notes above:
Point 1: I think it is reasonable. Your ApiOps Extract currently asks for an APIM instance as a source of truth. APIM enforces the API-name to be unique anyway. If we use one APIM instance to serve multiple environments for the same API, we need to differentiate them via the API-name anyway so technically, APIM will consider them as two separate APIs (as you are pointing it out in point 3)
Point 2: In our use case after examining your ApiOps Extract, I think we will be using your filter feature to limit the extracted resources. So I think when we publish to the next APIM, those global resources will be excluded?
Point 3: Could the NamePrefix be a token where we can place anywhere in the API-Name string? I am just thinking for organizations like ours who already have certain naming convention. Or, perhaps the environment be a "suffix" better?
Point 4: I think you are correct regarding deep-rewrite. I am not too familiar with the overall structure of the files yet.
BTW, is there an explanation on all the files that are extracted for the resources? At this moment, I am trying to understand the purpose of file apiInformation.json in the context of the extract and publish. Where does APIM use it?
I will be away next week so will be slow in responding to you in subsequent dialogues.
Thank you!
Henry

[petehauge]

Hi htwashere -

For 2 - you're correct, that if you filter on extract, anything filtered out don't have associated files on disk and aren't published

For 3 - could you share your naming scheme? We could do either a prefix or suffix - just wanted to confirm that we wouldn't need to replace tokens in the middle of the names?

We don't have any docs on explanation of the specific files but we could probably write something. If you need it - could you log a new github issue requesting the docs? In terms of apiInformation.json, these are typically the properties associated with an individual API. I included a screenshot of an example I have - things like the revision, path, display name, protocols, service URL, etc. It's the key properties we get back when we do a 'get' against the API in API Management. You'll see a lot of these things on the "Settings" tab for an API (2nd screenshot below)

[htwashere]

Hi Peter Hauge (@petehauge) , our naming convention is still being standardized but I see many of our teams are settling to: "dept env apiname" or "dept apiname env". I think we can settle for a suffix but if you could do replace tokens for the name, I can see that being the best? While we are on this name topic, I suppose with the displayName, we will continue to use the "override" feature to change the Display Name so no issue there?

Thanks for the apiInformation.json explanation. Are you using it for any logic processing in apiops publish? I am thinking if that file needs to be considered (overrides) for the target APIM?

[petehauge]

Hi htwashere - on the replacing tokens, this is hard because either:

1. There's some tag you put in all the right places like the word "env" in the live instance, and never have another name that uses the same text accidently (like envelope, envoy, etc), or
2. You manually fix all the artifacts on disk to add the token like "{env}" that we would replace

Via the cli, we could do the replacement, but I'm just concerned about the user experience for it - that it might be error prone and difficult for users. If we support both prefix & suffix, it sounds like that would be enough (and is straightforward from a user experience perspective)?

On the apiInformation.json file - there are some things we currently do during publish:

• After reading in the json file:
• Normalize MCP tool operation references
• Apply overrides (this is a deep merge)
• If there's a spec file next to apiInformation.json, we strip null-valued properties
• Fixup the properties.apiType field
• Sanitize the spec (add missing openAPI path parameters for example)
• Put the resulting payloads to APIM
• If we were replaying revisions, then run again to re-align the active-revision metadata

(and then if this issue is fixed, it will add another bullet to apply name prefix/suffix)

[petehauge]

Multi-Environment Publish — Design Proposal

Scoping assumptions:

• Extract stays as-is. Artifacts on disk hold canonical (unprefixed) names, produced from a dedicated "source-of-truth" APIM.
• Publish applies an environment namePrefix and/or nameSuffix declared in the override file.

1. Config surface

Extend the override YAML with a new top-level environment block:

# configuration.dev.yaml
environment:
  namePrefix: "dev-"      # optional
  nameSuffix: ""          # optional
  # Applies to which resource types (see §2).
  # Omit = apply to the default set. Explicit list = apply only to those.
  appliesTo:              # optional
    - apis
    - products
    - namedValues
    - backends
    - loggers
    - policyFragments
    - versionSets
    - tags
    - groups
    - subscriptions
    - workspaces
  # Independent from name affix — API URL path also needs uniqueness on shared APIM.
  apiPathPrefix: "dev/"   # optional; prepended to properties.path on APIs

apis:
  - name: petstore-api    # <-- canonical name, no prefix
    properties:
      serviceUrl: https://petstore-dev.contoso.com

Key rules:

• The override file continues to reference resources by canonical name (matches artifacts on disk). This preserves the "extract once, override per env" model.
• environment is optional. If absent, behaviour is identical to today (100% back-compat).
• Bump schema v1 → v2 per the repo's update-config-schema-version skill. v1 files continue to load (no environment block = no-op).

Why in the override file, not CLI flags: Env metadata belongs with env-specific values, is version-controlled, reviewable, and testable. CLI flags could be added later as an escape hatch but shouldn't be the primary UX.

2. Which resource types get affixed

Not everything can be renamed safely. Proposed default set:

Type	Affix?	Why
Api	✅	Primary case
Product	✅	Primary case
NamedValue	✅	Displayname is APIM's lookup key for {{ref}} — see §4
Backend	✅	Referenced by name in policies
PolicyFragment	✅	Referenced by name in policies
Logger	✅	Referenced by name in diagnostics
VersionSet, Tag, Group, Subscription, Workspace	✅	Global namespace on APIM
Gateway	⚠️ opt-in	Often shared infra; default off
ServicePolicy (singleton)	❌	Fixed name "policy"; can't be renamed
Child resources (ApiOperation, ApiDiagnostic, ApiPolicy, ApiRelease, ProductGroup, ProductApi, GatewayApi, ApiTag, etc.)	❌ directly, ✅ via parent	Names are scoped inside the parent; only the parent's segment is affixed
Documentation, GlobalSchema, PolicyRestriction	✅	Global namespace

Users override the default via appliesTo when needed.

3. Where the transform lives — a single env-mapper module

New file: src/services/env-mapper.ts. Exports:

export interface EnvMapping {
  prefix: string;
  suffix: string;
  appliesTo: Set<ResourceType>;
  apiPathPrefix?: string;
}

export function buildEnvMapping(overrides: OverrideConfig | undefined): EnvMapping | undefined;

// canonical → deployed name (used at publish time)
export function toDeployedName(name: string, type: ResourceType, m: EnvMapping): string;

// deployed → canonical (used by delete-unmatched to scope diff)
export function toCanonicalName(deployedName: string, type: ResourceType, m: EnvMapping): string | undefined;

// belongs to this env's namespace?
export function isInEnvNamespace(deployedName: string, type: ResourceType, m: EnvMapping): boolean;

// rewrite a descriptor's nameParts (parent segment + optionally child segments)
export function mapDescriptor(d: ResourceDescriptor, m: EnvMapping): ResourceDescriptor;

Design choice — map at the publish edge, not throughout the code. Everywhere that reads artifacts, applies overrides, does dependency graph work: keep using canonical names. Rewrite to deployed names in exactly two places:

• Right before client.putResource(...) in resource-publisher.ts
• Right before client.deleteResource(...) in the delete path

Advantages: filter, override-merger, artifact-store, dependency-graph, transitive-resolver all stay untouched. Testability stays high. applyOverrides() still matches by canonical name, which is what the user wrote in their YAML.

4. Cross-reference rewrites (the hard part)

At publish time the following JSON/XML payloads carry embedded references that must be rewritten from canonical → deployed:

Location	Ref shape	Fix
ProductApi, ProductGroup, GatewayApi association entries	entry.name	Map through toDeployedName for the child type (API/Group). Already loops per-entry in resource-publisher.ts:307-322 — inject the mapping there.
Subscription.properties.scope	/apis/<apiName> or /products/<productName>	Extend the existing normalizeSubscriptionScope path to also affix the trailing segment.
ApiRelease.properties.apiId	ARM path ending /apis/<apiName>[;rev=N]	Extend existing normalizeApiReleaseApiId to affix.
API revisions sourceApiId	ARM path	Same treatment.
Logger credentials {{ref}} values	Placeholder tokens resolved by NamedValue displayName	Existing rewriteNamedValueReferences already looks up the canonical named value. If NamedValue is in the affix set, its displayName override (if any) is already applied there — but if the user didn't set displayName, we must also affix it. Extend that function to affix displayName when NamedValue ∈ appliesTo and no explicit override was provided.
Policy XML content	{{namedValueRef}}, <include-fragment fragment-id="..."/>, <send-request> back-refs, etc.	New policy-xml rewriter (§5).
API properties.path	URL suffix	Prepend apiPathPrefix when present. Small, isolated change in the Api branch of resource-publisher.ts.
Workspace-scoped child descriptors	descriptor.workspace and child nameParts[0] (parent id)	mapDescriptor handles both segments.

5. Policy XML rewriting

New util src/services/policy-ref-rewriter.ts:

export function rewritePolicyRefs(xml: string, m: EnvMapping, ctx: {
  knownNamedValues: Set<string>;      // canonical names present in artifacts
  knownFragments: Set<string>;
  knownBackends: Set<string>;
}): string;

Approach:

• Do not parse XML. Use targeted regex passes for {{name}} tokens and known element/attribute patterns (<include-fragment fragment-id="...">, <set-backend-service backend-id="...">). This matches the existing "opaque payload" philosophy (FR-009) — we only touch tokens we can confidently identify.
• Only rewrite refs where the canonical name is known to be part of the artifact set for that type and that type is in appliesTo. Unknown tokens pass through unchanged (they may be runtime context refs like {{context.request.headers}}).
• Called from the policy branch of resource-publisher.ts before PUT.

Risk: false-positives if a customer's XML text incidentally matches. Mitigated by (a) restricting to tokens whose base name is in the known artifact set and (b) opt-out via appliesTo not listing that type.

6. delete-unmatched — the safety-critical piece

Today computeDeleteActions will happily delete a Prod resource when publishing Dev. Change:

1. client.listResources(context, type) returns ARM (deployed) names.
2. For each resource, if EnvMapping is present and this type ∈ appliesTo: skip anything not in this env's namespace (isInEnvNamespace(deployedName, type, m) === false).
3. For resources in-namespace: convert deployed name → canonical name via toCanonicalName, then do the existing "is this in local artifacts?" check.
4. Delete uses the deployed descriptor (rewritten via mapDescriptor).

This makes shared-APIM publish safe: Dev can never delete Prod because Prod resources are outside the dev-* namespace.

Edge case: if a user drops the prefix mid-life (namePrefix: "" after previously publishing with dev-), delete-unmatched can't distinguish the two envs anymore. Document as a known constraint.

7. Collision & validation checks (fail fast at publish start)

Add a pre-flight validator in publish-service.ts:

• If environment present, warn if prefix/suffix contain characters APIM disallows in resource names (must match ^[A-Za-z0-9-]+$ for most types).
• Error if appliesTo includes a type that can't be renamed (ServicePolicy, association children).
• Warn if an override entry references a canonical name that doesn't exist in artifacts.
• Error if apiPathPrefix is set but no API in the artifact set — likely a typo.

8. Extract stays canonical — but add one convenience

Per the scoping assumption, extract needs no code change. Worth adding one docs section: users should extract from a dedicated non-shared APIM (or an APIM whose resources happen to have no prefix). If someone points extract at a shared APIM, they'll get all envs' resources mixed together — call this out explicitly.

Optional (Phase 3, not blocking): apiops extract --strip-prefix dev- for teams that only have a shared APIM. Purely a name-strip pass at write time. Not required for MVP.

9. init changes

Minimal:

• The scaffolded configuration.<env>.yaml gets a commented-out environment: block with prefix/suffix examples.
• The Copilot prompt (configure-overrides-prompt.md) gains a section explaining when to enable it.
• No change to workflow templates — the same apiops publish --overrides configuration.<env>.yaml command handles both single-APIM-per-env and shared-APIM setups.

10. Testing

• Unit tests for env-mapper.ts (round-trip: canonical → deployed → canonical for every type; unaffected types unchanged).
• Unit tests for policy-ref-rewriter (known refs rewritten, unknown left alone, no XML corruption).
• Integration tests for resource-publisher.ts covering: association entries, subscription scope, apiRelease apiId, api path prefix, logger credentials.
• Delete-unmatched test using a mock APIM listing that returns both dev-foo and prod-foo — publish with namePrefix: dev- must skip prod-foo.
• Back-compat: existing test suite must pass unchanged with no environment block.