Skip to content

supersedes #2066 Downstream container-libs vendoring for CRI-O - #2076

Draft
QiWang19 wants to merge 4 commits into
openshift:masterfrom
QiWang19:crio-vendor-version
Draft

supersedes #2066 Downstream container-libs vendoring for CRI-O#2076
QiWang19 wants to merge 4 commits into
openshift:masterfrom
QiWang19:crio-vendor-version

Conversation

@QiWang19

@QiWang19 QiWang19 commented Aug 5, 2026

Copy link
Copy Markdown
Member

Summary

Describe how downstream CRI-O will vendor a downstream fork of container-libs using Go module replace directives, ensuring alignment with the container-libs version shipped in the target RHEL release.

Supersedes #2066

Changes from #2066

  • Reorganize workflows into numbered scenarios (1-4) for clearer reference
  • Add RHEL-to-container-libs version mapping table
  • Add detailed embargoed CVE process (openshift-priv workflow)
  • Add security fix SLA targets
  • Add note about OCP z-stream RHEL minor version shifts
  • Link to Container-tools Version Consolidation proposal

Open items

  • Confirm RHEL-to-container-libs version mapping with Container-tools team
  • Confirm embargo workflow details and SLA targets with security response team

Summary by CodeRabbit

  • Documentation
    • Added an enhancement proposal for vendoring downstream container libraries in CRI-O.
    • Documented version alignment, update workflows, testing, security procedures, upgrade behavior, risks, and support considerations.

bitoku added 3 commits July 23, 2026 12:57
…roposal

Describe how downstream CRI-O will vendor a downstream fork of
container-libs using Go module replace directives, ensuring alignment
with the container-libs version shipped in the target RHEL release.

Assisted-by: Claude Code <https://claude.com/claude-code>
…ment details

Fill in TODOs and resolve inconsistencies in the enhancement proposal:
- Add downstream container-libs repo link and concrete go.mod replace directives
- Expand user stories with specific personas, workflows, and rationale
- Add Dual Stream section covering RHEL 9/10 parity and future divergence handling
- Add Bug Fix / Feature Backport Workflow section
- Add RHEL Version Switch subsection to Upgrade/Downgrade Strategy
- Address version skew for dual-stream RHEL transitions
- Add container-libs version divergence detection test plan
- Fix Goal openshift#3 to not contradict dual-stream build tag/spec patching
- Clarify that MachineConfig-delivered custom configs may need updates on RHEL switch
- Correct upstream module paths to go.podman.io vanity imports
- Remove resolved open question about Go module proxy (already rejected in Alternatives)

Assisted-by: Claude Code <https://claude.com/claude-code>
…ture details

- Add CI Access to gitlab.cee section describing the intranet capability
  and GOPRIVATE configuration for Prow jobs
- Replace manual workflow with automated Prow job for container-libs
  vendoring updates
- Add presubmit divergence check to test plan
- Resolve open questions on automation and gitlab.cee accessibility
- Simplify upstream rebase workflow to use carry patch model
- Simplify dual-stream RHEL mitigation
- Update Infrastructure Needed with specific Prow job requirements

Assisted-by: Claude Code <https://claude.com/claude-code>
@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 5, 2026
@openshift-ci

openshift-ci Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@coderabbitai

coderabbitai Bot commented Aug 5, 2026

Copy link
Copy Markdown

Walkthrough

The enhancement adds a proposal for vendoring downstream container-libs modules in CRI-O. It defines Go module replacements, version alignment, update and security workflows, CI requirements, compatibility behavior, and support procedures.

Changes

Downstream container-libs vendoring

Layer / File(s) Summary
Vendoring design and compatibility model
enhancements/node/downstream-container-libs-vendoring-for-cri-o.md
Defines Go replace directives for downstream common, storage, and image modules. Documents version alignment, module behavior, topology, and downstream fork requirements.
Maintenance and security workflows
enhancements/node/downstream-container-libs-vendoring-for-cri-o.md
Documents dependency updates, upstream rebases, security fixes, backports, risks, mitigations, and alternatives.
Validation and support model
enhancements/node/downstream-container-libs-vendoring-for-cri-o.md
Documents CI and drift-detection tests, version-skew behavior, support procedures, open questions, and required infrastructure.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The pull request changes only one Markdown proposal and adds no Ginkgo test titles or test code; the deterministic-name check is not applicable.
Test Structure And Quality ✅ Passed The PR changes only one Markdown enhancement proposal; it adds no Ginkgo It blocks, cluster operations, waits, or test assertions to review.
Microshift Test Compatibility ✅ Passed The pull request changes only a Markdown enhancement proposal and adds no Ginkgo e2e tests, so MicroShift test compatibility checks do not apply.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The pull request changes only a Markdown enhancement proposal; it adds no Ginkgo e2e tests or test code that could make multi-node assumptions.
Topology-Aware Scheduling Compatibility ✅ Passed The commit changes only an informational Markdown proposal; no manifests, operator code, controllers, or scheduling constraints were added or modified.
Ote Binary Stdout Contract ✅ Passed The patch changes only one Markdown proposal file; it adds no Go source, OTE lifecycle code, or process-level stdout writes.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The pull request changes only one Markdown proposal file; it adds no Ginkgo e2e tests or executable network checks. External links are documentation only.
No-Weak-Crypto ✅ Passed The commit changes only one Markdown proposal, and scans of the proposal and added lines found no MD5, SHA1, DES, RC4, 3DES, Blowfish, ECB, custom crypto, or secret-comparison usage.
Container-Privileges ✅ Passed The PR changes only a Markdown proposal; it contains no container/Kubernetes manifest and no matches for the listed privilege settings or root execution.
No-Sensitive-Data-In-Logs ✅ Passed The proposal adds no logging code, log examples, or log payloads. It mentions internal GitLab and private repositories only as build and access references.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the main change: downstream container-libs vendoring for CRI-O, and it notes that the pull request supersedes #2066.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci

openshift-ci Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign deads2k for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 12

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@enhancements/node/downstream-container-libs-vendoring-for-cri-o.md`:
- Around line 248-253: Update the divergence checks comparing CRI-O and Podman
module versions to validate immutable source identity instead of version
strings. Resolve and compare the downstream commit or digest used by both
builds, or consume a shared version-to-commit mapping, ensuring distinct
downstream version numbers cannot be treated as proof of identical source.
- Around line 5-15: Complete the enhancement metadata by replacing the TODO
entries under reviewers, approvers, and tracking-link with the appropriate
owners and tracking reference; if the proposal remains draft-only, update status
to explicitly indicate that.
- Around line 277-279: Update the proposal’s dual-stream mitigation around the
“single set of replace directives” claim to define how each RHEL stream selects
and pins its container-libs version. Specify separate branches/manifests or
RPM-preparation generation of target-specific go.mod and vendor trees, and
require building and testing both RHEL artifacts.
- Around line 119-123: Update the “Scenario 2: Upstream CRI-O Rebase” procedure
to require rebase automation to run go mod tidy and go mod vendor after applying
the carry patch, regardless of whether it applies cleanly, then execute
downstream build tests to validate go.sum, vendor/, and vendor/modules.txt
against the resulting dependency graph.
- Line 52: Update the RHEL-to-container-libs mapping section in the
downstream-container-libs vendoring doc so each OCP target is keyed by a single
exact RHEL RPM suffix or build target instead of multiple ambiguous rows. Use
the mapping table and its related Podman/container-libs entries to record one
authoritative Podman commit and one container-libs module set per target, and
mark the OCP 4.x mappings as historical since the proposal begins with OCP 5.0.
- Around line 341-350: The CI status is contradictory between the Test Plan and
Infrastructure Needed sections. Reconcile the descriptions in “Test Plan,”
“Infrastructure Needed,” and the corresponding section around the later
duplicate occurrence: either mark the build, e2e, payload, and container-libs
version-check Prow jobs as existing and remove them from planned work, or
describe them as planned and move their details out of the test plan; explicitly
identify any operational versus required jobs.
- Around line 381-395: Expand the “Upgrade / Downgrade Strategy” and “Version
Skew Strategy” sections to define storage compatibility and rollback
requirements rather than claiming upgrades are transparent. Document migration
limits, backup and recovery procedures, custom storage.conf/containers.conf
handling, and any required storage wipe or recreation steps; add coverage for
persistent storage, custom configuration, mixed RHEL streams, upgrades, and
rollbacks.
- Around line 240-242: Update the “CI Access to gitlab.cee” documentation to
define read-only GitLab credential provisioning and secret handling for Prow,
ART/RPM, and developer build paths. Explicitly distinguish intranet and
GOPRIVATE from authentication, and document behavior when vendor/ is present
versus when go mod tidy or go mod vendor requires GitLab access.
- Around line 240-242: Update the CI access documentation/configuration around
the `openshift/cri-o` vendoring setup to define the embargo private-module
policy, including the embargo fork host and equivalent `GOPRIVATE` settings for
`openshift-priv/cri-o`. Add a regression test that verifies neither the module
proxy nor checksum database receives requests for these private modules.
- Around line 150-160: Update the security-fix policy section around the
expected turnaround table and its TODO to define, with the security response
team, the SLA clock start, downstream container-libs vendor-update deadline,
escalation owner, and ART rebuild target before publishing the policy. Replace
the unresolved TODO with these confirmed boundaries and ensure the stated
targets are consistent with them.
- Around line 262-264: Replace the TODO in the “Mitigation” section with a clear
reference to a follow-up operations runbook or implementation issue. Specify
that the follow-up must document the immutable source pin, go.mod and vendor/
updates, validation for each RHEL target, and the replacement RPM release path.
- Around line 88-104: Expand the downstream CRI-O vendoring guidance to document
each container-libs submodule’s module declaration, source directory,
release-tag prefix, and version, including the required /v5 major-version rule
for image. Explicitly state that existing go.podman.io/... imports remain
unchanged, and add validation steps from a downstream openshift/cri-o checkout
using go mod tidy, go mod vendor, and go list -mod=vendor ./....
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 8d9665cc-a59b-4816-800f-fdcdd1490eb0

📥 Commits

Reviewing files that changed from the base of the PR and between c79913a and 6948012.

📒 Files selected for processing (1)
  • enhancements/node/downstream-container-libs-vendoring-for-cri-o.md

Comment thread enhancements/node/downstream-container-libs-vendoring-for-cri-o.md
Comment thread enhancements/node/downstream-container-libs-vendoring-for-cri-o.md Outdated
Comment thread enhancements/node/downstream-container-libs-vendoring-for-cri-o.md Outdated
Comment thread enhancements/node/downstream-container-libs-vendoring-for-cri-o.md Outdated
Comment on lines +150 to +160
Expected turnaround times for security fixes (based on Red Hat Product Security internal targets):

| Severity | Target |
|----------|--------|
| Critical | 5 business days |
| Important | 21-28 calendar days (45 days for OCP container image rebuilds) |
| Moderate | 57 calendar days |

For container-libs specifically, if the upstream containers maintainers need to develop and release the fix first, the clock does not start until the fix is available to vendor. The full chain for a container-libs CVE is: upstream fix → downstream container-libs fork → CRI-O vendor update → ART rebuild, which may be longer than for components where the fix is entirely within OpenShift's control.

TODO: Confirm the above targets and the exact location where embargoed container-libs fixes are developed (gitlab.cee downstream fork, a private GitHub fork, or upstream private branch) with the security response team.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift

🧩 Analysis chain

🏁 Script executed:

# First, find and read the exact file at the specified lines
fd -t f "downstream-container-libs-vendoring-for-cri-o.md" --exec cat -n {} \; | head -200

Repository: openshift/enhancements

Length of output: 14461


Security SLA Policy Timing Gap (CWE-400): Uncontrolled Resource Consumption

Reachability: Internal · Exploitability: Theoretical

Define SLA clock boundaries for container-libs security fixes before publishing policy.

The table prescribes fixed response targets, but the text states that the clock does not start until upstream releases a fix. Define the clock start, downstream deadline, escalation owner, and ART rebuild target with the security response team before finalizing the policy.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@enhancements/node/downstream-container-libs-vendoring-for-cri-o.md` around
lines 150 - 160, Update the security-fix policy section around the expected
turnaround table and its TODO to define, with the security response team, the
SLA clock start, downstream container-libs vendor-update deadline, escalation
owner, and ART rebuild target before publishing the policy. Replace the
unresolved TODO with these confirmed boundaries and ensure the stated targets
are consistent with them.

Comment thread enhancements/node/downstream-container-libs-vendoring-for-cri-o.md Outdated
Comment thread enhancements/node/downstream-container-libs-vendoring-for-cri-o.md
Comment thread enhancements/node/downstream-container-libs-vendoring-for-cri-o.md
Comment thread enhancements/node/downstream-container-libs-vendoring-for-cri-o.md
Comment on lines +381 to +395
## Upgrade / Downgrade Strategy

Transparent to cluster upgrades and downgrades. Each OCP release ships a CRI-O binary with its vendored dependencies bundled at build time. Upgrading or downgrading OCP replaces the CRI-O binary entirely. No cluster-level configuration changes are required.

### RHEL Version Switch (Dual Stream)

When a cluster transitions from one RHEL version to another (e.g., RHEL 10 to RHEL 11) within the same OCP version, nodes are reprovisioned with a new RHCOS image. The new image includes the CRI-O RPM built for the target RHEL version, with the corresponding container-libs vendored. Reprovisioning replaces the OS image — including the CRI-O binary and default configuration files. However, custom node configurations delivered via MachineConfig (e.g., custom `storage.conf` or `containers.conf`) persist across reprovisioning. If a new container-libs version changes config file format or semantics, those custom configs may need to be updated.

During rolling reprovisioning, the cluster temporarily has nodes running CRI-O built with different container-libs versions (e.g., some nodes on RHEL 10's container-libs, others on RHEL 11's). This is safe because the CRI gRPC API between kubelet and CRI-O is unchanged by the container-libs version — the difference is internal to the CRI-O binary.

## Version Skew Strategy

CRI-O is a node-level component. During upgrades, nodes may temporarily run different CRI-O versions. This is unchanged by this proposal — container-libs vendoring is internal to the CRI-O binary and does not affect inter-component communication or APIs.

During dual-stream RHEL transitions, nodes may run CRI-O binaries built with different container-libs versions. This does not introduce inter-node version skew concerns because container-libs is statically linked into CRI-O and does not affect inter-component APIs. On-disk container storage formats managed by the `storage` library are local to each node; if a container-libs update changes the on-disk format, the impact is limited to the upgraded node and handled by CRI-O's existing storage migration logic.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -eu

file="enhancements/node/downstream-container-libs-vendoring-for-cri-o.md"

printf '%s\n' '--- target section ---'
sed -n '350,430p' "$file"

printf '%s\n' '--- related storage/configuration/migration references ---'
rg -n -i -C 3 'storage\.conf|containers\.conf|migration|on-disk|storage format|rollback|downgrade|version skew|RHEL 10|RHEL 11|transparent' "$file"

printf '%s\n' '--- repository files related to this proposal ---'
git ls-files | rg -i 'cri.?o|container.?libs|storage\.conf|containers\.conf|vendoring|enhancement'

Repository: openshift/enhancements

Length of output: 50378


🏁 Script executed:

#!/bin/bash
set -eu
file="enhancements/node/downstream-container-libs-vendoring-for-cri-o.md"
sed -n '350,430p' "$file"
rg -n -i -C 3 'storage\.conf|containers\.conf|migration|on-disk|storage format|rollback|downgrade|version skew|RHEL 10|RHEL 11|transparent' "$file"
git ls-files | rg -i 'cri.?o|container.?libs|storage\.conf|containers\.conf|vendoring|enhancement'

Repository: openshift/enhancements

Length of output: 50378


🏁 Script executed:

#!/bin/bash
set -eu
file="enhancements/node/downstream-container-libs-vendoring-for-cri-o.md"

printf '%s\n' '--- test plan and support procedures ---'
sed -n '330,365p;399,420p' "$file"

printf '%s\n' '--- proposal risks and assumptions ---'
sed -n '245,285p' "$file"

printf '%s\n' '--- focused repository references ---'
rg -n -i -C 2 --glob '*.md' \
  'containers/storage|storage migration|storage\.conf|containers\.conf|MachineConfig|rollback process|persistent state|upgrade.*downgrade' \
  enhancements/node enhancements/machine-config enhancements/hypershift | head -250

Repository: openshift/enhancements

Length of output: 36840


🌐 Web query:

containers/storage compatibility on-disk storage format migration downgrade configuration storage.conf containers.conf CRI-O

💡 Result:

The containers/storage library does not support automatic or safe "downgrade" migrations for its on-disk storage format [1][2]. Because the library manages filesystem layers, images, and container metadata using specific copy-on-write (COW) drivers (like overlay) and internal database structures, downgrading to a version of the software that predates a structural format change is generally not supported and can result in data corruption or loss [1][3][4]. Key points regarding migration and configuration: Configuration Management: The primary configuration is handled via /etc/containers/storage.conf (or user-specific variants in $XDG_CONFIG_HOME/containers/storage.conf), which uses TOML format [5][6][7]. CRI-O also references these defaults and allows for runtime-specific overrides in its own configuration files (e.g., /etc/crio/crio.conf) [8][9]. Data Persistence and Storage Paths: 1. graphroot: Specifies the persistent data directory (default: /var/lib/containers/storage) [6][7]. 2. runroot: Specifies the temporary, volatile state directory (default: /run/containers/storage) [7][8]. If you move or reconfigure these paths, you must ensure appropriate SELinux labeling (e.g., using semanage and restorecon) to maintain security policies [7][10]. Migration Warnings: There is no "downgrade" flag or configuration option. If a storage format upgrade has occurred (e.g., during a major version update of container tools), the existing on-disk storage is often incompatible with older library versions. In scenarios where a version mismatch occurs or storage is corrupted, the standard administrative approach is to clear the affected storage directory [1][8]. Specifically for CRI-O, the crio wipe command (or the legacy internal_wipe=true configuration) is used to clear storage when necessary, particularly after significant version upgrades or reboots [8][9]. If you need to move between versions, ensure you have backups of your images/containers, as you may need to re-pull images and recreate containers if the underlying storage format is no longer compatible with the target version. Always consult the official release notes for the specific container tools (Podman, Buildah, CRI-O) before performing major upgrades, as they will explicitly warn if an irreversible storage migration is part of the update.

Citations:


🌐 Web query:

site:github.com/containers/storage migration storage.conf backward compatibility database graphroot upgrade

💡 Result:

The storage.conf file is the standard configuration file used by the containers/storage library (which underpins tools like Podman) to define container and image storage settings, including the graphroot directory [1][2]. Regarding your points: 1. Backward Compatibility and Migration: The storage.conf file structure is designed to be relatively stable. Because it uses a TOML format [3], new options can often be added without breaking existing configurations. However, when migrating or upgrading, users must be aware that the configuration files operate under a strict override hierarchy [2][3]. Specifically, a file in a higher-precedence location (like $HOME/.config/containers/storage.conf) will completely override the system-wide defaults [3]. There is no native automatic "merge" feature for partial overrides [4], which means that if you update the system configuration (e.g., during an OS upgrade), custom local overrides may shadow the new system defaults [4]. 2. Graphroot Upgrade: The graphroot setting defines the primary read/write location for container storage [2]. If you decide to change or upgrade the graphroot path, you must handle the existing data manually, as the library does not perform automatic migration of stored images and containers between directories [1][5]. If you change this path on an SELinux-enabled system, you are required to manually update SELinux labels to ensure the new path has the same security context as the default /var/lib/containers/storage directory [1][5]. This is typically done using the following commands: semanage fcontext -a -e /var/lib/containers/storage /NEWSTORAGEPATH restorecon -R -v /NEWSTORAGEPATH Recommendations for Upgrades: - Before upgrading, audit your current /etc/containers/storage.conf and local user configurations. - Since there is no automatic merge, you may need to manually re-apply your custom settings to any updated template provided by the system package manager [4]. - When moving the graphroot, ensure the new location has sufficient storage and correct permissions/SELinux labels [1][5]. Existing containers and images will not be accessible from the new location unless you physically move (or rsync) the data from the old directory to the new one before pointing the configuration to it.

Citations:


🌐 Web query:

site:github.com/cri-o/cri-o containers/storage upgrade downgrade storage migration

💡 Result:

Managing upgrades, downgrades, and migration for containers/storage generally involves handling the underlying storage root and configuration files defined in /etc/containers/storage.conf [1]. There is no automated, universal migration path for changing storage drivers or migrating data between different storage architectures; instead, administrators typically manage these transitions through configuration updates, node reboots, and, in some cases, storage wipes [2][3]. Key considerations for storage management include: 1. Upgrade and Versioning: Upgrades of the containers/storage library are typically handled through the container runtime (e.g., CRI-O) via dependency bumps [4][5][6]. CRI-O may use internal version tracking to determine if storage should be wiped during an update to ensure compatibility with new formats [2]. This is often controlled by settings such as --clear-storage-on-update, which must be explicitly configured to prevent unintentional data loss [2]. 2. Downgrades and Compatibility: Downgrading container runtimes and storage libraries can be complex and is not always supported due to breaking changes in configuration formats or underlying drivers [3]. If a downgrade becomes necessary, it may require reverting to previous configuration files and performing a full node reboot to ensure that all mount points and processes are correctly re-initialized [3]. 3. Migration Strategy: - Configuration: If you are changing storage drivers (e.g., from devicemapper to overlay), you must modify /etc/containers/storage.conf [1][7]. Note that certain drivers, such as overlay, may have specific requirements for the underlying filesystem (e.g., XFS with prjquota enabled) [7]. - Storage Wipe: For significant transitions or when upgrading across major version gaps where data formats change, the standard approach is often to wipe the existing storage (found by default at /var/lib/containers/storage) and allow the runtime to repopulate the images and containers [2][8]. This should only be done if you can afford to lose local image data and container writeable layers. - Verification: After any storage change, verify the mount points and ensure that the runtime can correctly access the graphroot and runroot locations [9][1]. For specific, critical environments, it is recommended to test storage migration strategies in a non-production CI environment before applying changes to production nodes [10][11]. Consult the man page for containers-storage.conf (man 5 containers-storage.conf) for authoritative details on available storage options and configuration parameters [1].

Citations:


🏁 Script executed:

#!/bin/bash
set -eu
file="enhancements/node/downstream-container-libs-vendoring-for-cri-o.md"
sed -n '330,365p;399,420p' "$file"
sed -n '245,285p' "$file"
rg -n -i -C 2 --glob '*.md' \
  'containers/storage|storage migration|storage\.conf|containers\.conf|MachineConfig|rollback process|persistent state|upgrade.*downgrade' \
  enhancements/node enhancements/machine-config enhancements/hypershift | head -250

Repository: openshift/enhancements

Length of output: 36208


Define storage compatibility and rollback requirements before declaring upgrades transparent.

containers/storage does not provide a general safe downgrade migration. A CRI-O rollback does not roll back node-local storage state. Custom storage.conf and containers.conf can also require updates. Add tests for persistent storage, custom configuration, mixed RHEL streams, upgrades, and rollbacks. Document migration, backup, recovery, and any required storage wipe or recreation steps in the support procedures.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@enhancements/node/downstream-container-libs-vendoring-for-cri-o.md` around
lines 381 - 395, Expand the “Upgrade / Downgrade Strategy” and “Version Skew
Strategy” sections to define storage compatibility and rollback requirements
rather than claiming upgrades are transparent. Document migration limits, backup
and recovery procedures, custom storage.conf/containers.conf handling, and any
required storage wipe or recreation steps; add coverage for persistent storage,
custom configuration, mixed RHEL streams, upgrades, and rollbacks.

…ng, scenarios, and embargo process

- Reorganize workflows into numbered scenarios (1-4) for clearer reference
- Add RHEL-to-container-libs version mapping table based on consolidation proposal
  and OCP release page observations
- Add detailed embargoed CVE process (openshift-priv workflow)
- Add security fix SLA targets (to be confirmed with security response team)
- Add note about OCP z-stream RHEL minor version shifts
- Link to Container-tools Version Consolidation proposal

Co-Authored-By: Ayato Tokubi <atokubi@redhat.com>
Assisted-by: Claude Code <https://claude.com/claude-code>
Signed-off-by: Qi Wang <qiwan@redhat.com>
@QiWang19
QiWang19 force-pushed the crio-vendor-version branch from 6948012 to dbb1f6e Compare August 6, 2026 06:02
@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@enhancements/node/downstream-container-libs-vendoring-for-cri-o.md`:
- Around line 76-83: Align the Non-Goals item about downstream container-libs
maintenance with the feature-backport workflow in Scenario 4: either explicitly
include approved feature backports alongside bug fixes, or remove the
feature-backport workflow from Scenario 4. Update the corresponding scope
language consistently without changing unrelated goals.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7fc516d8-b55e-4f4c-9b87-5ab6a2f259a1

📥 Commits

Reviewing files that changed from the base of the PR and between c79913a and dbb1f6e.

📒 Files selected for processing (1)
  • enhancements/node/downstream-container-libs-vendoring-for-cri-o.md

Comment on lines +76 to +83
### Non-Goals

1. Establishing the downstream container-libs fork itself (repository creation, governance, CI) — that is a separate effort.

2. Changing how other OpenShift components (Podman, Buildah, Skopeo) vendor container-libs. These components also consume container-libs and may need the same approach in the future for consistency, but that is out of scope here.

3. Maintaining full upstream rebase parity in downstream container-libs — only bug fixes are expected.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Align the stated scope with Scenario 4.

The Non-Goals section says that only bug fixes are expected at Line 82. Scenario 4 also defines feature backports at Lines 161-168. Update the non-goal or remove the feature-backport workflow.

Proposed wording
-3. Maintaining full upstream rebase parity in downstream container-libs — only bug fixes are expected.
+3. Maintaining full upstream rebase parity in downstream container-libs — only targeted bug fixes and approved backports are expected.

Also applies to: 161-168

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@enhancements/node/downstream-container-libs-vendoring-for-cri-o.md` around
lines 76 - 83, Align the Non-Goals item about downstream container-libs
maintenance with the feature-backport workflow in Scenario 4: either explicitly
include approved feature backports alongside bug fixes, or remove the
feature-backport workflow from Scenario 4. Update the corresponding scope
language consistently without changing unrelated goals.

@QiWang19 QiWang19 changed the title Downstream container-libs vendoring for CRI-O supersedes #2066 supersedes #2066 Downstream container-libs vendoring for CRI-O supersedes #2066 Aug 6, 2026
@QiWang19 QiWang19 changed the title supersedes #2066 Downstream container-libs vendoring for CRI-O supersedes #2066 supersedes #2066 Downstream container-libs vendoring for CRI-O Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants