Please reconsider the Python support policy

[robsdedude]

The Problem

Initially when discovering pyo3, I was really positively surprised to find that it even supported legacy Python versions. However, recently support for Python 3.7 was dropped.

This is a big pain for me developing an extensions module for a Python library. The library follows SemVer and has an LTS branch. That means support for Python versions is kept even after they go EOL.

How to tackle it in Python

It's somewhat straight forward to deal with python dependencies that drop support for said EOL Python versions. Dependencies declared in pyproject.toml can be different versions for depending on the Python version. E.g.,

dependencies = [
    "Sphinx >= 8.1.3; python_version <= '3.10'",  # dropped support for Python 3.10 in 8.2.0
    "Sphinx >= 9.0.4; python_version == '3.11'",  # dropped support for Python 3.11 in 9.1.0
    "Sphinx >= 9.1.0; python_version >= '3.12'",
]

Trying to solve it it with build.rs

In Rust this isn't possible. Cargo isn't aware of the Python version at the time of having to select the pyo3 version to build against. So I'd have to stick to the newest pyo3 version which is supporting the oldest Python version I want to support. That's an option, but an unfortunate one as it means users of my library will not have access to security fixes (such as the recent possible oob read that got fixed in 0.29.0) even if they are on a non-EOL Python version.

Using build.rs to check which Python version to build for doesn't seem to be an option either as Cargo doesn't respect feature selection in build.rs:

Note that this does not affect Cargo’s dependency resolution. This cannot be used to enable an optional dependency, or enable other Cargo features.
—https://doc.rust-lang.org/cargo/reference/build-scripts.html#rustc-cfg

Trying to solve it with Cargo features

Maturin does allow to select package features depending on the Python version. Something like

# pyproject.toml
# [...]
[tool.maturin]
features = [
    { feature = "pyo3_0_28", python-version = "<3.8" },
    { feature = "pyo3_latest", python-version = ">=3.8" },
]
# [...]

But that doesn't get me anywhere either because adding this to my Cargo.toml

# [...]
[features]
pyo3_latest = ["pyo3"]
pyo3_0_28 = ["dep:pyo3_0_28"]

[dependencies]
pyo3 = { version = "0.29.0", optional = true }
pyo3_0_28 = { package = "pyo3", version = "0.28.3", optional = true  }
# [...]

Leaves me with the following compilation error

error: failed to select a version for `pyo3-ffi`.
    ... required by package `pyo3 v0.28.3`
    ... which satisfies dependency `pyo3_0_28 = "^0.28.3"` of package `neo4j-rust-ext v0.1.0 (/home/rouven/Neo4j/code/neo4j-python-driver-rust-ext)`
versions that meet the requirements `=0.28.3` are: 0.28.3

package `pyo3-ffi` links to the native library `python`, but it conflicts with a previous package which links to `python` as well:
package `pyo3-ffi v0.29.0`
    ... which satisfies dependency `pyo3-ffi = "=0.29.0"` of package `pyo3 v0.29.0`
    ... which satisfies dependency `pyo3 = "^0.29.0"` of package `neo4j-rust-ext v0.1.0 (/home/rouven/Neo4j/code/neo4j-python-driver-rust-ext)`
Only one package in the dependency graph may specify the same links value. This helps ensure that only one copy of a native library is linked in the final binary. Try to adjust your dependencies so that only one package uses the `links = "python"` value. For more information, see https://doc.rust-lang.org/cargo/reference/resolver.html#links.

failed to select a version for `pyo3-ffi` which could resolve this conflict

[LilyFirefly]

Why are your library's users on really old versions of Python? PyO3 looks at the cryptography library's download statistics for old Python versions to decide when to drop EOL Pythons. See #4581 for the most recent discussion.

If your library is a commercial project, maybe it would be worthwhile maintaining a fork of PyO3 (or paying someone to do it for you) which maintains support for old versions or backports fixes from new versions.

[robsdedude]

Thanks for your input.

For me this is about SemVer. I very much dislike how Python and much of its ecosystem started to not care about SemVer any longer. I want to offer a stable library API with fixes as needed (LTS), and despite how hard it is I want to follow SemVer.

The question of how many users are still using 3.7 is interesting, but no matter the answer it wouldn't change what I'm aiming for. The LTS was cut with 3.7 support in it so I want to maintain it. The other option, of course, would be to EOL the LTS version and cut a new one. But I feel like there should be some path forward keeping support for older Python versions while receiving fixes for newer ones. To achieve the goal of SemVer, I'm fine with having to do some version dances depending on the Python version (like I already do with Python dependencies). I'm also fine with conditional code-paths. But I just don't see any way whatsoever to support a wide range of Python versions using pyo3 (besides very hacky things like editing/copying the Cargo.toml in some pre-processing step, or absolute behemoth approaches like maintaining a pyo3 fork as you suggested).

[LilyFirefly]

The other option, of course, would be to EOL the LTS version and cut a new one.

I think this is probably your best option. Python 3.7, 3.8 and 3.9 are all EOL and not receiving security fixes from upstream. Similarly, your users won't receive any new security fixes from PyO3 until you upgrade.

[alex]

Just for clarity: pyo3 follows semver, as a pre-1.0 project, changes in teh X.Y.Z Y position are not required to be compatible.

[robsdedude]

Just for clarity: pyo3 follows semver

Yes, as far as I'm aware it does. I never thought or stated anything else. All I'm saying is that PyO3 dropping support for Python versions (even when done right by bumping the version according to SemVer) can be painful for maintainers of downstream projects because they are forced to also introduce a breaking change or to stay behind due to Cargo limitations (cf. original post).

[mejrs]

Supporting old python (and rust) versions is also painful for us, which is why we do need to drop support at some point. For example it can be very difficult or impossible to implement apis around new python features and polyfill it for older older ones. Gating new functionality behind new python versions can also be very complex and confusing for users.

If it was up to me our support policy would be much stricter - only supported python versions and current rust -5, so Python 3.10+ and Rust 1.92+. The current policy is a big courtesy to some big python packages that depend on us and we have no interest in pushing it further.

The LTS was cut with 3.7 support in it so I want to maintain it.

In my eyes it doesn't really make sense to support a version that Python itself hasn't supported for several years now.

[latot]

Hi! mm, make a breaking change should not be an issue with SymVer, it only says when something breaks, but the number does not means you API changed, or there is something new, is only a marker to know when something that will break shows, nothing more, so there should not be any issue...

But I get that say move from 1.x.x to 2.x.x sounds like a major change, and could be confusing, but SymVer is not perfect, you could up the version with a very big note about from where the breaking change comes.

You may want to think a little about SymVer and project dependencies, if any dep make a breaking change, will you always change the major version? if any dep add a new feature which is exposed by your crate, will you also increase the minor version?