There's a zoo of dependency management systems in different programming environments, so at first it looks like every one of them is unique. But actually there's a simple pattern:
- Those that keep all dependencies centralized on a file system and re-use them among projects (Python's site-packages)
- Those that separate dependencies per project (Java+Maven)
- And mixed approaches. E.g. JavaScript's NPM can work in both modes.
Similar distinction is in the Linux ecosystem. E.g. in Ubuntu you can install DEB a package globally, or you can use Snap to install it separately along with its dependencies.
So who's right?
Your use cases dictate the preference
Your preference will depend on the typical use case:
- If you write software, managing shared dependencies is VERY hard: if 2 packages need difference versions of the same dependency, there's no simple solution*. So you isolate every project by keeping the dependencies separated.
- On the other hand you don't want to install dependencies from scratch each time you write a single-use script to crunch data. You're not interested in distributing the project - you just need to get the insights quickly. And you probably use more or less the same packages anyway. All you want is install them once and jump into scripting right away.
*The complicated solution is used by Linux operating systems: each OS keeps repositories with the typical high-quality dependencies that they hand-pick. This means that the package provider can't just upload a new version and have it available for everyone. The committee needs to decide that this version is worthy. Yes, we get high quality, backward and forward compatible stuff, but with a lot more efforts and longer timelines.
Technical limitations of platforms
What's interesting is that there's no technical limitation in the platforms themselves:
- In Java you can put typical dependencies to JAVA_HOME/lib/ext. And you'll have those available w/o further configuration.
- In Python you can put each dependency into its own folder, and list the specific ones in PYTHONPATH for each project separately.
So it's more of a cultural thing - not a technical problem.
Trends in Java and Python
What's even more interesting is the trends:
- Applied folk (like scientists) are trying to write software these days. And having a single site-packages is very problematic for them. So much so that even for research purposes they almost exclusively rely on virtualenv and co.
- From the other end Java is trying to enter the ad-hoc scripting: it has a REPL jshell, can run Java source files w/o compilation, can import modules, and soon it'll allown main method outside of classes.
The convergence is coming! And maybe it'll change the tooling too.
Python and its backward compatibility issues
There's an additional obstacle for Python though. Regardless of the dependency management, Python itself isn't dependable for serious software development. It shamelessly breaks backward compatibility. So even if we get build tools with multi-PYTHONPATH support, there's still no way of getting rid of horrendous virtualenv and conda. You still need projects to use different versions of Python. And sometimes even test against multiple versions 🤷♂️
NodeJS NPM
NodeJS NPM chose an interesting path: you can have global and per-project dependencies. Usually the global ones are used for tools and not for the typical libraries & frameworks. This seems like another viable evolution direction for Python too.
But that's not all: with NPM the dependencies themselves can reference different versions of THEIR dependencies. And they will co-exist in a single project. So 2 libraries can use different versions of the same dependency at the same time.
The real issue that's being solved here is low-quality dependencies: libraries not keeping backward compatibility and libraries that themselves have a lot of dependencies.
To get notifications about new posts, Watch this repository.
There's a zoo of dependency management systems in different programming environments, so at first it looks like every one of them is unique. But actually there's a simple pattern:
Similar distinction is in the Linux ecosystem. E.g. in Ubuntu you can install DEB a package globally, or you can use Snap to install it separately along with its dependencies.
So who's right?
Your use cases dictate the preference
Your preference will depend on the typical use case:
*The complicated solution is used by Linux operating systems: each OS keeps repositories with the typical high-quality dependencies that they hand-pick. This means that the package provider can't just upload a new version and have it available for everyone. The committee needs to decide that this version is worthy. Yes, we get high quality, backward and forward compatible stuff, but with a lot more efforts and longer timelines.
Technical limitations of platforms
What's interesting is that there's no technical limitation in the platforms themselves:
So it's more of a cultural thing - not a technical problem.
Trends in Java and Python
What's even more interesting is the trends:
The convergence is coming! And maybe it'll change the tooling too.
Python and its backward compatibility issues
There's an additional obstacle for Python though. Regardless of the dependency management, Python itself isn't dependable for serious software development. It shamelessly breaks backward compatibility. So even if we get build tools with multi-PYTHONPATH support, there's still no way of getting rid of horrendous virtualenv and conda. You still need projects to use different versions of Python. And sometimes even test against multiple versions 🤷♂️
NodeJS NPM
NodeJS NPM chose an interesting path: you can have global and per-project dependencies. Usually the global ones are used for tools and not for the typical libraries & frameworks. This seems like another viable evolution direction for Python too.
But that's not all: with NPM the dependencies themselves can reference different versions of THEIR dependencies. And they will co-exist in a single project. So 2 libraries can use different versions of the same dependency at the same time.
The real issue that's being solved here is low-quality dependencies: libraries not keeping backward compatibility and libraries that themselves have a lot of dependencies.
To get notifications about new posts, Watch this repository.