Stability / Migration tools

[Ericson2314]

Where should we keep a list of all the things that would be nice to have? Is this a good place?

None of these are easy to implement, but the research has at least (with the exception of the last) already been done.

Dealing with many versions

• Deprecated reexports (individual items and entire modules) https://gitlab.haskell.org/ghc/ghc/-/issues/4879

• CPP checking multiple combinations https://www.youtube.com/watch?v=YupkE1vsZ4o, https://www.cs.cmu.edu/~ckaestne/pdf/FOSD10-typechef.pdf, https://web.engr.oregonstate.edu/~erwig/CC/index.html (Rust also interested https://github.com/rust-lang/rfcs/blob/master/text/1868-portability-lint.md)

• Enriched hi files preserving configuration conditionality info. Can align with CPP variable "model" to make sure downstream packages abide by upstream CPP, wherever it leaks into the interface.

• Extract backpack "signatures" from regular modules. Good to avoid relying on hi files too much.

• Compare two signatures / hi files, lint PVP violations. (cabal-diff on steroids.)

• Leverage all the above to check cabal metadata doing an "all combinations" type check.

• Leverage all of the above to CI plausible max bound increases when new versions are released, if CI passes notify maintainers/trustees.

• Less need for breakage in the first place

  • https://gitlab.haskell.org/ghc/ghc/-/issues/19507 Permit bundling pattern synonyms with type synonyms

Avoiding churn in the first place

• Private dependencies, to make version solving more forgiving Private dependencies haskell/cabal#4035

• Better exporting https://gitlab.haskell.org/ghc/ghc/-/issues/19507 and other issues.

• Make Orphans OK so dependencies can be fewer (and thus downstream breakage less cascading) https://gitlab.haskell.org/ghc/ghc/-/wikis/Rehabilitating-Orphans-with-Order-theory

[noughtmare]

Another option of resolving the orphan situation would be to make GHC's type class overlapping mechanism less permissive. Hugs has a mechanism that allows it to quickly check if there are any conflicting instances (in the world semantics). The only problem is that it reduces the variety of overlapping type classes that you can define. I think it would be interesting to see how many packages on Hackage make use of unusual type class hierarchies that aren't accepted by Hugs' overlap checker. I think it may well be the case that all existing code could at least be rewritten to satisfy that more restrictive system.

See a very short explanation about the actual algorithm here and a related later comment.

[Ericson2314]

@noughtmare those are good things, and I should incorporate them in the wiki page, but I think if this is the right place for such a tracking issue, it should focus on more the why more than how, at least until we get from benefits to costs. Does that make sense?

[telser]

Where should we keep a list of all the things that would be nice to have? Is this a good place?

I think this is a great place to collect up ideas! This is somewhat related to the work-in-progress proposals we already have in the repo. At least part of the idea there was to collect up things we would like to refine into HF Tech Track proposals. You're giving me the idea that if we kept things as a list of ideas then we would have such a list and if something needed to become a proposal to HF we could track that.

None of these are easy to implement, but the research has at least (with the exception of the last) already been done.

Dealing with many versions

• Deprecated reexports (individual items and entire modules) https://gitlab.haskell.org/ghc/ghc/-/issues/4879

I currently have a list of ideas/issues that I'm attempting to get to a similar level of problem/solution statement before bringing to the wider community. This is definitely on that list, which if nothing else tells me I'm at least in the right general direction there.

• CPP checking multiple combinations https://www.youtube.com/watch?v=YupkE1vsZ4o, https://www.cs.cmu.edu/~ckaestne/pdf/FOSD10-typechef.pdf, https://web.engr.oregonstate.edu/~erwig/CC/index.html (Rust also interested https://github.com/rust-lang/rfcs/blob/master/text/1868-portability-lint.md)
• Enriched hi files preserving configuration conditionality info. Can align with CPP variable "model" to make sure downstream packages abide by upstream CPP, wherever it leaks into the interface.

I've wondered about how much breakage CPP, as well as cabal flags actually cause as I do not think I've hit such breakage before. Which surprises me. Sounds like you have encountered this though?

• Extract backpack "signatures" from regular modules. Good to avoid relying on hi files too much.

This would be interesting to me as well as I have no "practical" usage of backpack under my belt yet, though I'm a bit concerned about the ahem interface for it and bringing the community up to speed there.

• Compare two signatures / hi files, lint PVP violations. (cabal-diff on steroids.)

Are you thinking of something similar to the last bullet point on this list?

• Leverage all the above to check cabal metadata doing an "all combinations" type check.
• Leverage all of the above to CI plausible max bound increases when new versions are released, if CI passes notify maintainers/trustees.

In the places already linked I have some ideas for leveraging CI/hackage to help with these things and it appears at least some movement towards getting email notifications is underway as part of summer of code.

Avoiding churn in the first place

Thank you for these in general. I hope we can continue to expand on tooling other than putting everything on GHC and those developers.

• Make Orphans OK so dependencies can be fewer (and thus downstream breakage less cascading) https://gitlab.haskell.org/ghc/ghc/-/wikis/Rehabilitating-Orphans-with-Order-theory

I must say I went down a rabbit hole of reading to understand this (sorry for the late reply as a result!), and it sounds both difficult to get and exciting for what it could mean once it went in place.

[Ericson2314]

I currently have a list of ideas/issues that I'm attempting to get to a similar level of problem/solution statement before bringing to the wider community. This is definitely on that list, which if nothing else tells me I'm at least in the right general direction there.

Glad to hear it! That one I think is the most wanted ATM, and so a good place to start.

I've wondered about how much breakage CPP, as well as cabal flags actually cause as I do not think I've hit such breakage before. Which surprises me. Sounds like you have encountered this though?

Mmm the problem I am trying to solve is CPP requiring big brain to think through all the combinations (and extensive CI, a slow debug cycle, to test all those assumptions).

With the things I proposed, all library breakage would be trivial to work around with CPP because (a single version of) the compiler would tell you whether you got it right or wrong.

This would make library maintenance a lot less stressful for me! Things I need to think carefully about -> meditative busywork; that's quite an improvement!

This would be interesting to me as well as I have no "practical" usage of backpack under my belt yet, though I'm a bit concerned about the ahem interface for it and bringing the community up to speed there.

This would be "implicit backpack". The user need not even notice that they are using backpack. It just exists to support "type checking all combinations simultaneously".

• Compare two signatures / hi files, lint PVP violations. (cabal-diff on steroids.)

Are you thinking of something similar to the last bullet point on this list?

It is definitely part of "Introduce language and compiler features to facilitate evolution of libraries without breaking users". But I don't think this specific mechanism is on that list.

In the places already linked I have some ideas for leveraging CI/hackage to help with these things and it appears at least some movement towards getting email notifications is underway as part of summer of code.

Glad to hear it!

I must say I went down a rabbit hole of reading to understand this (sorry for the late reply as a result!), and it sounds both difficult to get and exciting for what it could mean once it went in place.

:) Hope the excitement over the possibilities made the rabbit hole worth it!

[bergmark]

Make Orphans OK so dependencies can be fewer (and thus downstream breakage less cascading) https://gitlab.haskell.org/ghc/ghc/-/wikis/Rehabilitating-Orphans-with-Order-theory

This would be very useful to have. E.g. in aeson we struggle to decide which packages are "worthy" of having their batteries included.

From recent experience with Rust I just thought, why not copy what they are doing? E.g. fix (or implement something similar to) Cabal's "flags" (called "features" in rust) which have sometimes been used for the same purpose, even though they don't provide a good solution. Allow dependencies to specify required features and start treating them as additive. Perhaps the biggest downsides to the Rust approach is that it isn't clear (to me) that it would be less work to implement, and that a thorough test suite for a library would need to test every combination of features. Not sure it would solve the total ordering issue either!

Orphans are an issue, but I'm not sure they are the only issue. E.g. do you want to use warp with tls? Then you need to know to pull in warp-tls.

[gbaz]

I'm interested in contributing to a discussion on orphans. I agree that a "forward decl" of allowed locations for downstream instances is the right thing (and suggested this idea to edward yang a long time ago, iirc, because orphans are potentially a sticky bit for backpack). I think there's two parts here. First, extend ghc with some notion of a "forward decl" of an instance, such that it can handle it in a good way, compared to what it does now. Second, figure out syntactically where the forward decl lives.

I don't like the idea of putting it in a cabal file, since it makes reference to classes and datatypes, which cabal files have no notion of.

However, putting it in a module then requires that a module make reference to a package, which a module currently (barring package qualified imports) has no notion of.

I think that the right thing might be instead to have modules declare forward-decls of types vis a vis other module names -- something that makes sense even in the context of a single package.

Then when compiling other packages, if the module names match the forward-decl, it triggers. (And additionally, we have a restriction on uniqueness of module names when those modules contain implementation of forward decls).

[Ericson2314]

Glad you are interested in it.

Do you think it might better to make a separate thread to discuss the details so this can stay a more meta-thread tracking all such "technical solutions to social problems"?

I don't like the idea of putting it in a cabal file...

My hunch is we approaching this from two different angles but basically agree.