Initial Haskell support

[pbrisbin]

What would you like Renovate to be able to do?

I would like renovate-bot to support Haskell, specially Stackage-based projects using an LTS resolver and extra dependencies from Hackage:

Resolver

An overall configuration point in a Stackage-based Haskell project is the resolver. For example, resolver: lts-major.minor. I would like renovate-bot to find this and suggest an update to (for example) resolver: lts-major.(minor + 1) when it becomes available. There are more resolvers than these LTS versions, but this is our initial use-case.

Hackage

While choosing a resolver defines a strict set of exact versions available, you can make additional dependencies available via extra-deps:

extra-deps:
  - some-package-x.y.z@sha256:sha,size

I would like renovate-bot to compare this to what's available on Hackage and suggest updates accordingly. There are other kinds of dependencies that can be specified here, and the sha256 verification is optional, but this is our initial use-case.

Did you already have any implementation ideas?

Yaml configuration can be declared directly in a file, named stack.yaml by default:

resolver: lts-16.25
extra-deps:
  - bugsnag-haskell-0.0.4.0@sha256:44a48e25058e45c8cc351355f6a601d9b4bc6c9895e61e84b4dcd3366302a26b,7395

(This path can be changed by configuration, so renovate-bot would have to support that)

Or stack.yaml can point to a different file, with slightly different keys:

# stack.yaml
resolver: ./snapshot.yaml

# snapshot.yaml
resolver: lts-16.25
packages:
  - bugsnag-haskell-0.0.4.0@sha256:44a48e25058e45c8cc351355f6a601d9b4bc6c9895e61e84b4dcd3366302a26b,7395

We would need renovate-bot to:

• Find stack.yaml[resolver], probably limit things to only work with lts-.* OR <filepath>
• If it's pointing to another file, follow that file and read resolver and packages
• Otherwise, read the resolver and exra-deps directly in this file
• Limit things further, to only Hackage dependencies in packages/extra-deps
• Outcome: resolver = lts-16.25, hackage = [bugsnag-haskell, 0.0.4.1]
• Look up latest LTS resolver by following and reading https://www.stackage.org/lts?_accept=application/json
• Outcome: lts-16.27
• Look up latest for each Hackage item, by following and scraping https://hackage.haskell.org/package/:name
• Outcome: bugsnag-haskell-0.0.4.1
• Open a PR updating to this resolver and any new Hackage dependencies

  # snapshot.yaml
- resolver: lts-16.25
+ resolver: lts-16.27
  packages:
-   - bugsnag-haskell-0.0.4.0@sha256:44a48e25058e45c8cc351355f6a601d9b4bc6c9895e61e84b4dcd3366302a26b,7395
+   - bugsnag-haskell-0.0.4.1@sha256:...,...

A changelog can also be scraped from https://hackage.haskell.org/package/{name}-{version}/changelog, and the https://www.stackage.org/lts?_accept=application/json response includes any changed dependencies, for which changelogs can also be scraped. It would be amazing to include these changelog details in the renovate PR. We have a project that does some of this work here.

---

I'm willing to try and contribute this support, but I'm having trouble navigating the project enough to know where to add all the moving parts. Would you be able to offer some guidance, given the information above?

[rarkins]

Thanks @pbrisbin for the proposal.

Logically here's how we'd approach an implementation:

Datasources

Is there a need to access new APIs to retrieve versions of packages? In this case it seems like Hackage is needed. It would go into lib/datasource/hackage/*. There's a lot of existing datasources which could be used as inspiration. It looks like it would need to support digests too.

Versioning

Does Haskell use its own syntax to define versions or constraints/ranges? I'm seeing "pinned" versions with digests in your examples so maybe our "loose" versioning would work fine for things like 0.0.4.1.

Manager

The minimum you need to implement is a manager (e.g. lib/manager/stackage?) with and extractPackageFile() function. With a YAML parser, should hopefully be simple.

Is there any separate "artifacts" files that need updating, e.g. checksum or lock files?

Changelog

Currently our changelog only supports github and gitlab sources, but it sounds like they are published to the registry itself for Hackage? Do any packages also publish to a changelog file in their github source?

[pbrisbin]

Does Haskell use its own syntax to define versions or constraints/ranges?

It does, called the PVP. In short, it defines a 4-part version A.B.C.D where

• A.B is MAJOR (yes, together), incremented on breaking changes
• C is MINOR, incremented on backwards compatible additions
• D is PATCH (and is optional), incremented on bugfixes, etc

In case it's helpful, in my own packages, I set A to something like 0 or 1 and forget about it, then I just treat B.C.D as the typical Major.Minor.Patch of SemVer. This simplifies things for me, and the result still follows the PVP rules.

I'm seeing "pinned" versions with digests

Yes, a stack.yaml is effectively a "freeze file". Even the resolver bit is just an alias to a big set of pinned versions of available packages. So you are meant to put complete and explicit versions in it, when specifying your own in extra-deps. It doesn't support a bounds or ~0.4 or anything like that. The Digests are optional, but good practice.

(Note that this is just one way to specify dependencies in a Haskell project, so building this support in the way I describe would only support Haskell projects that choose this tooling. I'm happy to describe other approaches, if you're interested in understanding the broader space.)

Is there any separate "artifacts" files that need updating, e.g. checksum or lock files?

Ah yes, in fact. There is a stack.yaml.lock. I'm not sure how to update it after changing stack.yaml other than having a Haskell (stack) tool-chain available.

published to the registry itself for Hackage? Do any packages also publish to a changelog file in their github source?

Sort of both:

• The project will have a CHANGELOG in its source tree [example]
• The project defines "extra source file" in its packaging manifest [example]
• The registry, upon receiving the package, will look for files with certain names e.g. (CHANGELOG|ChangeLog)(.md)?
• That file is linked to from the main registry listing [example)

(Project README's are handled similarly.)

[rarkins]

I think we'll need a new versioning to handle A,B as major. Otherwise, if you use our loose versioning then it would consider changes to C to only be patch. It makes it easier though if you only need to deal with exact versions and not ranges.

If there's a lock file then it means we'd want to support "artifacts" updating like quite a few other managers.

For changelogs, they might "just work" if the majority of packages (a) define their source repo in metadata, and (b) include a changelog file. It doesn't look like we'll need to use the registry's changelogs.

I'd recommend doing this in this order, in separate PRs:

1. Versioning
2. Datasource
3. Extract + Artifacts

Steps 1 & 2 won't really achieve much on their own, but it's easier to do this incrementally than all in one go.

[pbrisbin]

Awesome, thanks. Granular steps make total sense, even if the first few don't actually get us there.

[github-actions]

Hi there,

You're requesting support for a new package manager. We need to know some basic information about this package manager first. Please copy/paste the new package manager questionnaire, and fill it out in full.

Once the questionnaire is filled out we will evaluate if adding support for this manager is something we want to do.

Good luck,

The Renovate team

[pbrisbin]

NOTE: I'm answering this questionnaire for the feature of specifically stack-related dependencies. If we want to automatically manage version bounds in a non-Stack project, or library making use of such bounds in its *.cabal file, things are more complicated.

New package manager questionnaire

Did you read our documentation on adding a package manager?

• I've read the adding a package manager documentation.

Basics

Name of package manager

Stack.

What language does this support?

Haskell.

How popular is this package manager?

Probably >50% of Haskell projects us it.

Does this language have other (competing?) package managers?

• Yes (Cabal, Nix, and to a lesser-extent Bazel)
• No

---

Package File Detection

What type of package files and names does it use?

• stack.yaml, or whatever $STACK_YAML holds
• Any other yaml path, if referenced from stack.yaml as a "custom snapshot"

What fileMatch pattern(s) should be used?

Can ${STACK_YAML:-stack.yaml} be used? If not, hard-coded stack.yaml should be sufficient -- let the user configure otherwise.

Is it likely that many users would need to extend this pattern for custom file names?

• Yes
• No

Possible, but not likely, IMO.

Is the fileMatch pattern likely to get many "false hits" for files that have nothing to do with package management?

No.

---

Parsing and Extraction

Can package files have "local" links to each other that need to be resolved?

Yes, stack.yaml can optionally use a "custom snapshot" by resolver: ./<other-yaml>.

Is there a reason why package files need to be parsed together (in serial) instead of independently?

I can't think of one.

What format/syntax is the package file in?

• JSON
• TOML
• YAML
• Custom (explain below)

How do you suggest parsing the file?

• Off the shelf parser
• Using regex
• Custom-parsed line by line
• Other

Does the package file structure distinguish between different "types" of dependencies? e.g. production dependencies, dev dependencies, etc?

• Yes, production and development dependencies
• No, all dependencies are treated the same

List all the sources/syntaxes of dependencies that can be extracted

In stack.yaml, resolver: x is either a named (remote) snapshot or path to a custom snapshot. For the case of a named (remote) snapshot, Renovate would not need to dig into specific dependencies but look at x itself and see if it's out of date and update it accordingly. The snapshot is itself the dependency in a way.

In the case of a path to a custom snapshot, you will find in that file a packages: key of the following syntax:

- {name}[-{version}[(@sha256:{sha}]]

- git: {repo}
  commit: {sha}

- ./<path>

stack.yaml (not the custom snapshot) can also contain a key extra-deps, which is a list of the same syntax to overlay over what came from above. These individual items would need to be inspected to be updated based on what they are.

Describe which types of dependencies above are supported and which will be implemented in future

---

Versioning

What versioning scheme does the package file(s) use?

Snapshots, use two formats:

• lts-{Major}.{Minor}
• nightly-{YYYY}-{MM}-{DD}

Non-git packages/extra-deps items follow PVP, which is A.B.C.D where:

• A.B is major, it must increment on breaking changes
• C is minor, incremented on non-breaking additions
• D is optional and patch

Does this versioning scheme support range constraints, e.g. ^1.0.0 or 1.x?

• Supports range constraints (e.g ^1.0.0 or 1.x)
• No

Not in this file. Items are specified fully, even down to checksum!

Is this package manager used for applications, libraries, or both? If both, is there a way to tell which is which?

• Applications
• Libraries
• Both (explain how to tell which is which)

A library commonly uses version bounds in its project files, for when it's build not by the developers of the project itself and perhaps across a wide range of dependencies. This feature-request is not concerned with such bounds.

An app commonly uses no version bounds in its project file, relying on the fact that it would only be built by the project maintainers itself, in a Stack-managed context, where the set of available dependencies is fixed by the stack.yaml.

I think either of these scenarios can be handled the same by Renovate.

If ranges are supported, are there any cases when Renovate should pin ranges to exact versions if rangeStrategy=auto?

N/A

---

Lookup

Is a new datasource required? Provide details

• Yes, provide details.
• No.

Available resolvers can be found on https://www.stackage.org. There are some JSON endpoints, but parsing HTML may be required at points.

Non-git packages/extra-deps would come from https://hackage.haskell.org. Again, there are some JSON endpoints, parsing HTML may be required, or you could download "the index" which is a gzipped tarball of information about al available dependencies. I would avoid this, personally, as it's a ton of data to pull and parse when you only need information about a subset of packages.

Will users need the capability to specify a custom host/registry to look up? Can it be found within the package files, or within other files inside the repository, or would it require Renovate configuration?

No.

Do the package files contain any "constraints" on the parent language (e.g. supports only v3.x of Python) or platform (Linux, Windows, etc) that should be used in the lookup procedure?

There is such a feature in Haskell, but I think Renovate could ignore it for now.

Will users need the ability to configure language or other constraints using Renovate config?

Not that I can think of.

---

Artifacts

Are lock files or checksum files used? Are they mandatory?

Yes. Not mandatory, but highly recommended.

If so, what tool and exact commands should be used if updating one or more package versions in a dependency file?

stack build --dry-run will recreate stack.yaml.lock after a change to stack.yaml.

If applicable, describe how the tool maintains a cache and if it can be controlled via CLI or env? Do you recommend the cache be kept or disabled/ignored?

• ~/.stack
• ./.stack-work

stack --work-dir x can control the latter, the former is fixed.

If applicable, what command should be used to generate a lock file from scratch if you already have a package file? This will be used for "lock file maintenance"

Same, stack build --dry-run

Other

Is there anything else to know about this package manager?

More details can be found in previous comments on this Issue.

[ysangkok]

Since part of this discussion is about adding a Hackage data source, I thought I'd link #31434, which contains such a data source. Since the data source is necessary for both Cabal and Stack support, maybe we can consolidate our efforts.

The PR was closed because we need to discuss requirements first. @secustor requested that I discuss the manager first, so that what I am focusing on in #31493 .

@rarkins Wouldn't it be ok if an initial version of the Hackage data source didn't support digests? Hash pinning is optional in stack.yaml. I'd argue that most dependencies are declared with a numeric version bound, and not with hash pinning.

@pbrisbin Are you still interested in implementing this? What do you think about the data source I drafted, do you think it could help you?

[viceice]

Since part of this discussion is about adding a Hackage data source, I thought I'd link #31434, which contains such a data source. Since the data source is necessary for both Cabal and Stack support, maybe we can consolidate our efforts.

The PR was closed because we need to discuss requirements first. @secustor requested that I discuss the manager first, so that what I am focusing on in #31493 .

@rarkins Wouldn't it be ok if an initial version of the Hackage data source didn't support digests? Hash pinning is optional in stack.yaml. I'd argue that most dependencies are declared with a numeric version bound, and not with hash pinning.

@pbrisbin Are you still interested in implementing this? What do you think about the data source I drafted, do you think it could help you?

you should start on versioning. then datasource and finally on manager.

[ysangkok]

@viceice Your order is inconsistent with what I was suggested in #31434 (which is to discuss the manager first). Also, not everyone uses PVP. So I don't understand why versioning need to be done first.

[wolfgangwalther]

Your order is inconsistent with what I was suggested in #31434 (which is to discuss the manager first).

I read #31434 (comment) differently. It says "discuss manager first, then implement version, datasource, manager in that order in separate PRs".

[ysangkok]

I think #32298 with PVP support might get merged soon-ish, and I have the datasource PR ready, which I will submit at that point.
It will use Hackage endpoints like https://hackage.haskell.org/package/base.json, and it would only use the keys of the object. This endpoint isn't currently document anywhere, but since I work on hackage-server, I could add that documentation. I'd probably add a manually written OpenAPI spec.
The datasource PR would be fairly small it's basically just a map over the JSON keys, with little branching.

[pbrisbin]

Hi there- I was trying out the hackage data-source with a custom-manager rule just now, and it almost worked.

Unfortunately, the logs [possibly-private link] show:

INFO: PVP can't handle this range strategy.
{
  "rangeStrategy": null
  "currentValue": "0.17.0.0"
  "newVersion": "0.18.0.0"
}

This looks great, updating 0.17.0.0 to 0.18.0.0 is exactly what I wanted renovate to do, but is there some way to solve this range strategy error?

[ysangkok]

This looks great, updating 0.17.0.0 to 0.18.0.0 is exactly what I wanted renovate to do, but is there some way to solve this range strategy error?

While searching for how to configure this, I found this PR:

• fix(manager/custom): Support range strategy with custom managers #33800

So I don't think this is a PVP/Haskell specific issue. When that PR is merged, you should be able to configure rangeStrategy to widen, which is the only strategy currently supported by PVP in Renovate.

EDIT: It looks like it won't actually be configurable? Not sure from reading the code whether the auto is passed to versioning. It if is, there will still be an issue. Previously discussed why I didn't accept auto in versioning: #33142 (comment)

[pbrisbin]

Hmm, I guess I'm just missing the context on what a range-strategy is or why it matters here. As a user, I just want renovate to replace currentValue with newVersion here, but it can't because ranges are somehow involved?

[ysangkok]

As a user, I just want renovate to replace currentValue with newVersion

The Haskell related code I wrote has three parts: manager (Cabal), datasource (Hackage) and versioning (PVP).

You have a custom manager, so you're obviously not using the Cabal code. It makes sense for you to use the Hackage code, but I wrote the PVP code mainly to handle version ranges, since that is commonly used in Cabal files.

Since you're not interested in any comparison of version numbers (as ranges do), maybe it would sense to swap out the versioning. The Hackage datasource should work with other versioning too, it just defaults to using PVP.

For example, there is regex versioning which sounds fairly generic. All it really has to be configured to, is to recognize which component in the version has changed. If it's early in the version number, it's a major change. If it's later, it's a minor change. Another option might be loose, which sounds like it would parse a PVP version number successfully too, with no further configuration needed.

If regex/loose versioning don't work, maybe this is a use case that could be better supported if I added the replace, bump or pin range strategies to PVP, which seem closer to what you want to do.

I'd like to hear if regex/loose work, first, though.

[pbrisbin]

Thanks for the suggestion. I changed the versioning to regex and it didn't seem to do anything. The only mention of this dep in the logs is the following DEBUG message:

      {
        "datasourceTemplate": "hackage",
        "depNameTemplate": "fourmolu",
        "deps": [
          {
            "currentValue": "0.17.0.0",
            "datasource": "hackage",
            "depName": "fourmolu",
            "packageName": "fourmolu",
            "replaceString": "ENV FOURMOLU_VERSION=0.17.0.0\n",
            "skipReason": "invalid-value",
            "versioning": "regex:^(?<major>[0-9]+\\.[0-9]+)\\.(?<minor>[0-9]+)\\.(?<patch>[0-9]+).$",
            "warnings": [],
            "updates": []
          }
        ],
        "matchStrings": [
          "ENV FOURMOLU_VERSION=(?<currentValue>.*?)\\n"
        ],
        "packageFile": "fourmolu/Dockerfile",
        "versioningTemplate": "regex:^(?<major>[0-9]+\\.[0-9]+)\\.(?<minor>[0-9]+)\\.(?<patch>[0-9]+).$"
      },

I assume invalid-value is why nothing's happening, but I'm not really sure which value is invalid or why.

This is likely nothing to do with your code though?

EDIT: trying loose now.

[pbrisbin]

OK, loose seems to work 🎉

So fair enough, I can run with that in this case. However, I still think there's a gap in the PVP versioning implementation worth addressing.

if I added the replace, bump or pin range strategies to PVP, which seem closer to what you want to do

The error I got was,

INFO: PVP can't handle this range strategy.
{
  "rangeStrategy": null
  ...
}

So the missing support is not for any of those, it's for null.

Perhaps its confirmation bias, but this makes sense to me. When used in this context of a custom matcher, there just is no ranges involved. So if some interface somewhere requires one, it passes null. If that's true, I don't think it's unreasonable to think that all versioning implementations should support null (by doing nothing with it, presumably). It feels like the PVP implementation is failing on it unnecessarily somewhere.

regex and loose both list "no range support" on their docs, but they don't fail when used here, so clearly they support being given null somehow.

[ysangkok]

So the missing support is not for any of those, it's for null.

I think the null is a bug fixed in #33800. There is a default global range strategy in Renovate. The PR provides more details on this. It may be that the default is somehow set to auto, which turns to replace (see PR), but that could still be overridden using packageRules. That said, if we can construct a realistic scenario that fails with an alternative range strategy, I'd definitely be more motivated to add it.

[pbrisbin]

if we can construct a realistic scenario that fails with an alternative range strategy, I'd definitely be more motivated to add it.

That makes sense, but are you saying "usable with a custom manager" isn't a realistic scenario to support? Can I ask why?

[ysangkok]

"Realistic" was a bad word to use. For me, it's really more a question of the expected popularity of the proposed feature.

Are you proposing that we expand the PVP range strategy support such that people won't need to specify loose versioning when using a custom manager? If so, I don't think it's worth the effort and code complexity.

Custom managers is an advanced use case. It's not something that is used out of the box, it's something you explicitly have to enable and configure. As you have found out, it seems that you need to override versioning in addition to specifying the data source. I don't think this is a horrible user experience, as people using custom managers are already into the weeds of Renovate. As I see it, your use case is already supported in Renovate, it just has to be configured. Previously, I used the wording "fails with an alternative range strategy". I don't think PVP is failing with an alternative range strategy. The reason is that PVP isn't even the right choice when no ranges are involved. As you previously said, things seem to work when configuring loose versioning. I don't have the statistics, but my suspicion is that maybe only 10% of repos using Renovate are using custom managers. The popularity of a potential feature should be kept in mind when choosing what to work on.

The Cabal manager, on the other hand, is enabled by default and works without any configuration. The default range strategy for the Cabal manager is widen, as that seems to be the norm around the Haskell ecosystem. The PVP versioning is was written to support this use case. I am hoping it is applicable to like 90% of Haskell packages hosted on Github.

I am trying to address the most pressing issues I see in the Haskell ecosystem. I have spent a lot of time ensuring that packages stay in Stackage Nightly. That's why I added this code to Renovate. I've added documentation to the haskell-action maintained by Andreas Abel, to help Renovate achieve wider adoption. If we can, in addition, get Haskell-CI to nudge users in the direction of Renovate, even more future repos could get automatic dependency bumping out of the box. That's something I see as having a larger potential impact. I need to talk to Oleg (the maintainer of Haskell-CI) about this...

I hope that this clarifies some of what motivates me. I am trying not to sound too grumpy, not sure if I am succeeding. I don't own this code, so of course anybody can submit PRs, and I might even help reviewing.

[pbrisbin]

Thanks for the detailed reply, as someone who also maintains software I understand wanting complexity to be properly motivated by a common enough use-case. And I don't think you sound grumpy.

If it seems like I'm pushing back, it's not because I think you're being unreasonable or that I think you're judging some complexity trade-off incorrectly, it's just because I'm still confused as to why certain range support is the complexity required for the user-facing outcome I want, which is just for PVP to work in a custom manager -- where there are no ranges to be found.

You're right, tactically speaking. If making PVP specifically support something only needed so it can be used in a custom manager helps 10% of users, then efforts should go elsewhere. The only thing I would say is that I'm trying to raise (poorly) a more strategic and architectural issue here, that could be higher value. Why does use in a custom manager require range support?

I just feel like, if a renovate user is configuring a dependency anywhere for any reason, and that dependency in reality uses some versioning that renovate supports, but they can't use that versioning for some technical reason, that's a problem with the technical design. Maybe that problem truly only impacts the intersection of custom managers and PVP, but maybe it's broader, maybe it has knock-on effects, maybe it's a smell worth listening it. 🤷

I realize we've been sort of, "arguing in the mentions" of the project where you don't own these areas of design, so I have no expectations of this going anywhere, I just find it all interesting to discuss. So thanks for engaging, and your continued Haskell work, it's been very helpful to me keeping these dependencies up to date.

[ysangkok]

a problem with the technical design [...] maybe it's broader, maybe it has knock-on effects

• Looking at the API for versioning, it seems wrong to me that methods like isGreaterThan and isLessThanRange? are defined on the concrete versioning itself. Many of these operations could be defined in terms of each other. Defining them centrally would enable enforcement of algebraic laws (like not ∘ lt = gte).
• Comparison functions could operate on abstract types. The versioning would just have to parse strings and return concrete types like PVPRange or PVPVersion that would implement generic interfaces.
• It's also an issue that getMajor returns an IEEE 754 float, since that means 0.20 shows as 0.2 for PVP.
• I could go even further, and question the use of Handlebars/JSONata in configuration. Why not allow arbitrary provably finite computation in custom managers, ala Dhall?

The problem with all of these blue sky ideas is that they span across Renovate, which means they'd need a massive amount of investment to get anywhere. And I am not even sure that developers are philosophically on board.