Use python-* naming convention for Python packages?

[certik]

For example, the yaml package is actually a python yaml package. It should not be called yaml, but either python-yaml, or pyyaml.

Though, it seems to me that the best policy is:

• if the name of the project is unique, for example numpy, then the package should just be called numpy
• if there is a C or C++ package, let's say netcdf4, and separate wrappers (also called netcdf4), then those should be called python-netcdf4. The same for readline / python-readline.

We sort of already follow this policy. But it would be nice to write this down somewhere, before closing this issue.

What do you think?

[ahmadia]

This was why I proposed namespace packages a while back. I'm not a huge fan of prepending python to everything, but I don't want to get in the way if everybody else thinks this is a good idea. I guess that makes me -0.1?

[certik]

@ahmadia just to clarify: you seem to agree with the policy that I proposed above? I.e. the -0.1 is to prepending python- to everything, which is not the policy I am proposing?

[cekees]

I have no problem with this as guidance of the form "If a python package
has the same name as another package, prepend 'python-' to the python
package name".

On Wed, May 27, 2015 at 1:01 AM, Ondřej Čertík [email protected]
wrote:

@ahmadia https://github.com/ahmadia just to clarify: you seem to agree
with the policy that I proposed above? I.e. the -0.1 is to prepending
python- to everything, which is not the policy I am proposing?

—
Reply to this email directly or view it on GitHub
#795 (comment).

[ahmadia]

Yeah, I don't like it. It means we're creating our own naming convention, as opposed to sticking to some established convention (PyPI, Ubuntu, Debian, etc...) It makes it harder to discover packages.

[dagss]

@ahmadia I don't remember the namespaced packages proposal, what is it?

Debian convention seems to be prefix python- to everything.

[certik]

@ahmadia you don't like my policy and you do not like the Debian policy of prefixing everything with python-? Besides namespaces, I don't know what else we could do, we either need to prefix python- to everything (like Debian) or we can only prefix it to resolve name conflicts (like my proposed policy above), but I think those are the only options.

You can read the Debian policy here: https://www.debian.org/doc/packaging-manuals/python-policy/ch-module_packages.html, section "3.3 Module Package Names":

Public modules used by other packages must have their binary package name prefixed with python-.

[ahmadia]

Yeah, I don't know what I want here. I had proposed that we do something like:

pkgs/python/numpy.yaml

Then the profile could do either python.numpy or in some other way refer to the numpy package in python. It's using the Python namespace concept as applied to the Debian naming approach. The official package name in HashStack might be python.numpy, but there should still be an easy way to refer to it as numpy.

[certik]

I see. I like this. Obviously, currently the pkgs/python directory is occupied by the python package itself, so we would need to resolve that.

[dagss]

I think the complexity and learning curve is higher than just doing straight-out python-numpy.

But in the interest of the discussion I guess it'd go like this:

python/scipy.yaml would contain

dependencies:
  build:
  - numpy  # same namespace
  - .blas     # '.blas' means "blas", while "blas" would mean "python.blas"

And in profile.yaml?

packages:
  python: # the namespace
    python: # python.python is the Python interpreter package
    numpy: # python.numpy
      blas: .blas  # root blas..
    scipy: # python.scipy
      numpy: numpy  # python.numpy

What I don't like is the mix of namespace and packages. So perhaps we require that everything is in exactly one namespace, and we say

packages:
  general:
    blas:
    zlib:
  python:
    numpy:
      blas: general.blas

But I'm not sold on this at all, I think I prefer python- prefix everywhere like other package managers.

[ahmadia]

My preferences goes something like: namespaced packages (+1), python everywhere (+0), mixed usage (-0.1).

[dagss]

I really can't vote until I see another fledged-out proposals for namespaced packages than the ones I drew up; whether it can work or not depends all on the details.

[ahmadia]

I like except:

  build:
  - numpy  # same namespace
  - .blas     # '.blas' means "blas", while "blas" would mean "python.blas"

I'd do it the other way. blas just means the top-level blas, while .blas means get the blas from the current namespace. The number of conflicts at the package level should actually be very small.

Regardless, this is basically how I see it working (with a few more details I hadn't imagined). Namespaces also make it much easier to refer to other people's hashstacks and allow you to use, for example, somebody's custom python build in your profile.

[dagss]

Right, so your proposal is

build:
- numpy # python namespace
- yaml # root namespace
- .yaml # python namespace

? And

packages:
  python: # the namespace
    python: # python.python is the Python interpreter package
    numpy: # python.numpy
      blas: blas  # root blas..
    scipy: # python.scipy
      numpy: numpy  # python.numpy

?

About somebody else's hashstack, I think it is more constructive to view every hashstack as a function with parameters, then you compose these together -- it's not really the same as namespaces at all IMO. But on the syntax level they probably overlap. We're onto something here but I feel it requires more consideration to get right.

I think I'm -1 on namespaces until after summer at least, this needs more thinking. If there's some combination with parametrizable profiles that give tangible benefits in return for the increased learning curve that's great.

If we adopt the python- prefix for all Python packages convention then moving to namespaces later could be a search and replace perhaps...

Finally, how about instead of this whole discussion, we do something like a "recommends" feature where the combination of two packages (say, "python" and "yaml") triggers a third, potentially arcanely-named package (like "yaml-python-wrapper"). Seems to fix this particular issue from a user perspective...

[ahmadia]

If we adopt the python- prefix for all Python packages convention then moving to namespaces later could be a search and replace perhaps...

I'm fine with this, or just keeping what we have and fixing collisions as they show up for now.

[dagss]

NOTE: Now that we soon support using {{numpy.version}}, we can't really introduce hyphens in package names.. at least without more work, {{python-numpy.version}} would invoke the minus operator. Needs to be python_numpy.version...

So that makes me a lot less keen on python- prefix everywhere.

Another namespace idea:

• Every package must live in something we might want to call "category" or "realm" instead of namespace because it's not nested and there's no root, there's only one level ("c", "python", "julia", etc.)
• You can always refer to the fully qualified name, python_python, python_numpy
• In general you can refer to a package without the namespace prefix. So "numpy" automatically turns into python_numpy or whatever.
• If there is a collision, like yaml, you have to say either c_yaml or python_yaml because it is ambigious without the prefix.
• We support this in folder structure, instead of foo.yaml you have to put it in c/foo.yaml or python/foo.yaml

[ahmadia]

Why don't we just use / as our separator for both the filename and the "realm"? I like namespaces but I'm not married to them.

[ahmadia]

If we do that, then your original propose of /blas being top-level makes more sense.

[dagss]

Nice idea... python/numpy.version could be made to work but looks a bit horrible. Require you to say (python/numpy).version perhaps...

[ahmadia]

python/numpy.version seems fine
or python/[email protected]

[certik]

Except that we agreed in hashdist/hashdist#325 to use / for separating the package name and its hash.

[dagss]

@ahmadia even in Python hooks? I can make python/numpy.version work, literally, but it's an awful mess with numpy.version lazily evaluated in case it's a divisor..

Now that {{ expr }} uses Python expressions, and with half a mind thinking we'll see more Python and less YAML in the future (that's just my private thoughts though), I'm thinking we should come up with a more pythonic syntax..

[dagss]

I'm thinking like this in terms of roadmap here:

1. Get used to the new parameter system

2. Figure out if profiles also needs to change their import/parameter system (so that I can e.g. do a "named import" of another stack, and use it with some overrides, or pull out some of the configuration but not other).

3. Continue this discussion.

And that for now we go for python_yaml, but numpy, with a mind to make things consistent later.

And also I propose that package names should be [a-z0-9]+ from now on. Enabling the dash is not worth the hassle IMO.

[cekees]

So far it looks to me like we should kick namespaces down the road a bit
and fix package name conflicts as they arise. If python- doesn't work
as guidance then something else simple, short, and clear seems like a
good recipe.

On Wed, May 27, 2015 at 3:34 PM, Dag Sverre Seljebotn <
[email protected]> wrote:

@ahmadia https://github.com/ahmadia even in Python hooks? I can make
python/numpy.version work, literally, but it's an awful mess with the
type of numpy.version lazily evaluated in case it's a divisor..

Now that {{ expr }} uses Python expressions, and with half a mind
thinking we'll see more Python and less YAML in the future (that's just my
private thoughts though), I'm thinking we should come up with a more
pythonic syntax..

—
Reply to this email directly or view it on GitHub
#795 (comment).