Adding tt-metalium #28854

blozano-tt · 2025-01-17T07:10:19Z

Checklist

github-actions · 2025-01-17T07:11:36Z

Hi! This is the staged-recipes linter and I found some lint.

File-specific lints and/or hints:

recipes/tt-metalium/meta.yaml:
- lints:
  - The following maintainers have not yet confirmed that they are willing to be listed here: afuller-TT. Please ask them to comment on this PR if they are.

conda-forge-admin · 2025-01-17T07:11:51Z

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipes/tt-metalium/meta.yaml) and found some lint.

Here's what I've got...

For recipes/tt-metalium/meta.yaml:

❌ The source section contained an unexpected subsection name. git_submodules is not a valid subsection name.

_{This message was generated by GitHub Actions workflow run https://github.com/conda-forge/conda-forge-webservices/actions/runs/12927119653. Examine the logs at this URL for more detail.}

conda-forge-admin · 2025-01-23T10:57:36Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/tt-metalium/meta.yaml) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipes/tt-metalium/meta.yaml:

ℹ️ Please depend on pytorch directly, in order to avoid forcing CUDA users to downgrade to the CPU version for no reason.

_{This message was generated by GitHub Actions workflow run https://github.com/conda-forge/conda-forge-webservices/actions/runs/12944803893. Examine the logs at this URL for more detail.}

blozano-tt · 2025-01-23T11:26:30Z

@h-vetinari would you be willing to review this? (Pretty please)

The C++ code builds.

I still need to iron out the python wheel generation and installation, but I believe all pieces are in place (just might be missing some python module dependencies).

afuller-TT · 2025-01-23T17:07:15Z

recipes/tt-metalium/meta.yaml

+extra:
+  recipe-maintainers:
+    - blozano-tt
+    - afuller-TT


I confirm I am willing to be listed here. ✅

…cipes into tt-metalium

jakirkham · 2025-01-24T05:45:38Z

recipes/tt-metalium/meta.yaml

+    - python
+    - pip
+    - wheel
+    - setuptools =69.5.1


Could this pin be relaxed?

Suggested change

- setuptools =69.5.1

- setuptools

Initially it was not constrained, and I noticed absurd numbers of warnings.

Probably some poor practices in our setup.py.

I pinned it to a tested version see if the warnings go away.

example of warning:

home/conda/staged-recipes/build_artifacts/tt-metalium_1737673554966/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.10/site-packages/setuptools/command/build_py.py:212: _Warning: Package 'tt_metal.common' is absent from the `packages` configuration. 2025-01-24T01:28:27.9339293Z !! 2025-01-24T01:28:27.9339614Z 2025-01-24T01:28:27.9516044Z ******************************************************************************** 2025-01-24T01:28:27.9520477Z ############################ 2025-01-24T01:28:27.9525248Z # Package would be ignored # 2025-01-24T01:28:27.9528979Z ############################ 2025-01-24T01:28:27.9532709Z Python recognizes 'tt_metal.common' as an importable package[^1], 2025-01-24T01:28:27.9537754Z but it is absent from setuptools' `packages` configuration. 2025-01-24T01:28:27.9542179Z 2025-01-24T01:28:27.9547546Z This leads to an ambiguous overall configuration. If you want to distribute this 2025-01-24T01:28:27.9552078Z package, please make sure that 'tt_metal.common' is explicitly added 2025-01-24T01:28:27.9557236Z to the `packages` configuration field. 2025-01-24T01:28:27.9557656Z 2025-01-24T01:28:27.9560939Z Alternatively, you can also rely on setuptools' discovery methods 2025-01-24T01:28:27.9564825Z (for example by using `find_namespace_packages(...)`/`find_namespace:` 2025-01-24T01:28:27.9569172Z instead of `find_packages(...)`/`find:`). 2025-01-24T01:28:27.9573084Z 2025-01-24T01:28:27.9577978Z You can read more about "package discovery" on setuptools documentation page: 2025-01-24T01:28:27.9582034Z 2025-01-24T01:28:27.9586458Z - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html 2025-01-24T01:28:27.9590206Z 2025-01-24T01:28:27.9595451Z If you don't want 'tt_metal.common' to be distributed and are 2025-01-24T01:28:27.9600171Z already explicitly excluding 'tt_metal.common' via 2025-01-24T01:28:27.9603896Z `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`, 2025-01-24T01:28:27.9607331Z you can try to use `exclude_package_data`, or `include-package-data=False` in 2025-01-24T01:28:27.9615269Z combination with a more fine grained `package-data` configuration. 2025-01-24T01:28:27.9615703Z 2025-01-24T01:28:27.9616050Z You can read more about "package data files" on setuptools documentation page: 2025-01-24T01:28:27.9616552Z 2025-01-24T01:28:27.9624712Z - https://setuptools.pypa.io/en/latest/userguide/datafiles.html 2025-01-24T01:28:27.9625050Z 2025-01-24T01:28:27.9625310Z 2025-01-24T01:28:27.9625638Z [^1]: For Python, any directory (with suitable naming) can be imported, 2025-01-24T01:28:27.9636130Z even if it does not contain any `.py` files. 2025-01-24T01:28:27.9637605Z On the other hand, currently there is no concept of package data 2025-01-24T01:28:27.9637978Z directory, all directories are treated like packages. 2025-01-24T01:28:27.9638373Z ******************************************************************************** 2025-01-24T01:28:27.9643265Z 2025-01-24T01:28:27.9643549Z !! 2025-01-24T01:28:27.9643835Z check.warn(importable) 2025-01-24T01:28:27.9644598Z /home/conda/staged-recipes/build_artifacts/tt-metalium_1737673554966/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.10/site-packages/setuptools/command/build_py.py:212: _Warning: Package 'tt_metal.hostdevcommon.api.hostdevcommon' is absent from the `packages` configuration. 2025-01-24T01:28:27.9648696Z !! 2025-01-24T01:28:27.9648997Z 2025-01-24T01:28:27.9649344Z ******************************************************************************** 2025-01-24T01:28:27.9649696Z ############################ 2025-01-24T01:28:27.9649924Z # Package would be ignored # 2025-01-24T01:28:27.9652893Z ############################ 2025-01-24T01:28:27.9653279Z Python recognizes 'tt_metal.hostdevcommon.api.hostdevcommon' as an importable package[^1], 2025-01-24T01:28:27.9653694Z but it is absent from setuptools' `packages` configuration. 2025-01-24T01:28:27.9654008Z 2025-01-24T01:28:27.9658543Z This leads to an ambiguous overall configuration. If you want to distribute this 2025-01-24T01:28:27.9658987Z package, please make sure that 'tt_metal.hostdevcommon.api.hostdevcommon' is explicitly added 2025-01-24T01:28:27.9659378Z to the `packages` configuration field. 2025-01-24T01:28:27.9659930Z 2025-01-24T01:28:27.9663458Z Alternatively, you can also rely on set

the warning is no a good enough reason to take such an old version; though of course it would be good to react to those warnings in the upstream project

Ok, I'll aim to resolve the warnings a different way.

jakirkham · 2025-01-24T05:47:49Z

recipes/tt-metalium/conda_build_config.yaml

+c_stdlib_version:          # [linux]
+  - 2.34                   # [linux]


Do we need this new of a version or could we use the default?

Unfortunately the code base currently has a dependency on elf.h from more modern glibc.

The only way to avoid this is to change upstream code, or patch it. However I may discover more points of dependence.

this code base has only been verified on Ubuntu 20.04 and newer.

more discussion here: conda-forge/linux-sysroot-feedstock#85

Would be good to leave a comment that you need this because you're compiling something that's targetting RISCV, which has only been validated on Ubuntu 20.04 (and up?), and so that's the safe choice (20.04 has glibc 2.31, so this is our only option that's >= that). For completeness, you could see if 2.28 happens to work, but no hard requirement.

jakirkham · 2025-01-24T05:48:24Z

recipes/tt-metalium/meta.yaml

+
+build:
+  number: 0
+  skip: true  # [not linux or py!=310]


Am curious why it needs to be Python 3.10 only. Is there more context on this?

3.8 and 3.10 are the only versions known to work. I was seeing some Python packaging issues and decided to try constraining this. I’ll see if I can open it back up.

There's very little variation between python versions that would make this ("only version to work") plausible. I'm also for removing the constraint (even if something fails, I expect this to be quite manageable).

Suggested change

skip: true # [not linux or py!=310]

skip: true # [not linux]

jakirkham · 2025-01-24T05:48:58Z

recipes/tt-metalium/meta.yaml

+  version: {{ version }}
+
+source:
+  - url: https://github.com/tenstorrent/tt-metal/releases/download/v{{ version }}-rc13/tt-metalium.tar.gz


Is there a reason this is using an RC (rc13)?

I am putting a lot of fixes into main to support this effort. I needed a very recent version of the code. We upload release candidates daily. We cut releases very rarely.

recipes/tt-metalium/meta.yaml

jakirkham · 2025-01-24T05:51:58Z

@conda-forge-admin , please relint

recipes/tt-metalium/meta.yaml

Co-authored-by: jakirkham <[email protected]>

conda-forge-admin · 2025-01-24T07:16:29Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/tt-metalium/meta.yaml) and found it was in an excellent condition.

h-vetinari · 2025-01-24T07:21:47Z

As a meta-comment, I note the -tt in both your handles, and was looking around the tenstorrent GH org a bit. I'm not 100% sure what setup you're aiming for, but if you have some time/funding for this, you might be interested to pursue conda-forge/conda-forge.github.io#1744. Bootstrapping a new architecture is a substantial amount of work, but OTOH you save yourself the hassles of emulation or other cross-platform shenanigans.

blozano-tt · 2025-01-24T07:54:38Z

As a meta-comment, I note the -tt in both your handles, and was looking around the tenstorrent GH org a bit. I'm not 100% sure what setup you're aiming for, but if you have some time/funding for this, you might be interested to pursue conda-forge/conda-forge.github.io#1744. Bootstrapping a new architecture is a substantial amount of work, but OTOH you save yourself the hassles of emulation or other cross-platform shenanigans.

I'll keep it in mind!

The common scenario we currently have is x86_64 host CPU that does all the orchestrating, and one or more accelerator chips hanging off the PCIe interface. (Reductionist) The accelerators are arrays of RISCV cores that run data movement/compute kernels. So we don't want Anaconda on the device.

However, I have heard of a few efforts where the host is desired to be RISCV, and in that case this could be of interest.

Add first shot

aad8e55

blozano-tt added 16 commits January 17, 2025 07:49

Adjust

88b02fd

Progress

4cbe34f

dunno

43ca25a

dunno

6fb0f13

Fix

d0dada2

Add libmpc as a dependency for sfpi

b935312

Try LD_LIBRARY_PATH

9bcf248

Try higher sysroot

f7a7349

Not sure

605a5f8

Add missing dep zlib

5d06a8c

Update to build and install python wheel

8ccf7fc

Add python build deps

dc68535

Please

d1b88a2

Reduce load on CPU

f522568

Try restricting python

e858c4b

Progress maybe

2d3f737

Fix skip issue

bbc7d92

afuller-TT reviewed Jan 23, 2025

View reviewed changes

recipes/tt-metalium/meta.yaml

extra:

recipe-maintainers:

- blozano-tt

- afuller-TT

Copy link

afuller-TT Jan 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I confirm I am willing to be listed here. ✅

blozano-tt added 4 commits January 23, 2025 22:55

Fix python module installation hopefully

04d3263

Merge branch 'main' into tt-metalium

ff638a1

Hacks to fix python packaging maybe

8678be7

Merge branch 'tt-metalium' of https://github.com/blozano-tt/staged-re…

293f3f7

…cipes into tt-metalium

jakirkham reviewed Jan 24, 2025

View reviewed changes

recipes/tt-metalium/meta.yaml Outdated Show resolved Hide resolved

Merge branch 'main' into tt-metalium

65387d1

jakirkham reviewed Jan 24, 2025

View reviewed changes

recipes/tt-metalium/meta.yaml Outdated Show resolved Hide resolved

jakirkham reviewed Jan 24, 2025

View reviewed changes

recipes/tt-metalium/meta.yaml Outdated Show resolved Hide resolved

blozano-tt and others added 3 commits January 23, 2025 22:53

Update recipes/tt-metalium/meta.yaml

9842d1d

Co-authored-by: jakirkham <[email protected]>

Update recipes/tt-metalium/meta.yaml

163f19b

Co-authored-by: jakirkham <[email protected]>

Update meta.yaml

2a3032d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding tt-metalium #28854

Adding tt-metalium #28854

blozano-tt commented Jan 17, 2025 •

edited

Loading

github-actions bot commented Jan 17, 2025 •

edited

Loading

conda-forge-admin commented Jan 17, 2025 •

edited

Loading

conda-forge-admin commented Jan 23, 2025 •

edited

Loading

blozano-tt commented Jan 23, 2025 •

edited

Loading

afuller-TT Jan 23, 2025

jakirkham Jan 24, 2025

blozano-tt Jan 24, 2025 •

edited by h-vetinari

Loading

h-vetinari Jan 24, 2025

blozano-tt Jan 24, 2025

jakirkham Jan 24, 2025

blozano-tt Jan 24, 2025

h-vetinari Jan 24, 2025

jakirkham Jan 24, 2025

blozano-tt Jan 24, 2025

h-vetinari Jan 24, 2025 •

edited

Loading

jakirkham Jan 24, 2025

blozano-tt Jan 24, 2025

jakirkham commented Jan 24, 2025

conda-forge-admin commented Jan 24, 2025

h-vetinari commented Jan 24, 2025

blozano-tt commented Jan 24, 2025

Adding tt-metalium #28854

Are you sure you want to change the base?

Adding tt-metalium #28854

Conversation

blozano-tt commented Jan 17, 2025 • edited Loading

github-actions bot commented Jan 17, 2025 • edited Loading

conda-forge-admin commented Jan 17, 2025 • edited Loading

conda-forge-admin commented Jan 23, 2025 • edited Loading

blozano-tt commented Jan 23, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blozano-tt Jan 24, 2025 • edited by h-vetinari Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

h-vetinari Jan 24, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jakirkham commented Jan 24, 2025

conda-forge-admin commented Jan 24, 2025

h-vetinari commented Jan 24, 2025

blozano-tt commented Jan 24, 2025

blozano-tt commented Jan 17, 2025 •

edited

Loading

github-actions bot commented Jan 17, 2025 •

edited

Loading

conda-forge-admin commented Jan 17, 2025 •

edited

Loading

conda-forge-admin commented Jan 23, 2025 •

edited

Loading

blozano-tt commented Jan 23, 2025 •

edited

Loading

blozano-tt Jan 24, 2025 •

edited by h-vetinari

Loading

h-vetinari Jan 24, 2025 •

edited

Loading