Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CCL operations with replica_groups=1 #17804

Open
tapspatel opened this issue Feb 10, 2025 · 0 comments
Open

CCL operations with replica_groups=1 #17804

tapspatel opened this issue Feb 10, 2025 · 0 comments
Labels
feature-request External feature request

Comments

@tapspatel
Copy link
Contributor

tapspatel commented Feb 10, 2025

When running through compiler passes in tt-forge, if the CCL operation is done via the following configuration

1x8

all_gather on cluster_axis=0

Therefore, we are doing all_gather on each device with itself. This doesn't produce anything useful computationally but for generality, it would be nice to have an optimization that could return back the original tensors unchanged. This is very useful for being able to run a sweep of tests with various different configurations that build on-top of tt-metal. Currently, it fails with the following error

Error, All-gather was unable to identify either a sender or receiver device ID and atleast one must be identified for a valid all-gather configuration. The input mesh tensor or all-gather arguments may be incorrect

Sample test case:

import ttnn
import torch

mesh_device = ttnn.open_mesh_device(ttnn.MeshShape(1, 8))

torch_input_tensor = torch.rand((1, 1, 8192,800), dtype=torch.bfloat16)

mesh_tensor = ttnn.from_torch(
    torch_input_tensor,
    layout=ttnn.TILE_LAYOUT,
    device=mesh_device,
    mesh_mapper=ttnn.ShardTensor2dMesh(mesh_device, mesh_shape=list(mesh_device.shape), dims=(2, 3)),
)

output_tensor = ttnn.all_gather(
    mesh_tensor,
    dim=2,
    cluster_axis=0,
    mesh_device=mesh_device,
    topology=ttnn.Topology.Linear,
)

ttnn.close_mesh_device(mesh_device)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request External feature request
Projects
None yet
Development

No branches or pull requests

1 participant