Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Speed up array declaration when possible in iomodule #1089

Merged
merged 6 commits into from
Aug 18, 2023
Merged

Speed up array declaration when possible in iomodule #1089

Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
11f8755
replaced factories.array
shahpratham Feb 5, 2023
ee2cba9
Merge branch 'main' into enhancement/797-io-dndarray
mrfh92 Jun 26, 2023
d019d76
Merge branch 'main' into enhancement/797-io-dndarray
ClaudiaComito Aug 17, 2023
1ffb3fb
Update heat/core/io.py
ClaudiaComito Aug 18, 2023
64f1b4f
Update heat/core/io.py
ClaudiaComito Aug 18, 2023
fe82322
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions heat/core/io.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -893,8 +893,20 @@ def load_csv(
# In case there are some empty lines in the csv file
local_tensor = local_tensor[:actual_length]

resulting_tensor = factories.array(
local_tensor, dtype=dtype, is_split=0, device=device, comm=comm
total_actual_lines = torch.empty(size, dtype=torch.int32)
comm.Allgather(torch.tensor([actual_length], dtype=torch.int32), total_actual_lines)
ClaudiaComito marked this conversation as resolved.
Show resolved Hide resolved

cumsum = total_actual_lines.cumsum(dim=0).tolist()
gshape = (cumsum[-1], columns[0].item())
ClaudiaComito marked this conversation as resolved.
Show resolved Hide resolved

resulting_tensor = DNDarray(
local_tensor,
gshape=gshape,
dtype=dtype,
split=0,
device=device,
comm=comm,
balanced=None,
)
resulting_tensor.balance_()

Expand Down
Loading