Added support for local lambdas with (value_type&) signatures for arr… #133
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ays with constexpr compiler guards.
|
This adds functionality requested in iss #118 |
Small fix to `static_assert` message in `array::local_for_all()`
bwpriest
added a commit
to bwpriest/ygm
that referenced
this pull request
Apr 11, 2023
LLNL#133) * Added support for local lambdas with (value_type&) signatures for arrays with constexpr compiler guards.
steiltre
added a commit
that referenced
this pull request
Apr 16, 2024
* Increment YGM version number in CMakeLists.txt * Sends comm:cerr() to std::cerr instead of std::cout (#78) * feature/msg_tweak (#77) * Increment YGM version number in CMakeLists.txt * changed messages to use instead of hard-coded value. * Cleaned up cmake build, and moved the library target description to one place. Co-authored-by: Trevor Steil <[email protected]> * Defaults CMAKE_BUILD_TYPE to Release (#76) * Initial ygm::io::multi_output functionality (#75) * Initial ygm::io::multi_output functionality * Removes unnecessary closing of ofstreams in destructor * Checks prefix path provided to ygm::io::multi_output does not exist as a regular file, and changes filename variable name to subpath * Adds tests for ygm::io::multi_output * Adds ygm::io::daily_output and simple tests * Adds test to check correct files are created by ygm::io::multi_output * Fixing clean-up of test files written * Forces filename_prefix given to ygm::io::multi_output constructor to be a directory * Feature/ci boost 1.78 (#80) Adds Boost 1.78 to CI. * Initial ygm::array implementation (#79) * Initial ygm::array implementation * Adds safety checks of array sizes * Adds barrier after resizing completes * Changes name of async_put to async_set * Adds array::async_unary_op_update_value and helpers for commonly used binary and unary operators * Adds functions for getting YGM pointers to arrays and array sizes * Adding container maptrix and SpMV. (#68) * Adding maptrix apis and impls. * Adding async_visit API+impl. * Some more details. * Maptrix API changes. * Adding new maptrix design. * Adding SpMV - first take. * Adding Structure * Porting to develop. * Adding SpMV as a standalone function. * Adding pagerank in examples and other restructuring changes. * Maptrix Impl. * Adding for_all over row_id and col_id. * Moving to experimental directory * Delete assoc_vector, replaced with ygm map. * Delete assoc_vector_impl, replaced with ygm map_impl. * Delete maptrix.hpp, in experimental mode. * Delete adj_impl.hpp, in experimental. * Delete csc_impl.hpp, in experimental. * Delete csr_impl.hpp, in experimental. * Delete maptrix_impl.hpp, in experimental. * Delete spmv.hpp, in experimental. * Moved as a part of alg_spmv.hpp. * Moved as a part of alg_spmv.hpp. * Add nicer examples or, tests. * Brain storming spmv row. * Moved within containers. * Moved within containers. * Adding new changes. * Adding timing details. * . * Changing API to insert_if_missing_else_visit. * Fix. * Fix. * Adding norm check. * Added OpPlus OpTimes. * Adding OpTimes. * Adding fixes from pull-request. * Adding fixes. * Delete spmv_row, not supporting for now. * Adding references. * Fixing references. * Changing line parsing. * Changing to 32bit. * Cleaning up webgraph example to use as general SpMV example * Removing extra headers * Switching to webgraph_spmv.cpp as new alg_spmv.cpp * Removing maptrix_visit * Cleaning up alg_pagerank.cpp * Removing unused 2D hasher * Removes unneeded header and changes a function name in adj_impl.hpp * Removing unused code in SpMV * Removing extra headers in column_view_impl.hpp * Removes unused headers in maptrix_impl and passes default value to row_view and column_view * Removes unused headers in maptrix_impl and row_view_impl * clang-format on maptrix.hpp Co-authored-by: Ancy Sarah Sarah Tom <[email protected]> Co-authored-by: Trevor Steil <[email protected]> * Feature/buffer multi output (#81) * Manually buffers ygm::io::multi_output writing * Adds buffering to ygm::io::daily_output * Updates ygm::io output tests to use buffering * Adds ability for csv_parser to read fields as unsigned integers (#83) * ygm::comm::Layout class (#82) * initial pass on implementing the Layout class. ygm::comm::layout() returns a const reference. * Removed comments and cleaned up ygm::Layout internals. * Reordered ygm::comm::Layout member for consistency. * Added tests for ygm::Layout::is_strided() and ygm::Layout::is_local(), and added functions to get const refs to all local and strided ranks. * Refactored ygm::Layout -> ygm::detail::layout. * Adds missing functions to counting set (#84) * Adds missing function to get a YGM pointer to a counting set * Adds comm() function to counting_set * Fixes typo in counting_set.comm() * Adds test for counting_set's YGM pointer * Adds topk to counting set * Adds barrier to beginning of map's topk * Adds example of counting_set topk * Consolidated ci jobs into a single job with matrixed dispatch on gcc version and mpi types. Also caching boost to avoid downloading/untaring with every job. (#86) * updated CI triggers such that PRs to main and develop and pushes to feauture/** and hotfix/** branches trigger jobs. (#88) * Adds possibility to send during processing of receive queue (#91) * Adds additional opportunities to flush buffers while processing received messages * Tweaked contributing guidelines to reflect CI changes and fixed some … (#89) * Tweaked contributing guidelines to reflect CI changes and fixed some typos. * Update CONTRIBUTING.md Co-authored-by: Trevor Steil <[email protected]> * Fixes bug in ygm::io::line_parser based on the split between files occurring on a newline. (#93) * Added get_ygm_ptr() to disjoint sets (#92) Co-authored-by: Sudharshan Srinivasan <[email protected]> * Switches ygm::container::array to block partitioning (#95) * Switches ygm::container::array to block partitioning * Gives last rank fewer elements during resize to avoid issues in for_all * Removes ability to resize ygm::container::array. No longer sensible with block partitioning * Properly sets local block size of last rank when block is full-sized * gather data to single std::vector added (#94) * gather data to single std::vector added * cleaned up collective operation, fixed test case, added to all support * Update bag_impl.hpp m_res -> p_res removes unnecessary `mailbox` from `gatherer` lambda passes `outer_data` to `gatherer` by reference uses m_comm to build ygm_ptr (preferred method) * Update bag_impl.hpp Makes vector arguments to lambdas in gather_to_vector const Co-authored-by: Dozier <[email protected]> Co-authored-by: Trevor Steil <[email protected]> * Hotfix/remove local receive (#96) * Adds ygm_tag to comm::impl * Removes local_receive function * Feature/interrupt mask (#97) * Adds initial ygm::detail::interrupt_mask implementation * Adds test for ygm::detail::mask_interrupt * Adds detail::interrupt_mask to map_impl to prevent iterator invalidation during map visits * Adds missing include statement for interrupt_mask in map_impl * Fixes namespacing for interrupt_mask use in map_impl * Gets m_comm from pmap in map_impl::async_visit_group * Adds missing mpi_typeof function for floats (#99) * Adds comm() function to set and multiset (#100) * Updates version number given in example CMake snippet (#101) * Added small fixes to clear gcc12 compiler warnings. (#102) * Feature/single threaded overhaul (#103) Significant overhaul removing listener thread and MPI_THREAD_MULTIPLE. * Removal of the listener thread. Removes requirement for MPI_THREAD_MULTIPLE. Improves performance substantially for the triangle counting benchmark. * NR & NLNR routing are supported. * Added the ability to capture primitive values in the async lambda to avoid Cereal overhead. This improves performance a bit, but mostly it makes user code easier to read/debug. Caution: there are no safety rails to prevent capturing a local pointer (but does prevent local references). * Added a full environment setting system, and it now controls the major settings in the runtime. * comm::welcome() added a welcome banner that prints out the current configuration and MPI settings. * API minor change removed the optional parameter to comm::comm(…, int buffer_capacity) that controlled the buffer size. Now it must be controlled by the environment. * Added new mechanism to indicate remote dispatch functions via static initialization. * Removes x86intrin.h inclusion in comm_impl (#104) * Inserted a std::stringstream into ygm::comm::cout()-style functions so that multi-rank print operations are less likely to get garbled. (#107) * Feature/ci update (#108) * Adds Github action to run GCC-8 tests on Ubuntu 20.04 as it is unavailable on Ubuntu 22.04 * Updating version of checkout action to get rid of javascript warning in Github Actions * Updating version of cache action to get rid of javascript warning in Github Actions * Develop arrow (#98) * arrow parquet file reader * arrow parquet file reader test * Update arrow_parquet_stream_reader.cpp Fixes typo in schema_to_string name * Update arrow_parquet_parser.hpp Fixes typo in schema_to_string name * arrow parquet file reader - updated how files in a directory are read Co-authored-by: Trevor Steil <[email protected]> * Small updates to Parquet parser (#109) * Updates Parquet reader tests and examples * Removes commented include * Updates arrow_parquet_parser to only check files from rank 0 during construction * Arrow/Parquet CI (#110) * arrow parquet file reader * arrow parquet file reader test * Adds installation of Apache Arrow to Github CI * Using Arrow version 9.0 * Downloading Arrow 9.0.0 for CI * Trying Arrow 10 for CI * Downloading newest Apache Arrow and installing 9.0.0 in CI * Trying to download Apache Arrow 9.0.0 source instead * Trying version range for Arrow * Changes CI runner to install Arrow 10.0 and explicitly check for Arrow 8, 9, and 10 in CMake to work around issues with Arrow version compatibility. * Investigating output test_arrow_parquet_stream_reader to find cause of failure in CI * Adds guards around finding different versions of Arrow Co-authored-by: tahsinreza <[email protected]> * Feature/nlnr bcast (#105) * Adds NLNR broadcast * Removes debug statements * Removes old bcast code * Reorganizes examples (#112) * Bumps version number to 0.5 (#111) * Release Prep * Adds dummy header to bcast messages when routing is used to process messages properlyin handle_next_receive (#120) * Loops through test_comm once for each routing scheme (#119) * Added support for (key, value) lambdas while still supporting (kv_pair) lambdas for ygm::container::map (#123) * Changed local lambda signature of ygm::container::map to expect separate (key, value) pairs. * Made ygm::container::map's expected lambda signature backwards compatible. * introduced constexpr compile-time guard to check that remote lambdas ygm::container::map adhere to legacy signature expectations. * ygm::container::map can now accept remote lambdas whose signatures expect either pairs or separate key, value arguments. pair visitors with optional map pointer arguments and no visitor arguments MUST specify that the second argument is a pair to compile correctly. * Update map_visit_optional_arguments_legacy.cpp * Update map_impl.hpp --------- Co-authored-by: Trevor Steil <[email protected]> * adding inline definition for release_assert_fail in order to support mulit-object targets; fix for Issue #126 (#127) * Feature/routing consistency (#125) * Initial comm_router implementation * Removes comments, adds user access to a comm's router, and spells out assumptions on routing in comm_router.hpp * Changes bcast to use same remote channels as new NLNR * wip/reducing adapter (#129) * Adds ygm::container::map::async_reduce operation * Fixes bug in test_map * Initial reducing_adapter for ygm::container::map without reduction tree * Adds ygm::container::array::key_type as an alias to Index for use in ygm::container::reducing_adapters * Adds container_traits for inspecting YGM container types and moves always_false from map_impl to ygm/detail/ygm_traits.hpp for use in other contexts * Adds ygm::container::reducing_adapter for use with ygm::container::array using functionality from ygm/container/detail/container_traits.hpp to handle both types * Adds is_counting_set and provides tests for container_traits * Stores reduction operation in ygm::container::reducing_adapter * modified ygm::container::bag::for_all() to support separated (first, second) lambdas (#128) * created ygm::container::detail::bag_impl::for_all_pairs() to be used with pair bags. * created make_similar() functions for map and array containers that return empty containers with the same comm and default value (and size for arrays) * Added missing cereal include for std::pair * moved template metaprogramming boilerplate into its own header. * Added special functionality to bag::for_all() so that it can accept split (first, second) signatures. * removed vestigial header * Added compiler guard to bag::for_all() and added more helpful compiler error messages. * removed the make_similar function in favor of something more disciplined in the future. * removed make_similar from multimap * remove legacy pair signature support for ygm::container::map (#132) * removed support for pair arguments in local and remote map lambdas. * Added more informative compiler error messages for map lambda signature checks * Removed outdate comment text [skip ci] * minor cleanup of compiler error message [skip ci] * Added support for local lambdas with (value_type&) signatures for arr… (#133) * Added support for local lambdas with (value_type&) signatures for arrays with constexpr compiler guards. * Opens ygm::container::array::owner() call to public API and adds ygm::container::array::is_mine() operation to match other containers (#136) * Reducing Adapter Reduction Tree (#139) * Adds basic caching layer to reducing_adapter without multi-hop caching * Adds reduction tree to reducing_adapter * Adds sanity check that reducing_adapter cache slot is empty after flushing * Fixes bug where reducing_adapter cache was declared non-empty when a value was placed in the underlying container * Adds missing check of pthis in reducing_adapter * Moves reducing_adapter class to ygm::container::detail * Added compile-time guards to all local and remote container lambdas (#138) * placed compiler guards on remote array lambdas * Added compiler guards to local set lambdas * changed local lambda signatures of disjoint_set to separated [](const value_type &, const value_type &) format to match map. Also added compiler guards to disjoint_set local and remote lambdas. * Feature reduce by key (#144) * Added reduce_by_key and started new traits features. * Adds communicator collectives. (#143) * Adds communicator collectives. * fixed MPI_Comm. * bcast & is_same * Added rank-aware RNG wrapper. It can be modified with different rank/… (#140) * Added rank-aware RNG wrapper. It can be modified with different rank/seed strategies. * updated random namespace to mimic STL * fixed namespace device -> engine * removed shared_random_device. Will reintroduce when we add a post barrier callback concept * moved most random machinery into ygm::detail and made std::mt19937 the default random engine * Hotfix/recursive double receive (#147) * Adds .git to Git repos fetched in CMakeLists.txt (#150) * Added new set methods. (#151) * Update comm_impl.hpp (#152) Removed old debug asserts. * static_assert no longer triggers when using a lambda with disjoint_set::async_union_and_execute that requires a pointer to the disjoint set (#158) * Feature/bag shuffle (#130) * Added local and global shuffles to bag container * Added test case for shuffles * Cleaning up a little code * Updated global bag shuffle to accept RNG as well * Updated bag shuffles to utilize ygm:default_random_engine * Removed some comments * Adding template for RandomEngine and functions where no rng argument is passed * Finished templating shuffle functions properly * Bugfix/disjoint set logic (#159) * Overcomplicated disjoint set with path splitting storing parents and their ranks * Simplifies disjoint_set * Adds missing const in async_union_and_execute * Avoid running up tree when other_item is my_parent * Avoid running up tree when other_item is my_parent * Finishes initial path-splitting union-by-rank disjoint set implementation * Adds missing barrier at beginning of all_compress() * Bugfix/disjoint set optional pointer (#160) Stops static_assert from triggering when disjoint_set::async_union_and_execute lambda requires a pointer to the disjoint set * Feature/bag balancing (#146) * Rough draft for rebalencing bag function * WIP: Second iteration of bag rebalance code * Added second iteration of rebalance method. * Appended rebalance tests to test_bag.cpp * Updated rebalancing algorithm to reduce space complexity. Added local_pop(int n) function to pop multiple values at once. * Updated test to reflect bag rebalancing sizes being congruent with ygm arrays * static_assert no longer triggers when using a lambda with disjoint_set::async_union_and_execute that requires a pointer to the disjoint set (#158) * Feature/bag shuffle (#130) * Added local and global shuffles to bag container * Added test case for shuffles * Cleaning up a little code * Updated global bag shuffle to accept RNG as well * Updated bag shuffles to utilize ygm:default_random_engine * Removed some comments * Adding template for RandomEngine and functions where no rng argument is passed * Finished templating shuffle functions properly * Bugfix/disjoint set logic (#159) * Overcomplicated disjoint set with path splitting storing parents and their ranks * Simplifies disjoint_set * Adds missing const in async_union_and_execute * Avoid running up tree when other_item is my_parent * Avoid running up tree when other_item is my_parent * Finishes initial path-splitting union-by-rank disjoint set implementation * Adds missing barrier at beginning of all_compress() * Bugfix/disjoint set optional pointer (#160) Stops static_assert from triggering when disjoint_set::async_union_and_execute lambda requires a pointer to the disjoint set * Rough draft for rebalencing bag function * Merged with updated develop branch to prep for pull request. * Added second iteration of rebalance method. * Appended rebalance tests to test_bag.cpp * Updated rebalancing algorithm to reduce space complexity. Added local_pop(int n) function to pop multiple values at once. * Updated test to reflect bag rebalancing sizes being congruent with ygm arrays * Several small fixes to make rebalancing run with better space efficiency * Fixed async_insert vector pass by value error. Made small fix for accessing map elements to save memory. * Fixed O(p) time complexity issue when sending values for rebalance * Adding support for Arrow 12.0. (#162) * arrow parquet file reader * arrow parquet file reader * arrow parquet file reader * arrow parquet file reader * arrow parquet file reader * arrow parquet file reader * arrow parquet file reader * arrow parquet file reader * arrow parquet file reader * arrow parquet file reader test * Update arrow_parquet_stream_reader.cpp Fixes typo in schema_to_string name * Update arrow_parquet_parser.hpp Fixes typo in schema_to_string name * arrow parquet file reader - updated how files in a directory are read * merge develop into develop-arrow * adding support for Arrow 12.0 * merge develop into develop-arrow * Added missing for_all test for counting set (#165) Co-authored-by: Stephen Thaddaeus Youd <[email protected]> * added tagged bag container (#163) Added tagged_bag container * Feature/container traits (#161) Added ygm_container_type tags to containers. Additionally added compile time functions to check container type. * Move Constructor for Set (#166) * Adds test of std::vector of YGM sets * Fixes size check in test of vector of YGM sets * Adds move constructor to YGM set * Removes comm_impl and bag_impl (#167) * Removes comm::impl * Properly passes lambdas to pack_lambda operations and removes mention of comm::impl from interrupt_mask * Removes comm copy constructor * Changes comm in arrow_parquet_parser to a reference * Removes comm_impl.hpp includes from comments * Removes bag_impl * Added missing limits include (#168) * Feature/comm progress (#171) * Added comm::progress() and comm::wait_until(Fn). * Added comm::process_incoming() * Adding consume_all to set. (#169) * Adding consume_all to set. * Made adapter more clear. * Hotfix/multi output filesystem (#176) * Adds missing filesystem include to multi_output.hpp * Replaces csv_parser.hpp include in ndjson_parser.hpp with line_parser.hpp * Removes unnecessary ygm::io::detail namespace * Renamed 3 local comm member functions. (#173) * fix type of the get_ygm_ptr call in set.hpp (#178) * fix type of set get_ygm_ptr * add a test for the ygm set pointer * also fix the set pointer for multiset --------- Co-authored-by: Grace Jingying Li <[email protected]> * Hotfix/arrow_ci (#180) * Adding recent version of Arrow to CMakeLists.txt * make -j * Adds CMake flag to require a version of Arrow to be found, specifically for use in CI to make it obvious when Apache Arrow is not being found * Adds disjoint_set::clear() function (#183) * Feature/context aware progress (#182) * Check calling context in comm.local_progress() * Fix spelling error and increase test trip count * Add Parquet -> JSON Converter (#181) * Change parquet parser wrt schema Specifically, schema() returns physical type rather than logical type. * Add paquet -> json converter * Update Parquet JSON reader example * Arrow 14.0 (#185) Change the cmake file to support Apache Arrow v14.0 * Fixes array partitioning and bag rebalancing to be more balanced. (#187) * Copies all private variables in ygm::container::detail::array_impl copy constructor (#189) * Removes impl from ygm::container::array (#190) * Parallel Parquet File Reading (v2) (#188) * Parallel parquet reader v2 * Add comments * Add tests for parquet reader (#191) * Add tests for parquet reader - Add test for parquet2json convertor - Add additional parallel parquet reading test * Fix wrong indent * Brush up on parque reader test * Updates Regarding Arrow Parquet (#193) * Read values using std::optional in the ParquetToJson converter * Support Arrow v15 * Bugfix/disjoint set compress (#195) * Adds print statements to see if obvious bugs exist in disjoint_set::all_compress() * Removes race condition in disjoint_set::all_compress() by preparing all parent queries before sending any * Removing print statements from disjoint_set * Adds check of parent ranks before compressing disjoint sets * Adds check of parent ranks before compressing disjoint sets * Adds check of parent ranks before compressing disjoint sets * Avoid querying parent during disjoint_set::all_compress() when item is a root * Fixes missing reference for local_rep_query inside of update_rep_functor of disjoint_set::all_compress() * More detailed debugging output in all_compress() * Fixing logic for when a parent request comes for an item that had to query its own parent in current all_compress() round * Fixing logic for when a parent request comes for an item that had to query its own parent in current all_compress() round * Cleaning up all_compress() queries as they return * Cleaning up all_compress() queries as they return * Adding additional debug printing to disjoint_set::all_compress() * Avoids potential race condition where all_compress() is using whether parent of item being queried has returned in place of directly tracking whether item has found its root * Cleaning up disjoint_set::all_compress() code * Removing unused code * Add parquet2variant converter (#194) Co-authored-by: Keita Iwabuchi <[email protected]> * Bugfix/disjoint set compress (#196) * Adds missing async_visit function to disjoint_set * Fixes stopping criteria on while loop in disjoint_set::all_compress() * Fixes tracking of ongoing compression updates (#197) * Updating version number to 0.6 (#198) * Removes performance directory (#199) * Fixing Github actions to run on pull requests to master branch --------- Co-authored-by: Benjamin Priest <[email protected]> Co-authored-by: Roger Pearce <[email protected]> Co-authored-by: ancysarahtom <[email protected]> Co-authored-by: Ancy Sarah Sarah Tom <[email protected]> Co-authored-by: Sudharshan <[email protected]> Co-authored-by: Sudharshan Srinivasan <[email protected]> Co-authored-by: Ryan Dozier <[email protected]> Co-authored-by: Dozier <[email protected]> Co-authored-by: tahsinreza <[email protected]> Co-authored-by: tahsinreza <[email protected]> Co-authored-by: jleidel <[email protected]> Co-authored-by: Roger Pearce <[email protected]> Co-authored-by: Lance <[email protected]> Co-authored-by: youd3 <[email protected]> Co-authored-by: Stephen Thaddaeus Youd <[email protected]> Co-authored-by: Seth Bromberger <[email protected]> Co-authored-by: Grace <[email protected]> Co-authored-by: Grace Jingying Li <[email protected]> Co-authored-by: Preston Piercey <[email protected]> Co-authored-by: Keita Iwabuchi <[email protected]> Co-authored-by: Keita Iwabuchi <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…ays with constexpr compiler guards.