prov/rxm: add FI_PEER support to rxm #10510
Open
+697
−1,239
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
(will move first 3 commits into separate PR, leaving here for testing) |
aingerson
force-pushed
the
rxm
branch
2 times, most recently
from
f21fe03
to
c860a1e
Compare
The peer srx should return entries with the FI_MSG/FI_TAGGED and FI_RECV flags set The msg_size field in the peer_rx_entry needs to be set in the expected path to the number of bytes allowed to be copied. This is either the size of the message (from the attr->msg_size paramter) or, if the buffer is not large enough to hold the entire message, the size of the buffer. This also fixes setting the message size and flag fields on the unexpected multi receive path. This case is a bit different because it not only has to account for the message size and buffer size, but also for the owner entry's message size and flags Signed-off-by: Alexia Ingerson <[email protected]>
The util srx/peer srx should return the entry with the op flags (FI_MSG/FI_TAGGED + FI_RECV) already set. Remove from the shm code Signed-off-by: Alexia Ingerson <[email protected]>
ubertest implementation currently requires FI_RMA_EVENT when using RMA and counters This will cause tcp to return ENODATA for these combinations and cause runfabtests to fail. This should get updated in ubertest to not require it but remove testing for now Signed-off-by: Alexia Ingerson <[email protected]>
The rxm SAR segment type enum was defined inside another struct. While techincally ok, this made it difficult for editors to find the type and reported compiler errors. This cleans it up to make it more readible and easier for editors to find the type Signed-off-by: Alexia Ingerson <[email protected]>
Add application side support for FI_AV_USER_ID which requires saving the fi_addr input as the internal fi_addr (for both the peer API srx use case and for reporting unique source address information). When supporting the capability for the application, remove it form the core provider information as it is only required on the top layer Signed-off-by: Alexia Ingerson <[email protected]>
Support using the peer APIs by default using the util peer helper functions. Instead of going through the rxm-specific functions to write to CQs and counters, use the ofi_peer_cq/cntr APIs which use the owner ops. In the default case where rxm is not being used as a peer these will go to the regular ofi_cq_write functions. Signed-off-by: Alexia Ingerson <[email protected]>
Remove rxm implementation of receive queues and leverage the util srx implementation which supports the peer srx API. This allows rxm to use the peer API calls to match receives. To do this, move the rxm protocol information from the receive entry into the rx_buf and allocate it dynamically as needed to track protocol information. This allows rxm to use the default peer_rx_entry instead of its own custom receive entry. With this last piece of the peer API implemented, rxm can also now advertise full support of the FI_PEER capability. Just like the FI_AV_USER_ID capability, rxm removes the bit from the core provider info as it is only a requirement from the application side and not from the message provider Signed-off-by: Alexia Ingerson <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add rxm support for FI_PEER (srx, cq, and cntr) and FI_AV_USER_ID to be used with the link provider