Update BULDING.md for the yi-src-multi #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1. Added the name field from arguments to the serialize/deserialize functions since they are needed on the generated stages files. 2. Added the new argument type ArgumentRT to PipelineComponentBase. 3. Re-generated NormalizationComp and Segmentation stages code. The arguments names weren't matching the ones in the workflow descriptor. To do that the stages descriptors were updated with the correct names. 4. The RegionTemplate reference containing all data regions is now inserted on each stage that has at least one data region.
1. Corrected generated stages regarding RT/DR variables initialization, 2. Updated ArgumentBase to hold an io_type, representing if a variable is an input or output. This will be used later to perform memory cleaning on arguments. Serialization/deserialization methods updated to hold this state between messages. 3. Commented memory cleaning line PipelineComponentBase.cpp:42. This operation resulted on premature deallocaion of reused arguments and it would possibly result in a racing condition, on which if a reusable argument is read by a second task before a first task deallocated it.
…leanning will be performed later.
…ge argument from witch the output value will be collected.
1. Output IDs are now correctly assigned. Still need to get the output values from the diff stage. Notes: 1. System is currently working with any parameter set with only two mpi processes. Still need to debug why is that. 2. Although the workflow is returning the correct diff output, its value is only seen thanks to a debug cout, not being correctly retrieved from the Argument object.
Previously the memory cleanup of Arguments was commented as a workaround but now it has been permanently corrected. 1. The delete lines have been uncommented, 2. Each argument from a generated workflow was cloned, thereby not having the same reference for Arguments being used by multiple stages.
Also, the clonning of PCB now incorporates the ReusableTask list. System is compiling and working.
…till not integrated/used. System compiling and working.
was tested. System compiling and working
…dded the merge non-tested implementation. System compining and working.
SegmentationTask serialization/deserializarion had misordered variables and the double variables were being sent as float.
Not working when 'reusable' is called due to the DRs being NULL.
Using Segmentation as an example, this stage has at least one task, which is verifiable for reuse with another task, and if so, the stage can be merged with another stage. Still need to: - Break the Segmentation task in three, - Enforce reuse on parallel tasks, - Reimplement Stage code generator.
single workflow (no actual reuse yet).
Reuse happens and stages are merged, However, each stage has only one DR of each type, this way the DR value is overwritten by the last final task.
If there was a stage that were dependent of another stage that ended up being reused, the parent id of the first stage wouldn't be updated to the stage that the second stage was merged. This means that the first stage dependencies would be solved on the begining of the execution, running without the propper inputs. This behaviour has corrected but another bug was found. Sometimes the merging of tasks won't update the task dependencies properly. This behaviour was found when testing with the standard arguments and two values for arguments 'minSizePl' and 'minSizeSeg'.
…rging strategies.
Generated code compiles but isn't running propperly. Header code generation not yet implemented.
When testing with 2 parameters and 2 values for each (i.e 4 workflows) it works; When testing with 3 parameters and 2 values for each (i.e 8 workflows it doesn't work; This is independent of which parameters are used.
It was possible for a reusable task to break dependencies when being reused.
E.g: A__B__C__G
\_D__E__H
\_F__I
Given the above task dependency tree if a stage S' was to be merged to the
stage S (above), both of three tasks per workflow, and tasks A, D and C were
said to be reusable by S' tasks the correct merged stage should be:
A____B__C__G
\___S'2__S'3__S'4
\__D__E__H
\_F__I
And instead the algorithm was retunning:
A__B___C___G
\ \__S'4
\_D__E___H
\_F___I
Breaking the dependency of C to B with D.
The algorithm was fixed and tested successfully with 4 parameters with 2
values each, on -np 7.
As far as it was tested it doesn't present any bug. In order to compile it the boost lib is required. The output isn't still finished (i.e outputs on stdout only). Also, the cut weight ins't implemented yet, so the cut value on the output is a miscalculated value. The min-cut algorithm isn't integrated on the system yet.
This implementation is extreamely naive, thus, returning a rather low reuse (~9%).
A new argument needs to be sent to the application in order to select the eanted algorithm. The added/tested algorithms were: - no FGO merging - naive - smart recursive cut - reuse-tree The reuse-tree algorithm will have a separated optimized and non-optimized versions.
…zation on and off.
… into FGO-bucket-star
…n algorithm, against the stage-clustered generation model.
…es between stages aren't being enforced. Reproductible by profiling WORKER_PROFILER with clustered stage generation.
…ed stage generation. Explanation on PipelineManager.cpp:947
Updates to the new options required and dependencies. Change all "make" to "make -j$(nproc)" for a faster compile time while still keeping generic.
georgeteodoro
approved these changes
Apr 18, 2017
Leak was due to the non deletion of the DDR2Ds of DiffMaskComp.cpp:run() whenever one of the two DDR2D was NULL. This was due to some bad stage merging, which caused the input arguments to have the wrong (reused stage) dependency ID, thus making such arguments unavailable. The arguments' IDs were fixed.
… without deleting the old ones).
…nor bug of non-matching of tasks.
Fgo bucket star
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
@georgeteodoro Could you make sure it's ok the update on BUILDING.md? It was all we had to change to compile with the yi-src-multi.