Upgrade process_results to include plots

[brunofavs]

I find comparing and analyzing the results after a batch very time-consuming and prone to error.

It would be great if the processing also outputted a plot of the results.

There is a problem with this though. This requires ATOM to know which experiments are related and which are not.

For example in this tree : ( from my odometry experiments)

├── nig_0.1
├── nig_0.15
├── nig_0.15-ntfv_0.1
├── nig_0.15-ntfv_0.15
├── nig_0.15-ntfv_0.2
├── nig_0.15-ntfv_0.25
├── nig_0.15-ntfv_0.3
├── nig_0.1-ntfv_0.1
├── nig_0.1-ntfv_0.15
├── nig_0.1-ntfv_0.2
├── nig_0.1-ntfv_0.25
├── nig_0.1-ntfv_0.3
├── nig_0.2
├── nig_0.25
├── nig_0.25-ntfv_0.1
├── nig_0.25-ntfv_0.15
├── nig_0.25-ntfv_0.2
├── nig_0.25-ntfv_0.25
├── nig_0.25-ntfv_0.3
├── nig_0.2_no_odom_calib
├── nig_0.2-ntfv_0.1
├── nig_0.2-ntfv_0.15
├── nig_0.2-ntfv_0.2
├── nig_0.2-ntfv_0.25
├── nig_0.2-ntfv_0.2_no_odom_calib
├── nig_0.2-ntfv_0.3
├── nig_0.3
├── nig_0.3-ntfv_0.1
├── nig_0.3-ntfv_0.15
├── nig_0.3-ntfv_0.2
├── nig_0.3-ntfv_0.25
├── nig_0.3-ntfv_0.3
└── perfect_sim

How would you know which experiments were part of the same plot?

My first guess was to compare which experiments had the same alphabetic characters but this is very easily prone to a lot of error.

These 2 for example should be on the same plot:

├── nig_0.3-ntfv_0.2
├── nig_0.3-ntfv_0.25

But this shouldn't even though they have same alphabetic chars.

├── nig_0.2-ntfv_0.2

There is no way for the code to know which experiments are related.

There is a super easy fix but would break backwards compatibility : Add a field in data.yml called groups, like so :

experiments:
  
  group_1:  
      #Varying ntfv with fixed nig = 0.1
    - {name: nig_0.1-ntfv_0.1 ,  nig_value: 0.1,  ntfv_value: 0.1,   calibrate_odom : true}
    - {name: nig_0.1-ntfv_0.15,  nig_value: 0.1,  ntfv_value: 0.15,  calibrate_odom : true}
    - {name: nig_0.1-ntfv_0.2 ,  nig_value: 0.1,  ntfv_value: 0.2,   calibrate_odom : true}
    - {name: nig_0.1-ntfv_0.25,  nig_value: 0.1,  ntfv_value: 0.25,  calibrate_odom : true}
    - {name: nig_0.1-ntfv_0.3 ,  nig_value: 0.1,  ntfv_value: 0.3,   calibrate_odom : true}

  group_2:
      #Varying ntfv with fixed nig = 0.2
    - {name: nig_0.1-ntfv_0.1 ,  nig_value: 0.2,  ntfv_value: 0.1,   calibrate_odom : true}
    - {name: nig_0.1-ntfv_0.15,  nig_value: 0.2,  ntfv_value: 0.15,  calibrate_odom : true}
    - {name: nig_0.1-ntfv_0.2 ,  nig_value: 0.2,  ntfv_value: 0.2,   calibrate_odom : true}
    - {name: nig_0.1-ntfv_0.25,  nig_value: 0.2,  ntfv_value: 0.25,  calibrate_odom : true}
    - {name: nig_0.1-ntfv_0.3 ,  nig_value: 0.2,  ntfv_value: 0.3,   calibrate_odom : true}

I'll work on this even if its just for myself and if it proves useful we consider making it on to noetic-devel

[miguelriemoliveira]

There was the idea of trying to use weights and biases for this (and for the batch execution). Do you think it would be applicable? * Miguel Riem de Oliveira Professor Auxiliar * email: ***@***.*** | ***@***.*** zoom: https://videoconf-colibri.zoom.us/j/7155402102 office: 22.2.18 | ext: 23887 Dep. de Engenharia Mecânica, Universidade de Aveiro *Campus Universitário de Santiago, 3810-193 Portugal*

…

On Thu, May 9, 2024 at 5:21 PM Bruno Silva ***@***.***> wrote: I find comparing and analyzing the results after a batch very time-consuming and prone to error. It would be great if the processing also outputted a plot of the results. There is a problem with this though. This requires ATOM to know which experiments are related and which are not. For example in this tree : ( from my odometry experiments) ├── nig_0.1 ├── nig_0.15 ├── nig_0.15-ntfv_0.1 ├── nig_0.15-ntfv_0.15 ├── nig_0.15-ntfv_0.2 ├── nig_0.15-ntfv_0.25 ├── nig_0.15-ntfv_0.3 ├── nig_0.1-ntfv_0.1 ├── nig_0.1-ntfv_0.15 ├── nig_0.1-ntfv_0.2 ├── nig_0.1-ntfv_0.25 ├── nig_0.1-ntfv_0.3 ├── nig_0.2 ├── nig_0.25 ├── nig_0.25-ntfv_0.1 ├── nig_0.25-ntfv_0.15 ├── nig_0.25-ntfv_0.2 ├── nig_0.25-ntfv_0.25 ├── nig_0.25-ntfv_0.3 ├── nig_0.2_no_odom_calib ├── nig_0.2-ntfv_0.1 ├── nig_0.2-ntfv_0.15 ├── nig_0.2-ntfv_0.2 ├── nig_0.2-ntfv_0.25 ├── nig_0.2-ntfv_0.2_no_odom_calib ├── nig_0.2-ntfv_0.3 ├── nig_0.3 ├── nig_0.3-ntfv_0.1 ├── nig_0.3-ntfv_0.15 ├── nig_0.3-ntfv_0.2 ├── nig_0.3-ntfv_0.25 ├── nig_0.3-ntfv_0.3 └── perfect_sim How would you know which experiments were part of the same plot? My first guess was to compare which experiments had the same alphabetic characters but this is very easily prone to a lot of error. These 2 for example should be on the same plot: ├── nig_0.3-ntfv_0.2 ├── nig_0.3-ntfv_0.25 But this shouldn't even though they have same alphabetic chars. ├── nig_0.2-ntfv_0.2 There is no way for the code to know which experiments are related. There is a super easy fix but would break backwards compatibility : Add a field in data.yml called groups, like so : experiments: group_1: #Varying ntfv with fixed nig = 0.1 - {name: nig_0.1-ntfv_0.1 , nig_value: 0.1, ntfv_value: 0.1, calibrate_odom : true} - {name: nig_0.1-ntfv_0.15, nig_value: 0.1, ntfv_value: 0.15, calibrate_odom : true} - {name: nig_0.1-ntfv_0.2 , nig_value: 0.1, ntfv_value: 0.2, calibrate_odom : true} - {name: nig_0.1-ntfv_0.25, nig_value: 0.1, ntfv_value: 0.25, calibrate_odom : true} - {name: nig_0.1-ntfv_0.3 , nig_value: 0.1, ntfv_value: 0.3, calibrate_odom : true} group_2: #Varying ntfv with fixed nig = 0.2 - {name: nig_0.1-ntfv_0.1 , nig_value: 0.2, ntfv_value: 0.1, calibrate_odom : true} - {name: nig_0.1-ntfv_0.15, nig_value: 0.2, ntfv_value: 0.15, calibrate_odom : true} - {name: nig_0.1-ntfv_0.2 , nig_value: 0.2, ntfv_value: 0.2, calibrate_odom : true} - {name: nig_0.1-ntfv_0.25, nig_value: 0.2, ntfv_value: 0.25, calibrate_odom : true} - {name: nig_0.1-ntfv_0.3 , nig_value: 0.2, ntfv_value: 0.3, calibrate_odom : true} I'll work on this even if its just for myself and if it proves useful we consider making it on to noetic-devel — Reply to this email directly, view it on GitHub <#951>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACWTHVWIEV36GVXN5CNUANDZBOPAXAVCNFSM6AAAAABHPDHQYCVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI4DQMBTGI4DKMY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[brunofavs]

What do you mean weights and biases? I didn't get it.

Attribute a "weight" (more like a id the way im thinking) to each related experience and relate them after in processing that way? I don't think thats it is it?

[brunofavs]

I'm going back on my previous take.

Other than breaking backwards compatibility, the way the plots would be generated would be fixed.

After discussing with @manuelgitgomes, I settled on saving a yaml with the experiment settings on the experiment dir and then use it later in processing to plot whichever plots the user wants based on n conditions.

[brunofavs]

Now each batch has this both in results and results processed :

Meaning now a new script will read off of a yaml a set of conditions and plot something based on that.
And should be fully dynamic

[miguelriemoliveira]

What do you mean weights and biases? I didn't get it.

I meant this:

https://wandb.ai/site

as @lucasrdalcol to show you what it can do.

[brunofavs]

as @lucasrdalcol to show you what it can do.

I will mention it whenever I seem him next week :)

Plot graphs script

This is the new fully fledged plot maker in atom.

Functionalities :

• Plot based on set of conditions based on the settings.yml mentioned on my prior comment here.

• Plot whichever variable present on the collected files

• Overlap whichever plots wanted

• Save as many figures as imaginable.

• Individual line settings for each plot and global settings for each figure

  • Colors
  • Linewidths
  • Linestyles
  • Markers
  • Markersizes
  • etc

• Possibility of setting one set of settings for all plots in a given figure

• Saves high res images.

The script is already fully documented on the yaml with all possible functionalities. It is also full with informativeprints and custom debug prints as well.

Example yaml :

#
#           █████╗ ████████╗ ██████╗ ███╗   ███╗
#          ██╔══██╗╚══██╔══╝██╔═══██╗████╗ ████║
#          ███████║   ██║   ██║   ██║██╔████╔██║
#          ██╔══██║   ██║   ██║   ██║██║╚██╔╝██║
#   __     ██║  ██║   ██║   ╚██████╔╝██║ ╚═╝ ██║    _
#  / _|    ╚═╝  ╚═╝   ╚═╝    ╚═════╝ ╚═╝     ╚═╝   | |
#  | |_ _ __ __ _ _ __ ___   _____      _____  _ __| | __
#  |  _| '__/ _` | '_ ` _ \ / _ \ \ /\ / / _ \| '__| |/ /
#  | | | | | (_| | | | | | |  __/\ v  v / (_) | |  |   <
#  |_| |_|  \__,_|_| |_| |_|\___| \_/\_/ \___/|_|  |_|\_\
#  https://github.com/lardemua/atom
#
#  This yaml file describes which plots the script plot_graphs will compute.
#
# Example plot figure
# To create more plots, create another dictionary with the same structure below, being careful to change the name
# as its used to save each plot and would cause undesired behaviors due to overwrite protection.
# plot_1 :
#   options :
#     title : "title"
#     xlabel : "xlabel"
#     ylabel : "ylabel"
#     linewidths : 2
#
#     # Possible linetypes
#     # dotted dashed dashdot loosely dotted dotted
#     # densely dotted long dash with offset loosely dashed
#     # dashed densely dashed loosely dashdotted dashdotted
#     # densely dashdotted dashdotdotted loosely dashdotdotted
#     # densely dashdotdotted None solid
#
#     # Define it either as a list for each plot line or as a str for all plot lines
#     linestyles : ["dotted","solid"] # 
#
#     #    Possible markers
#     #".",",","o","v","^","<",">","1","2","3","4","8","s","p","P","*","h",
#     #"H","+","x","X","D","d","|","_",0,1,2,3,4,5,6,7,8,9,10,11
#
#     # Define it either as a list for each plot marker or as a str for all plot markers
#     markers : "."
#     # Define it either as a list for each plot marker size or as a int for all plot marker sizes
#     markersizes : 10
#     #   Possible colors
#     # https://matplotlib.org/stable/gallery/color/named_colors.html
#     #
#     # Define it either as a list for each plot line color or as a str for all plot line colors
#     colors : "blue"
#     grid : true
#  
#   data : 
#     # --- Define all properties that define the plot, according to the experiment_settings.yml
#     #  --> These fields will be used to compute a lambda filtering function for each plot line
#     #  --> Define either as int or as a list with [lower_bound,upper_bound]
#     #
#     - nig_value : 0.1
#       ntfv_value : [0.1,0.3]
#     # ----//----//----//----//----//----//----//----//----//----//----//----//----
#
#       # Define which field from the experiments_settings.yml will be in the x axis
#       xfield : "ntfv_value"
#
#       # Define which field from the desired file captured during the batch execution to be on the y axis
#       ydata : 
#         file : "calibration_errors.csv"
#         field : "front_left_camera [px]"
#       # Legend for the plot. If not defined for any plot for a given figure, the final image won't have it.
#       legend : " front_left_camera"
#
#
#     # One can add overlapping plots by adding another entry in the data dictionary.
#     - nig_value : 0.1
#       ntfv_value : [0.1,0.3]
#       xfield : "ntfv_value"
#       ydata : 
#         file : "calibration_errors.csv"
#         field : "front_right_camera [px]"
#       legend : " front_right_camera"

Help section :

rosrun atom_batch_execution plot_graphs -df plot.yml -rf results_processed -of plots -sp -ow -h
usage: plot_graphs [-h] [-v] [-ow] -rf RESULTS_FOLDER -of OUTPUT_FOLDER -df DATA_FILENAME [-sp]

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         Prints the stdout_data of each command to the terminal.
  -ow, --overwrite      Overwrites output folder.
  -rf RESULTS_FOLDER, --results_folder RESULTS_FOLDER
                        Folder containing the processed results
  -of OUTPUT_FOLDER, --output_folder OUTPUT_FOLDER
                        Folder where to store the plots' images
  -df DATA_FILENAME, --data_filename DATA_FILENAME
                        Yaml containing variables used to compute the plots.
  -sp, --show_plots     Shows plots

Example execution :

Example graphs produced :

What needs to be improved/enhanced :

• Possibility to change limits of x and y axis, its automatic at the moment
• Possibly expand to more interesting graphs other than 2D plots, like histograms or even 3D surfs.
• And other small things too I'm sure but I'm too sleepy to think properly now.

@miguelriemoliveira @manuelgitgomes @Kazadhum I would like your opinion on this feature :)

[miguelriemoliveira]

This looks great! Congrats @brunofavs !

[brunofavs]

Possibly migrate to seaborn. Could move the plotting to a separate function and give the user the option to choose between seaborn/matplotlib.
Great feature but due to lack of time I'm postponing it