Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenCV calibration results for RIWMPBOT: Using -cpt flag doesn't change the results #965

Closed
Kazadhum opened this issue Jun 28, 2024 · 22 comments
Assignees
Labels
discussion Issue for discussing a topic

Comments

@Kazadhum
Copy link
Collaborator

Hello @miguelriemoliveira and @manuelgitgomes!

This issue is related to #912. When running the single_rgb_evaluation script for the real riwmpbot:

rosrun atom_evaluation single_rgb_evaluation -train_json $ATOM_DATASETS/riwmpbot_real/merged/hand_eye_tsai_rgb_world.json -test_json $ATOM_DATASETS/riwmpbot_real/merged/dataset.json -uic

we get:

Errors per collection
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
| Collection | rgb_world-forearm_pattern [px] | rgb_world-hand_pattern [px] | rgb_world-upper_arm_pattern [px] | all [px] |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
|    000     |              ---               |           16.1874           |             17.9642              | 17.0758  |
|    001     |              ---               |           40.3688           |               ---                | 40.3688  |
|    002     |            18.0104             |           17.6613           |               ---                | 17.8358  |
|    003     |            15.2647             |           12.0823           |             19.2538              | 15.5336  |
|    004     |            12.5290             |           21.1855           |               ---                | 16.8572  |
|    005     |            12.8367             |             ---             |               ---                | 12.8367  |
|    006     |              ---               |           22.3721           |               ---                | 22.3721  |
|    007     |            17.0504             |           15.6758           |             23.4709              | 18.7324  |
|    008     |              ---               |           21.4439           |             11.9072              | 16.6755  |
|    009     |              ---               |           15.1291           |             11.1548              | 13.1420  |
|    010     |            14.7559             |            9.1495           |               ---                | 11.9527  |
|    011     |              ---               |           75.8064           |             21.5539              | 48.6801  |
|    012     |            13.2013             |            6.9178           |             13.2680              | 11.1290  |
|    013     |            14.5878             |            7.4840           |             14.1242              | 12.0653  |
|    014     |            12.3743             |           11.9303           |             12.7859              | 12.3635  |
|    015     |            12.1077             |            7.9040           |             12.9636              | 10.9918  |
|    016     |            16.0507             |           15.7387           |             10.5250              | 14.1048  |
|    017     |            12.3408             |           14.2890           |             10.7092              | 12.4463  |
|    018     |            12.5703             |            8.6694           |             14.0718              | 11.7705  |
|    019     |            12.6222             |            8.8785           |             14.0640              | 11.8549  |
|    020     |            12.3549             |            9.1815           |             13.9210              | 11.8191  |
|    021     |            12.5369             |            8.4254           |             13.7464              | 11.5696  |
|    022     |            15.6758             |           11.3628           |             13.9776              | 13.6720  |
|    023     |            15.4064             |           12.2721           |               ---                | 13.8392  |
|    024     |            14.7710             |           17.3818           |             13.8704              | 15.3411  |
|    025     |            10.9966             |           21.9458           |             13.8156              | 15.5860  |
|    026     |              ---               |           18.1955           |              5.9865              | 12.0910  |
|    027     |             9.1156             |           25.7116           |             11.5430              | 15.4567  |
|    028     |             9.1889             |           12.1272           |             11.5278              | 10.9480  |
|    029     |            16.0854             |           13.4232           |             10.7438              | 13.4175  |
|    030     |            14.2482             |           20.0801           |              9.8539              | 14.7274  |
|    031     |            10.6545             |           12.8982           |              9.6996              | 11.0841  |
|    032     |            10.2569             |           17.8822           |              9.8786              | 12.6726  |
|    033     |             9.9701             |           14.9893           |               ---                | 12.4797  |
|    034     |            23.4957             |           20.6005           |              9.6060              | 17.9007  |
|    035     |            13.4241             |           16.6745           |              8.2331              | 12.7772  |
|    036     |              ---               |           15.7315           |               ---                | 15.7315  |
|    037     |              ---               |           14.7373           |               ---                | 14.7373  |
|    038     |              ---               |           20.9509           |               ---                | 20.9509  |
|    039     |              ---               |           12.8692           |             17.7784              | 15.3238  |
|    040     |              ---               |           18.0856           |              8.1911              | 13.1383  |
|    041     |            13.4349             |           17.8438           |               ---                | 15.6393  |
|    042     |            12.4400             |           22.2936           |               ---                | 17.3668  |
|    043     |              ---               |           15.2988           |             12.9546              | 14.1267  |
|    044     |            13.1669             |           12.6968           |              9.8356              | 11.8998  |
|    045     |            14.7645             |           31.6338           |             11.7435              | 19.3806  |
|    046     |            16.0500             |           12.7483           |               ---                | 14.3992  |
|    047     |            16.1103             |           12.4365           |             17.5215              | 15.3561  |
|    048     |            14.3394             |           17.4489           |             15.8972              | 15.8952  |
|    049     |            18.6939             |           12.3785           |               ---                | 15.5362  |
|    050     |            15.9424             |             ---             |             14.1353              | 15.0388  |
|    051     |            14.4328             |           20.2937           |               ---                | 17.3633  |
|    052     |            16.8556             |             ---             |             16.2511              | 16.5533  |
|    053     |            13.6989             |           15.5607           |             15.7976              | 15.0191  |
|    054     |            13.4410             |           16.0333           |             11.6631              | 13.7125  |
|    055     |            13.4557             |           11.2173           |             13.2751              | 12.6493  |
|    056     |            16.1554             |           13.2640           |               ---                | 14.7097  |
|    057     |            14.2936             |           25.6690           |             16.8080              | 18.9235  |
|    058     |            14.8951             |           17.7585           |             15.9702              | 16.2079  |
|    059     |              ---               |           41.2438           |             15.0717              | 28.1578  |
|    060     |              ---               |           30.1723           |               ---                | 30.1723  |
|    061     |              ---               |           18.1083           |             15.7013              | 16.9048  |
|    062     |            15.6351             |           12.7338           |             16.8903              | 15.0864  |
|  Averages  |            14.0912             |           17.6872           |             13.5490              | 16.0976  |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+

Following your suggestion, I checked whether the pattern transforms were being copied over to the mixed_dataset. They weren't, but the flag -cpt/--copy_pattern_transforms does it. The results are the same. The flag's implementation is as follows:

if args['copy_pattern_transforms']:
for pattern_key, pattern in train_dataset['calibration_config']['calibration_patterns'].items():
if pattern['fixed'] is True:
copyTFToDataset(pattern['parent_link'], pattern['link'], train_dataset, mixed_dataset)

Which led me to believe that, since the results were the same with and without using -cpt, maybe the calibration configuration didn't have the patterns as 'fixed'. However, they do.

@Kazadhum Kazadhum added the discussion Issue for discussing a topic label Jun 28, 2024
@Kazadhum Kazadhum self-assigned this Jun 28, 2024
@Kazadhum
Copy link
Collaborator Author

I think I found the issue. Basically, the OpenCV calibration doesn't save the calibrated pattern's TF to the calibrated dataset, which means it doesn't matter if the -cpt flag is used. Working on a fix now...

@Kazadhum Kazadhum changed the title OpenCV calibration results for RIWMPBOT: Using -cpt flag doesn't chenge the results OpenCV calibration results for RIWMPBOT: Using -cpt flag doesn't change the results Jun 28, 2024
@Kazadhum
Copy link
Collaborator Author

Turns out this wasn't the issue; the OpenCV calibration function used only returns the base-to-camera TF (in the eye-to-hand case). I remember I initially tried using another OpenCV function which also returned the other TF but I remember it not working so we settled on using cv2.calibrateHandEye().

Either way, I think these results might warrant further discussion in our next meeting.

@Kazadhum
Copy link
Collaborator Author

Kazadhum commented Jul 1, 2024

The idea now is to create a script that copies the pattern TFs from an ATOM-calibrated dataset to a target dataset, so that when we compare the results from the evaluation procedures, it's a fair comparison.

@Kazadhum
Copy link
Collaborator Author

Hi @miguelriemoliveira! A base working script is done!

An example to run:
rosrun atom_evaluation copy_tfs_from_dataset.py -sd $ATOM_DATASETS/riwmpbot_real/merged/atom_calibration.json -td $ATOM_DATASETS/riwmpbot_real/merged/dataset.json -pll flange -cll charuco_200x200_8x8

As we talked about today, before copying the tf, it checks whether that tf is static:

# ---------------------------------------
# Verifications
# ---------------------------------------
# Check if parent link list and child link list have the same number of elements
if len(args["parent_link_list"]) != len(args["child_link_list"]):
atomError('The parent link and child link lists have a different number of elements. Check your arguments.')
# Organize the -pll and -cll args into a list of tuples [(parent1,child1), ...]
parent_child_list = [(args["parent_link_list"][i], args["child_link_list"][i]) for i in range(len(args["parent_link_list"]))]
# Check if TF exists and if it is fixed; we do this by comparing the TF from one of the collections to each other
# I think it only makes sense to do this for the sensor dataset
is_fixed = True
for parent_child_tuple in parent_child_list:
tf_name = parent_child_tuple[0] + '-' + parent_child_tuple[1]
selected_collection_quat = source_dataset["collections"][selected_collection_key]["transforms"][tf_name]["quat"]
selected_collection_trans = source_dataset["collections"][selected_collection_key]["transforms"][tf_name]["trans"]
print("Selected collection quat (" + tf_name + "):")
print(selected_collection_quat)
for collection_key, collection in source_dataset["collections"].items():
if (selected_collection_quat == collection["transforms"][tf_name]["quat"] and selected_collection_trans == collection["transforms"][tf_name]["trans"]):
continue
else:
atomError("Indicated transform " + tf_name + "is not fixed!")

It then copies the tfs and outputs a new dataset!

I'll clean up some things about the script and test getting the results we wanted using this later today.

@miguelriemoliveira
Copy link
Member

miguelriemoliveira commented Jul 12, 2024 via email

@Kazadhum
Copy link
Collaborator Author

Kazadhum commented Jul 15, 2024

So the results aren't great. I'll lay out what I did.

First off, I ran the following:

rosrun atom_evaluation copy_tfs_from_dataset.py -sd $ATOM_DATASETS/riwmpbot_real/merged/atom_calibration.json -td $ATOM_DATASETS/riwmpbot_real/merged/dataset_test.json -pll flange forearm_link upper_arm_link -cll charuco_200x200_8x8 charuco_170x100_3x6 charuco_200x120_3x6

This copied the calibrated pattern TFs to the dataset used for testing. I copied these for the testing dataset because the mixed dataset generated during evaluation uses the test dataset's pattern TFs.

Afterwards, I ran the evaluation:

rosrun atom_evaluation single_rgb_evaluation -train_json $ATOM_DATASETS/riwmpbot_real/merged/hand_eye_tsai_rgb_world.json -test_json $ATOM_DATASETS/riwmpbot_real/merged/dataset_test_tfs_copied.json -uic

and got the results:

Errors per collection
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
| Collection | rgb_world-forearm_pattern [px] | rgb_world-hand_pattern [px] | rgb_world-upper_arm_pattern [px] | all [px] |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
|    020     |             4.2982             |           30.9588           |             15.7057              | 16.9876  |
|    022     |             8.2882             |           31.3949           |             15.4948              | 18.3926  |
|    024     |             7.1295             |           36.4628           |             15.5105              | 19.7009  |
|    026     |              ---               |           40.9061           |             16.7319              | 28.8190  |
|    028     |             2.9268             |           37.8922           |             15.6873              | 18.8354  |
|    029     |             9.9091             |           32.1625           |             15.7567              | 19.2761  |
|    032     |            12.9047             |           52.9646           |             15.8240              | 27.2311  |
|    038     |              ---               |           54.2720           |               ---                | 54.2720  |
|    039     |              ---               |           38.5529           |             23.8295              | 31.1912  |
|    041     |            17.2032             |           47.4689           |               ---                | 32.3361  |
|    042     |            29.1017             |           60.2927           |               ---                | 44.6972  |
|    049     |            30.0060             |           37.3603           |               ---                | 33.6831  |
|    052     |            25.9819             |             ---             |             22.8159              | 24.3989  |
|  Averages  |            14.7749             |           41.7241           |             17.4840              | 28.4478  |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+

I'm not sure what might be going wrong here. Will keep investigating...

EDIT: I accidentally closed the issue, and then reopened it.

@Kazadhum Kazadhum reopened this Jul 15, 2024
@Kazadhum
Copy link
Collaborator Author

Kazadhum commented Jul 15, 2024

I'll try to re-do the process from beginning to end to make sure nothing went wrong along the way. So this is the process:

  • Split dataset.json into dataset_train.json and dataset_test.json
  • Run ATOM Calibration
  • Run CV calibration (tsai) for the train dataset
  • Copy the pattern's TFs from the ATOM calibration dataset to the test dataset
  • Run evaluation

@miguelriemoliveira
Copy link
Member

I think that's it but 1 and 2 are not in order.

@miguelriemoliveira
Copy link
Member

miguelriemoliveira commented Jul 15, 2024

and got the results:

What you are doing (the procedure) seems to make sense.

Now we get these errors

Averages | 14.7749 | 41.7241 | 17.4840

but what were the errors reported for tsai before doing the copy_tf? Worse or better?

This copied the calibrated pattern TFs to the dataset used for testing.

did you test your script already? Did you confirm these transforms are copied?

@miguelriemoliveira
Copy link
Member

miguelriemoliveira commented Jul 15, 2024

We could do an alternative to your procedure, where we first inject the atom estimated pattern transformations to a new train_dataset.json which is used by the tsai method ... perhaps it makes no difference.

@Kazadhum
Copy link
Collaborator Author

I think that's it but 1 and 2 are not in order.

Edited it so it's in the right order now

but what were the errors reported for tsai before doing the copy_tf? Worse or better?

They were worse. Running with the same datasets (there are several variants of the same dataset in the folder):

Errors per collection
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
| Collection | rgb_world-forearm_pattern [px] | rgb_world-hand_pattern [px] | rgb_world-upper_arm_pattern [px] | all [px] |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
|    020     |            31.8233             |           14.2793           |             42.5092              | 29.5373  |
|    022     |            37.2385             |           18.8750           |             42.3106              | 32.8080  |
|    024     |            35.6258             |           29.1844           |             42.1935              | 35.6679  |
|    026     |              ---               |           30.2334           |             33.6481              | 31.9407  |
|    028     |            26.4336             |           24.7470           |             39.6163              | 30.2656  |
|    029     |            38.0187             |           15.4459           |             38.7572              | 30.7406  |
|    032     |            25.3285             |           38.7704           |             37.8667              | 33.9885  |
|    038     |              ---               |           40.4913           |               ---                | 40.4913  |
|    039     |              ---               |           30.2138           |             38.3071              | 34.2604  |
|    041     |            42.1981             |           39.5055           |               ---                | 40.8518  |
|    042     |            49.0016             |           51.3867           |               ---                | 50.1942  |
|    049     |            56.4271             |           29.3602           |               ---                | 42.8936  |
|    052     |            52.8242             |             ---             |             41.4615              | 47.1428  |
|  Averages  |            39.4919             |           30.2077           |             39.6300              | 36.9833  |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+

did you test your script already? Did you confirm these transforms are copied?

Yes, I double checked the datasets and the TFs were indeed copied...

We could do an alternative to your procedure, where we first inject the atom estimated pattern transformations to a new train_dataset.json which is used by the tsai method ... perhaps it makes no difference.

This option seems smarter I think, though I think the atom estimated pattern transforms need to go in the test dataset (I can also copy it to both the test and train datasets, just to make sure).

@miguelriemoliveira
Copy link
Member

This option seems smarter I think, though I think the atom estimated pattern transforms need to go in the test dataset (I can also copy it to both the test and train datasets, just to make sure).

Yes, that was my idea.

But create a second train dataset ...

@Kazadhum
Copy link
Collaborator Author

My new results, with the procedure made more explicit. To note: I'm using a copy of dataset_filtered_2.json in a different folder, which I named diogo_tests and will be uploading to the NAS once I get home.

Splitting the dataset

rosrun atom_calibration split_atom_dataset -json $ATOM_DATASETS/riwmpbot_real/diogo_tests/dataset_filtered.json -tp 70

Running ATOM calibration:

rosrun atom_calibration calibrate -json $ATOM_DATASETS/riwmpbot_real/diogo_tests/dataset_filtered_train.json -uic -v

Results:

Errors per collection (anchored sensor,  max error per sensor, not detected as "---")
+------------+----------------+
| Collection | rgb_world [px] |
+------------+----------------+
|    000     |     1.3764     |
|    002     |     1.1596     |
|    003     |     1.0611     |
|    005     |     1.8711     |
|    008     |     1.2267     |
|    009     |     1.5285     |
|    013     |     1.4239     |
|    015     |     1.1903     |
|    017     |     1.3284     |
|    018     |     1.2473     |
|    020     |     1.0048     |
|    021     |     1.0539     |
|    022     |     1.4039     |
|    023     |     1.2587     |
|    024     |     1.2427     |
|    025     |     7.9379     |
|    027     |    11.8498     |
|    028     |     1.7224     |
|    029     |     4.2113     |
|    030     |     1.5674     |
|    032     |     7.9976     |
|    035     |     1.3805     |
|    036     |     1.2026     |
|    037     |     0.9917     |
|    038     |     1.5707     |
|    039     |     1.0164     |
|    040     |     1.1030     |
|    043     |     0.7522     |
|    044     |     0.8830     |
|    047     |     1.1439     |
|    049     |     1.4810     |
|    050     |     0.4854     |
|    051     |     0.9175     |
|    052     |     0.8133     |
|    053     |     0.9848     |
|    054     |     0.9714     |
|    055     |     0.7784     |
|    056     |     1.0362     |
|    057     |     9.0901     |
|    058     |     0.9189     |
|    059     |     2.0956     |
|    060     |     1.7169     |
|  Averages  |     2.0476     |
+------------+----------------+

Copying the ATOM-calibrated pattern transformations to the test and train datasets

rosrun atom_evaluation copy_tfs_from_dataset.py -sd $ATOM_DATASETS/riwmpbot_real/diogo_tests/atom_calibration.json -td $ATOM_DATASETS/riwmpbot_real/diogo_tests/dataset_filtered_test.json -pll flange forearm_link upper_arm_link -cll charuco_200x200_8x8 charuco_170x100_3x6 charuco_200x120_3x6

rosrun atom_evaluation copy_tfs_from_dataset.py -sd $ATOM_DATASETS/riwmpbot_real/diogo_tests/atom_calibration.json -td $ATOM_DATASETS/riwmpbot_real/diogo_tests/dataset_filtered_train.json -pll flange forearm_link upper_arm_link -cll charuco_200x200_8x8 charuco_170x100_3x6 charuco_200x120_3x6

Calibrating with tsai

rosrun atom_evaluation cv_eye_to_hand.py -json $ATOM_DATASETS/riwmpbot_real/diogo_tests/dataset_filtered_train_tfs_copied.json -c rgb_world -p hand_pattern -hl flange -bl base_link -ctgt -uic -mn tsai -sfr

Results:

Ground Truth b_T_c=
[[ 0.     0.259 -0.966  0.95 ]
 [ 1.     0.     0.     0.35 ]
 [ 0.    -0.966 -0.259  0.8  ]
 [ 0.     0.     0.     1.   ]]
estimated b_T_c=
[[ 0.109  0.173 -0.979  0.848]
 [ 0.992 -0.083  0.095  0.325]
 [-0.065 -0.981 -0.18   0.806]
 [ 0.     0.     0.     1.   ]]
Etrans = 12.769 (mm)
Erot = 4.942 (deg)
+----------------------+----------+------------+
|      Transform       |  Et [m]  | Erot [rad] |
+----------------------+----------+------------+
| world-rgb_world_link | 0.051442 |  0.08647   |
+----------------------+----------+------------+

Evaluation:

rosrun atom_evaluation single_rgb_evaluation -train_json $ATOM_DATASETS/riwmpbot_real/diogo_tests/hand_eye_tsai_rgb_world.json -test_json $ATOM_DATASETS/riwmpbot_real/diogo_tests/dataset_filtered_test_tfs_copied.json -uic

Results:

Errors per collection
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
| Collection | rgb_world-forearm_pattern [px] | rgb_world-hand_pattern [px] | rgb_world-upper_arm_pattern [px] | all [px] |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
|    001     |              ---               |           116.5696          |               ---                | 116.5696 |
|    004     |            48.5999             |           84.7744           |               ---                | 66.6871  |
|    006     |              ---               |           68.6656           |               ---                | 68.6656  |
|    007     |            50.1819             |           62.8507           |             83.4875              | 65.5067  |
|    010     |            27.1802             |           68.2652           |               ---                | 47.7227  |
|    012     |            28.5182             |           68.3223           |             68.8623              | 55.2343  |
|    014     |            29.9734             |           58.8814           |             67.6381              | 52.1643  |
|    016     |            21.5303             |           51.3686           |             67.6527              | 46.8505  |
|    019     |            28.9181             |           65.5064           |             68.4807              | 54.3017  |
|    026     |              ---               |           59.1901           |             68.4876              | 63.8388  |
|    031     |            36.1384             |           60.6747           |             68.0662              | 54.9598  |
|    033     |            38.2886             |           64.0277           |               ---                | 51.1582  |
|    034     |            19.6634             |           65.7725           |             68.3388              | 51.2582  |
|    041     |            34.6151             |           58.0650           |               ---                | 46.3401  |
|    042     |            44.0757             |           81.8453           |               ---                | 62.9605  |
|    046     |            71.8332             |           70.2443           |               ---                | 71.0388  |
|    048     |            64.8605             |           76.5416           |             76.6835              | 72.6952  |
|    061     |              ---               |           63.3847           |             77.9106              | 70.6476  |
|    062     |            42.2372             |           57.3939           |             76.6138              | 58.7483  |
|  Averages  |            39.1076             |           68.5444           |             72.0202              | 61.9657  |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+

So clearly there's something wrong somewhere... Will keep investigating.

@miguelriemoliveira
Copy link
Member

Hi @Kazadhum ,

thanks for the clear and detailed post. I do not see anything conceptually wrong with what you are doing.
If you want we can talk tomorrow.

@Kazadhum
Copy link
Collaborator Author

Hi @miguelriemoliveira!

Thanks! I think it might be best if we meet, to analyze the problem together. In the meanwhile, I'll try to look at the different scripts to check if everything makes sense or if some script is using the wrong TFs or something of that nature...

@Kazadhum
Copy link
Collaborator Author

Just fixed the cv_eye_to_hand.py script to support partial detections!

Simulated results for the Tsai method seem to validate this change:

After filtering, will use 45 collections: ['000', '001', '002', '003', '004', '005', '006', '007', '008', '009', '010', '011', '012', '013', '014', '015', '016', '017', '018', '019', '020', '022', '023', '024', '025', '026', '027', '028', '029', '030', '031', '032', '033', '034', '035', '036', '037', '038', '039', '040', '041', '042', '043', '044', '045']
Selected collection key is 000
Ground Truth b_T_c=
[[ 1.    -0.     0.002  0.37 ]
 [-0.002 -0.309  0.951 -0.781]
 [ 0.    -0.951 -0.309  0.883]
 [ 0.     0.     0.     1.   ]]
estimated b_T_c=
[[ 1.    -0.005  0.004  0.368]
 [-0.006 -0.309  0.951 -0.777]
 [-0.003 -0.951 -0.309  0.883]
 [ 0.     0.     0.     1.   ]]
Etrans = 0.167 (mm)
Erot = 0.148 (deg)
+--------------------------------------+----------+------------+
|              Transform               |  Et [m]  | Erot [rad] |
+--------------------------------------+----------+------------+
| tripod_center_support-rgb_world_link | 0.002176 |  0.002587  |
+--------------------------------------+----------+------------+

I will now check if this works with the real system.

@miguelriemoliveira
Copy link
Member

Nice. This is very good news. Congratullations.

@Kazadhum
Copy link
Collaborator Author

New results, step-by-step, now using partial detections:

Splitting the dataset:

rosrun atom_calibration split_atom_dataset -json $ATOM_DATASETS/riwmpbot_real/diogo_tests/dataset_filtered.json -tp 70

Running ATOM calibration:

rosrun atom_calibration calibrate -json $ATOM_DATASETS/riwmpbot_real/diogo_tests/dataset_filtered_train.json -uic -v

Results:

-----------------------------
Optimization finished in 22.73863 secs: `ftol` termination condition is satisfied.
Errors per collection (anchored sensor,  max error per sensor, not detected as "---")
+------------+----------------+
| Collection | rgb_world [px] |
+------------+----------------+
|    000     |     1.9090     |
|    001     |     2.1246     |
|    005     |     2.2832     |
|    007     |     1.7333     |
|    008     |     1.0726     |
|    009     |     1.5787     |
|    012     |     0.7597     |
|    013     |     1.1691     |
|    015     |     1.3520     |
|    016     |     1.2844     |
|    017     |     1.3695     |
|    018     |     1.1951     |
|    019     |     1.0625     |
|    020     |     1.1252     |
|    021     |     0.9778     |
|    022     |     1.1258     |
|    023     |     1.0741     |
|    026     |     1.4969     |
|    027     |    12.6096     |
|    028     |     2.2095     |
|    029     |     2.9146     |
|    031     |     1.5675     |
|    032     |     9.7631     |
|    034     |    11.1804     |
|    035     |     2.0615     |
|    036     |     1.4300     |
|    040     |     0.8513     |
|    041     |     1.1899     |
|    042     |     1.8635     |
|    043     |     0.8877     |
|    046     |     1.3733     |
|    047     |     1.0710     |
|    048     |     1.0910     |
|    049     |     1.6751     |
|    050     |     0.8172     |
|    051     |     0.7591     |
|    053     |     1.3492     |
|    055     |     0.7948     |
|    057     |     9.4277     |
|    059     |     1.8455     |
|    060     |     1.5044     |
|    062     |     1.0565     |
|  Averages  |     2.2854     |
+------------+----------------+

Copying the ATOM-calibrated pattern transformations to the test and train datasets

rosrun atom_evaluation copy_tfs_from_dataset.py -sd $ATOM_DATASETS/riwmpbot_real/diogo_tests/atom_calibration.json -td $ATOM_DATASETS/riwmpbot_real/diogo_tests/dataset_filtered_test.json -pll flange forearm_link upper_arm_link -cll charuco_200x200_8x8 charuco_170x100_3x6 charuco_200x120_3x6

rosrun atom_evaluation copy_tfs_from_dataset.py -sd $ATOM_DATASETS/riwmpbot_real/diogo_tests/atom_calibration.json -td $ATOM_DATASETS/riwmpbot_real/diogo_tests/dataset_filtered_train.json -pll flange forearm_link upper_arm_link -cll charuco_200x200_8x8 charuco_170x100_3x6 charuco_200x120_3x6

Calibrating with Tsai:

rosrun atom_evaluation cv_eye_to_hand.py -json $ATOM_DATASETS/riwmpbot_real/diogo_tests/dataset_filtered_train_tfs_copied.json -c rgb_world -p hand_pattern -hl flange -bl base_link -uic -mn tsai

Evaluation:

rosrun atom_evaluation single_rgb_evaluation -train_json $ATOM_DATASETS/riwmpbot_real/diogo_tests/hand_eye_tsai_rgb_world.json -test_json $ATOM_DATASETS/riwmpbot_real/diogo_tests/dataset_filtered_test_tfs_copied.json -uic

Results:

Errors per collection
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
| Collection | rgb_world-forearm_pattern [px] | rgb_world-hand_pattern [px] | rgb_world-upper_arm_pattern [px] | all [px] |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
|    002     |            249.0149            |           210.8102          |               ---                | 229.9126 |
|    003     |            189.0249            |           127.1891          |             272.9342             | 196.3827 |
|    004     |            257.9503            |           261.8265          |               ---                | 259.8884 |
|    006     |              ---               |           113.1503          |               ---                | 113.1503 |
|    010     |            111.9038            |           202.8443          |               ---                | 157.3741 |
|    014     |            69.0700             |           118.2856          |             191.4684             | 126.2747 |
|    024     |            100.4533            |           76.8384           |             195.3046             | 124.1988 |
|    025     |            55.0044             |           136.4222          |             195.2363             | 128.8876 |
|    030     |            104.9610            |           84.7881           |             192.5299             | 127.4264 |
|    033     |            49.7408             |           141.5100          |               ---                | 95.6254  |
|    037     |              ---               |           136.2527          |               ---                | 136.2527 |
|    038     |              ---               |           155.0993          |               ---                | 155.0993 |
|    039     |              ---               |           126.7555          |             281.2948             | 204.0252 |
|    044     |            137.0123            |           133.3534          |             216.0034             | 162.1230 |
|    052     |            265.7035            |             ---             |             274.5523             | 270.1279 |
|    054     |            190.2166            |           173.5052          |             239.5485             | 201.0901 |
|    056     |            217.3809            |           160.1219          |               ---                | 188.7514 |
|    058     |            221.4238            |           152.1699          |             268.1086             | 213.9008 |
|    061     |              ---               |           145.4629          |             265.6546             | 205.5588 |
|  Averages  |            158.4900            |           147.5770          |             235.6941             | 173.4763 |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+

There's probably some detail wrong somewhere... I'll keep investigating after lunch.

@miguelriemoliveira
Copy link
Member

miguelriemoliveira commented Jul 17, 2024

There's probably some detail wrong somewhere... I'll keep investigating after lunch

Yeah. Something's going wrong still. I think after finding the problem with the partial collections we should go back and test without copying the pattern transforms to the train and test.

In those cases, what do we get?

In any case do not give up. You are close to the solution, I think.

By the way, if you run this experiment in the simulated dataset (where you know Tsai actually calibrated very well) what do you get in the evaluation?

@Kazadhum
Copy link
Collaborator Author

Hi @miguelriemoliveira!

Yeah. Something's going wrong still. I think after finding the problem with the partial collections we should go back and test without copying the pattern transforms to the train and test.

Just did this, and the results weren't much better.

I re-did the OpenCV calibration on a dataset without the copied TFs and ran the evaluation using a test dataset without copied TFs. The results:

Errors per collection
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
| Collection | rgb_world-forearm_pattern [px] | rgb_world-hand_pattern [px] | rgb_world-upper_arm_pattern [px] | all [px] |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
|    002     |            250.4335            |           225.9548          |               ---                | 238.1941 |
|    003     |            191.0826            |           138.6962          |             284.4266             | 204.7351 |
|    004     |            259.2164            |           280.2662          |               ---                | 269.7413 |
|    006     |              ---               |           130.3819          |               ---                | 130.3819 |
|    010     |            111.3733            |           203.0632          |               ---                | 157.2183 |
|    014     |            67.8253             |           122.2924          |             209.7189             | 133.2789 |
|    024     |            98.0107             |           85.5595           |             214.2771             | 132.6158 |
|    025     |            54.6084             |           142.5132          |             214.2026             | 137.1081 |
|    030     |            104.0178            |           97.9303           |             207.4952             | 136.4811 |
|    033     |            51.9518             |           150.7526          |               ---                | 101.3522 |
|    037     |              ---               |           150.5074          |               ---                | 150.5074 |
|    038     |              ---               |           172.5333          |               ---                | 172.5333 |
|    039     |              ---               |           140.4444          |             277.5505             | 208.9975 |
|    044     |            138.7631            |           140.2097          |             226.2732             | 168.4153 |
|    052     |            266.0931            |             ---             |             276.2865             | 271.1898 |
|    054     |            191.9570            |           188.5460          |             245.4553             | 208.6528 |
|    056     |            220.4130            |           174.4837          |               ---                | 197.4484 |
|    058     |            222.9370            |           168.8970          |             274.0001             | 221.9447 |
|    061     |              ---               |           162.4351          |             271.7120             | 217.0736 |
|  Averages  |            159.1916            |           159.7482          |             245.5816             | 181.9931 |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+

So there's something wrong here still...

By the way, if you run this experiment in the simulated dataset (where you know Tsai actually calibrated very well) what do you get in the evaluation?

Testing first without copying the ATOM-estimated pattern TFs...

OpenCV calibration of the train dataset:

rosrun atom_evaluation cv_eye_to_hand.py -json $ATOM_DATASETS/riwmpbot/train/dataset_train.json -c rgb_world -p hand_pattern -hl flange -bl base_link -ctgt -uic -mn tsai -sfr

Results:

After filtering, will use 31 collections: ['001', '002', '003', '004', '005', '007', '008', '009', '010', '012', '016', '018', '019', '020', '025', '026', '027', '028', '029', '030', '031', '032', '033', '034', '035', '036', '037', '038', '040', '043', '045']
Selected collection key is 001
Ground Truth b_T_c=
[[ 1.    -0.     0.002  0.37 ]
 [-0.002 -0.309  0.951 -0.781]
 [ 0.    -0.951 -0.309  0.883]
 [ 0.     0.     0.     1.   ]]
estimated b_T_c=
[[ 1.    -0.002  0.005  0.367]
 [-0.005 -0.309  0.951 -0.778]
 [-0.001 -0.951 -0.309  0.882]
 [ 0.     0.     0.     1.   ]]
Etrans = 0.017 (mm)
Erot = 0.095 (deg)
+--------------------------------------+----------+------------+
|              Transform               |  Et [m]  | Erot [rad] |
+--------------------------------------+----------+------------+
| tripod_center_support-rgb_world_link | 0.002023 |  0.001668  |
+--------------------------------------+----------+------------+

So like we saw in the post above, all good so far. Now for the evaluation against itself (sanity check):

rosrun atom_evaluation single_rgb_evaluation -train_json $ATOM_DATASETS/riwmpbot/train/hand_eye_tsai_rgb_world.json -test_json $ATOM_DATASETS/riwmpbot/train/hand_eye_tsai_rgb_world.json -uic

Results:

Errors per collection
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
| Collection | rgb_world-forearm_pattern [px] | rgb_world-hand_pattern [px] | rgb_world-upper_arm_pattern [px] | all [px] |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
|    001     |             1.1232             |            1.6209           |              3.1173              |  1.9538  |
|    002     |             0.8004             |            1.2368           |              3.1112              |  1.7161  |
|    003     |             1.1465             |            1.2887           |              3.2077              |  1.8809  |
|    004     |             1.5929             |            1.5751           |              3.1652              |  2.1111  |
|    005     |              ---               |            1.3505           |              2.9597              |  2.1551  |
|    007     |              ---               |            1.7540           |               ---                |  1.7540  |
|    008     |             0.8573             |            1.2769           |              3.1679              |  1.7674  |
|    009     |             1.0782             |            1.5807           |              3.1417              |  1.9335  |
|    010     |             1.0709             |            1.5780           |              3.1405              |  1.9298  |
|    012     |             1.5784             |            1.3003           |              3.4173              |  2.0987  |
|    016     |             0.9697             |            1.6829           |              2.9477              |  1.8668  |
|    018     |             1.4438             |            2.3920           |              2.6675              |  2.1678  |
|    019     |             1.2430             |            2.1710           |              2.6919              |  2.0353  |
|    020     |             1.2436             |            1.9297           |              2.6913              |  1.9548  |
|    025     |             1.1383             |            2.7628           |              2.7921              |  2.2311  |
|    026     |             1.1575             |            1.8397           |              2.7008              |  1.8993  |
|    027     |             1.1241             |            2.4978           |              2.7397              |  2.1205  |
|    028     |             0.7145             |            1.2530           |              2.6656              |  1.5444  |
|    029     |             0.3299             |            1.0606           |              2.8579              |  1.4161  |
|    030     |             1.1170             |            1.0638           |              3.1784              |  1.7864  |
|    031     |             1.6291             |            1.5872           |              3.2856              |  2.1673  |
|    032     |             1.6410             |            1.5777           |              3.2849              |  2.1679  |
|    033     |             1.1667             |            1.9961           |              2.8353              |  1.9994  |
|    034     |             1.2831             |            3.2580           |              2.8103              |  2.4504  |
|    035     |             0.7928             |            2.5605           |              2.7946              |  2.0493  |
|    036     |             1.4278             |            2.2573           |               ---                |  1.8425  |
|    037     |             1.1635             |            1.9191           |              2.6025              |  1.8950  |
|    038     |             1.1749             |            1.5781           |              3.1036              |  1.9522  |
|    040     |             1.6951             |            1.8358           |              3.2111              |  2.2473  |
|    043     |             1.4893             |            1.5164           |              3.2608              |  2.0888  |
|    045     |             1.7980             |            1.2732           |              2.9743              |  2.0152  |
|  Averages  |             1.2066             |            1.7605           |              2.9836              |  1.9741  |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+

Also looks good here. Now we perform the actual evaluation (train dataset vs. test dataset):

rosrun atom_evaluation single_rgb_evaluation -train_json $ATOM_DATASETS/riwmpbot/train/hand_eye_tsai_rgb_world.json -test_json $ATOM_DATASETS/riwmpbot/train/dataset_test.json -uic

Results:

Errors per collection
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
| Collection | rgb_world-forearm_pattern [px] | rgb_world-hand_pattern [px] | rgb_world-upper_arm_pattern [px] | all [px] |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
|    000     |             0.8703             |            1.4097           |              2.7239              |  1.6680  |
|    006     |             1.0189             |            1.5183           |              3.3349              |  1.9574  |
|    011     |             1.1764             |            1.5439           |              3.2106              |  1.9770  |
|    013     |             1.4106             |            1.7189           |              3.1038              |  2.0777  |
|    014     |             1.3960             |            2.3625           |              2.9287              |  2.2291  |
|    015     |             1.0045             |            1.6752           |              2.9444              |  1.8747  |
|    017     |             1.4213             |            2.1859           |              2.9531              |  2.1868  |
|    022     |             0.5842             |            2.1581           |              2.7879              |  1.8434  |
|    023     |             1.4525             |            2.9199           |              2.7657              |  2.3793  |
|    024     |             1.3612             |            3.0360           |              2.7991              |  2.3988  |
|    039     |             1.7359             |            1.8181           |              3.1570              |  2.2370  |
|    041     |             1.2178             |            1.3912           |              3.0044              |  1.8711  |
|    042     |             0.6219             |            1.1020           |              2.9743              |  1.5661  |
|    044     |             2.4345             |            1.7098           |               ---                |  2.0722  |
|  Averages  |             1.2647             |            1.8964           |              2.9760              |  2.0242  |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+

So these results look good, but they'll probably look better if I copy the pattern TFs from an ATOM calibration. When I copy the pattern TFs, I get:

Errors per collection
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
| Collection | rgb_world-forearm_pattern [px] | rgb_world-hand_pattern [px] | rgb_world-upper_arm_pattern [px] | all [px] |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
|    000     |             0.9589             |            1.0241           |              2.7391              |  1.5740  |
|    006     |             1.0834             |            1.4364           |              3.3423              |  1.9540  |
|    011     |             1.3786             |            1.1923           |              3.2382              |  1.9363  |
|    013     |             1.6003             |            1.3309           |              3.1310              |  2.0207  |
|    014     |             1.6654             |            1.7186           |              2.9603              |  2.1148  |
|    015     |             1.3059             |            1.2189           |              2.9762              |  1.8337  |
|    017     |             1.6890             |            1.6483           |              2.9839              |  2.1071  |
|    022     |             0.7702             |            1.7303           |              2.8160              |  1.7722  |
|    023     |             1.6777             |            2.3205           |              2.7935              |  2.2639  |
|    024     |             1.5613             |            2.4933           |              2.8280              |  2.2942  |
|    039     |             2.0315             |            1.3395           |              3.1915              |  2.1875  |
|    041     |             1.3841             |            1.2686           |              3.0287              |  1.8938  |
|    042     |             0.7933             |            0.9304           |              2.9968              |  1.5735  |
|    044     |             2.6725             |            1.4490           |               ---                |  2.0607  |
|  Averages  |             1.4694             |            1.5072           |              3.0020              |  1.9705  |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+

miguelriemoliveira added a commit that referenced this issue Jul 18, 2024
Kazadhum added a commit that referenced this issue Jul 18, 2024
@Kazadhum
Copy link
Collaborator Author

Hi @miguelriemoliveira! In light of today's progress, do you think we can close this issue?

@miguelriemoliveira
Copy link
Member

Yes, its very big now. Might as well start a new one if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discussion Issue for discussing a topic
Development

No branches or pull requests

2 participants