Improve NCH with a better strategy on samples selection

[toncho11]

Current strategy:
Using random samples for the hulls.

Closest n matrices :
For each class we calculate the closest n (n=10) matrices to the test sample. This will be done using distance_logeuclid() from pyriemann. Next we use these n + n matrices to calculate again 2 distances using qdistance_logeuclid_to_convex_hull(). The shorter distance predicts the class.

[toncho11]

Using the closest covmats to the test sample:

2024-03-14 13:54:48,785
BrainInvaders2013a-WithinSession: 100%|██████████| 3/3 [1:17:35<00:00, 1551.69s/it]
Averaging the session performance:
             score       time
pipeline                     
XD+MDM    0.912369   0.182806
XD+NCH    0.762527  38.410767

With 20 covmats per class/hull and SlsqpOptimizer.

=========================================================

BrainInvaders2013a-WithinSession: 100%|██████████| 3/3 [54:58<00:00, 1099.57s/it]
Averaging the session performance:
              score       time
pipeline                      
XD+MDM     0.912933   0.189794
XD+NCH     0.749386   5.466629
XD+NCH+LR  0.875280  21.609724

with n_samples_per_hull=5

=========================================================

BrainInvaders2013a-WithinSession: 100%|██████████| 1/1 [31:54<00:00, 1914.18s/it]
Averaging the session performance:
                       score       time
pipeline                               
XD+MDM              0.948125   0.190382
XD+NCH+MIN_HULL     0.847031  13.374832
XD+NCH+RANDOM_HULL  0.890312  34.032013

with:

• QuanticNCH(n_hulls_per_class=1, n_samples_per_hull=10,quantum=True,n_jobs=12, strategy="min-hull")
• QuanticNCH(n_hulls_per_class=3, n_samples_per_hull=10,quantum=True,n_jobs=12, strategy="random-hull")

[toncho11]

Implemented in #253 .

[toncho11]

Here is another result:

BrainInvaders2013a-WithinSession: 100%|██████████| 3/3 [1:33:19<00:00, 1866.45s/it]
Averaging the session performance:
                    score       time
pipeline                            
NCH+MIN_HULL     0.756123  12.676102
NCH+RANDOM_HULL  0.821275  33.578224
XD+MDM           0.912214   0.190351

Using:

QuanticNCH(
        n_hulls_per_class=3,
        n_samples_per_hull=10,
        n_jobs=12,
        hull_type="random-hull",
        quantum=True,
    )
 QuanticNCH(
        n_hulls_per_class=1,
        n_samples_per_hull=10,
        n_jobs=12,
        hull_type="min-hull",
        quantum=True,
    )

As this has Quantum = True it uses the Cobyla optimizer.

Random hull probably varies on each run, but in general it gives better performance currently. But it is worth mentioning that min-hull uses 10 samples and random-hull uses 3 x 10 = 30.

[toncho11]

BrainInvaders2013a-WithinSession: 100%|██████████| 3/3 [3:12:07<00:00, 3842.46s/it]  
Averaging the session performance:
                    score       time
pipeline                            
NCH+MIN_HULL     0.759261  60.735703
NCH+RANDOM_HULL  0.827175  34.919315
XD+MDM           0.911268   0.186330

QuanticNCH(
       n_hulls_per_class=3,
       n_samples_per_hull=10,
       n_jobs=12,
       hull_type="random-hull",
       quantum=True,
   )

QuanticNCH(
       n_hulls_per_class=1,
       n_samples_per_hull=30,
       n_jobs=12,
       hull_type="min-hull",
       quantum=True,
   )

Quantum = True using Cobyla optimizer, 3 subjects

[gcattan]

1. Quantum=True should use QAOA + MinimumEigenOptimizer. Can you check this?

2. In general, it looks like it is working better with random strategy and quantum=True ?

3. Can you try to do NCH with quantum=True and random strategy, and at the end LR or LDA?

[toncho11]

1. I used:
   print("Optimizer name check:", optimizer.__class__.__name__) in the weights_logeuclid_to_convex_hull()
   and it says NaiveQAOAOptimizer when using Quantum = True
2. Random yes, but changing the optimizer does not make a difference. I just tested:

3. BrainInvaders2013a-WithinSession: 100%|██████████| 3/3 [2:23:14<00:00, 2864.80s/it]
Averaging the session performance:
                                 score       time
pipeline                                         
NCH+RANDOM_HULL_NO_QUANTUM    0.820808  35.493435
NCH+RANDOM_HULL_WITH_QUANTUM  0.828150  35.707268
XD+MDM                        0.912771   0.192875

[gcattan]

1. Ok, so it means that the log message is not right. It displays COBYLA by default, right?
2. Ok, thanks for checking.

[qbarthelemy]

Before understanding the impact of subsampling strategies on quantum NCH,
is it possible to have a comparison between non-quantum classifiers: MDM versus standard NCH (ie, one hull by class, with all the matrices) using a classical optimizer?

[gcattan]

Ok, we may try this. But this will take maybe one week or two to get the results.
I know Anton is trying to push some examples on a grid.
@qbarthelemy do you have maybe this possibility too?

[toncho11]

Before understanding the impact of subsampling strategies on quantum NCH, is it possible to have a comparison between non-quantum classifiers: MDM versus standard NCH (ie, one hull by class, with all the matrices) using a classical optimizer?

We have this result that I did on 13/03. It was non-quantum:

(n_hulls=1, n_samples_per_hull=-1) for one subject and it takes more than 13 hours already on my laptop for 1 subject with MOABB. I have to stop it.
Here are some half-way results:

XD+NCH | BrainInvaders2013a | 1 | 1: Score 0.896
XD+MDM | BrainInvaders2013a | 1 | 1: Score 0.970

XD+NCH | BrainInvaders2013a | 1 | 2: Score 0.754
XD+MDM | BrainInvaders2013a | 1 | 2: Score 0.890