Ratios on a list of notes (n*n) comparison

[pradeepranwa1]

I have a list of transactions with transaction notes I want to compare each note with all other notes, so total nn comparison, which is taking lot of time in big data, I want to speed up using parallelization, Is there any way i can send all n notes and it gives me nn comparison

[pradeepranwa1]

My pseudo code looks like

trans = [dicts]

match_list = []
already_match_set = set()

for tranx in trans:
     cur_match_list = []
     if transx["hash"] not in already_match_set:
            cur_match_list.append(transx)
            already_match_set.add(transx["hash"])
            for transy in trans:
                   notex = transx['note']
                   notey = transy['note']
                   fuzz_ratio=max(partial_token_sort_ratio,ratio,token_set_ratio on notex,notey)
                   
                   condition= now few condtions on length of note and fuzz_ratio to check similarity 
                   if contidion and transy['hash'] not in already_match_set
                        cur_match_list.append(transy)
                        already_match_set.append(transy['hash'])
       if len(cur_match_list)>2
             match_list.append(cur_match_list)
             
             

[pradeepranwa1]

I found solution for this, which is using cdist

process.cdist(list, list, scorer=rfuzz.partial_token_sort_ratio,workers=6,score_cutoff=59)

[maxbachmann]

Yes you would probably want to:

1. deduplicate the list based on the hash
2. use cdist for the matching
3. use numpy to further process the results

This means that for a combined condition like you used you would either need to run cdist multiple times or write your own scorer in C/C++ that combines the two. Writing your own scorer isn't really documented and certainly quite a bit more work, but can be faster. Unless it's necessary for performance reasons I would just run cdist multiple times.