Convert

[amn41]

No description provided.

[github-actions]

Intent Cross-Validation Results (3 folds)

class	support	f1-score	confused_with
macro avg	337	0.9520	N/A
weighted avg	337	0.9581	N/A
inform	78	0.9542	pay_cc(4), greet(1)
check_balance	47	0.9783	search_transactions(1), pay_cc(1)
transfer_money	36	0.9730	N/A
pay_cc	33	0.9296	N/A
search_transactions	23	0.9787	N/A
greet	20	0.9756	N/A
affirm	15	1.0000	N/A
check_recipients	13	0.9167	transfer_money(1), goodbye(1)
check_earnings	13	1.0000	N/A
thankyou	11	0.9565	N/A
deny	10	0.9000	inform(1)
help	10	1.0000	N/A
goodbye	10	0.8421	thankyou(1), deny(1)
ask_transfer_charge	9	1.0000	N/A
human_handoff	9	0.8750	transfer_money(1), inform(1)

Entity Cross-Validation Results (5 folds)

entity	support	f1-score	precision	recall
micro avg	244	0.9463	0.9542	0.9385
macro avg	244	0.9507	0.9604	0.9415
weighted avg	244	0.9461	0.9540	0.9385
credit_card	103	0.9463	0.9510	0.9417
account_type	86	0.9474	0.9529	0.9419
payment_amount	34	0.9091	0.9375	0.8824
vendor_name	21	1.0000	1.0000	1.0000

[amn41]

wdyt @akelad & @melindaloubser1 ? Not sure if we should merge this, maybe just in a branch for eval from the CDD hackathon? Including both convert & spacy in this starter pack seems a bit heavy. ( have taken spaCy out in this PR for the purpose of testing)

[indam23]

Ja I think originally we removed convert because of the heaviness - On the other hand, maybe it's good if one starter pack uses it.

[akelad]

what were the original results like? we def need spacy in :D

[amn41]

see here: #71 (comment)

so weighted average intent f1 goes from 0.86 to 0.96 (!)

entities not really affected

[indam23]

put spacy back so we can see how long eval takes. Isn't it also a thing that windows users can't use convert right now because of tensorflow_text? That would be a reason not to do it.

[github-actions]

Intent Cross-Validation Results (3 folds)

class	support	f1-score	confused_with
macro avg	337	0.9226	N/A
weighted avg	337	0.9355	N/A
inform	78	0.9342	pay_cc(5), transfer_money(2)
check_balance	47	0.9677	search_transactions(1), pay_cc(1)
transfer_money	36	0.9351	N/A
pay_cc	33	0.9014	transfer_money(1)
search_transactions	23	0.9583	N/A
greet	20	1.0000	N/A
affirm	15	1.0000	N/A
check_earnings	13	1.0000	N/A
check_recipients	13	0.9600	goodbye(1)
thankyou	11	0.9167	N/A
goodbye	10	0.8571	thankyou(1)
help	10	1.0000	N/A
deny	10	0.6667	inform(2), thankyou(1)
human_handoff	9	0.8000	transfer_money(1), search_transactions(1)
ask_transfer_charge	9	0.9412	transfer_money(1)

Entity Cross-Validation Results (5 folds)

entity	support	f1-score	precision	recall
micro avg	244	0.9390	0.9315	0.9467
macro avg	244	0.9444	0.9336	0.9557
weighted avg	244	0.9390	0.9316	0.9467
credit_card	103	0.9378	0.9245	0.9515
account_type	86	0.9357	0.9412	0.9302
payment_amount	34	0.9275	0.9143	0.9412
vendor_name	21	0.9767	0.9545	1.0000

[indam23]

Looks like it's a matter of ~4 min vs ~5 min; the only reason I'd say not to do this is if it prevents windows users from using the starterpack as is (tensorflow-text requirement)

[amn41]

I think when we switch to tf2.3 that won't be a problem anymore? but not sure

[akelad]

yeah let's wait until this is compatible for windows users as well

[indam23]

closing since Convert is no longer a publicly available model