Hyperparams for results reported in the paper

[pclucas14]

Hello,

is it possible to have hyperparameters values that reproduce the NVRNN and LM results on the PTB dataset ?

Many thanks,
Lucas

[jiacheng-xu]

Hi Lucas,
thanks for your interest. I am sorry that I didn’t put all the commands on the github page because there are too many tables ( and it’s hard to keep track of all the configuration of all the results).
Here are some of my intuition about training PTB: 1) large learning rate with decay (--lr 10 for example) 2) train longer with sgd (--epochs 100) 3) gradient clip and dropout. This repo provides an amazing configuration of training a LM. PyTorch example of word language model.

I listed the instance name of saved results. Possibly you can reproduce the results given the hyper-parameters.
Detailed Configuration:
Note: zero=lm; nor=gaussian; vmf=von mises-fisher
Standard Setting:

• Dataptb_Distvmf_Modelnvrnn_EnclstmBiFalse_Emb100_Hid400_lat50_lr10.0_drop0.5_kappa35.0_auxw0.0001_normfFalse_nlay1_mixunk0.0_inpzTrue_cdbit0_cdbow0
• Dataptb_Distvmf_Modelnvrnn_EnclstmBiFalse_Emb100_Hid400_lat50_lr10.0_drop0.5_kappa5.0_auxw0.0001_normfFalse_nlay1_mixunk0.0_inpzTrue_cdbit0_cdbow200

Inputless Setting
Condition on Bag-of-words:

• Dataptb_Distzero_Modelnvrnn_EnclstmBiFalse_Emb100_Hid400_lat100_lr10.0_drop0.5_kappa0.1_auxw0.0001_normfFalse_nlay1_mixunk1.0_inpzTrue_cdbit0_cdbow200
• Dataptb_Distnor_Modelnvrnn_EnclstmBiFalse_Emb100_Hid400_lat100_lr10.0_drop0.5_kappa0.1_auxw0.0001_normfFalse_nlay1_mixunk1.0_inpzTrue_cdbit0_cdbow200
• Dataptb_Distvmf_Modelnvrnn_EnclstmBiFalse_Emb100_Hid400_lat100_lr10.0_drop0.5_kappa50.0_auxw0.0001_normfFalse_nlay1_mixunk1.0_inpzTrue_cdbit0_cdbow200

Not Condition:

• Dataptb_Distzero_Modelnvrnn_EnclstmBiFalse_Emb100_Hid400_lat100_lr10.0_drop0.5_kappa0.1_auxw0.0001_normfFalse_nlay1_mixunk1.0_inpzTrue_cdbit0_cdbow0
• Dataptb_Distnor_Modelnvrnn_EnclstmBiFalse_Emb100_Hid400_lat100_lr10.0_drop0.5_kappa0.1_auxw0.0001_normfFalse_nlay1_mixunk1.0_inpzTrue_cdbit0_cdbow0
• Dataptb_Distvmf_Modelnvrnn_EnclstmBiFalse_Emb100_Hid400_lat50_lr10.0_drop0.5_kappa80.0_auxw0.0001_normfFalse_nlay1_mixunk1.0_inpzTrue_cdbit0_cdbow0

[pclucas14]

Hi,

thanks for the fast response, and for your insight on PTB! is it also possible to get LM paramaters in the standard setting ?

[jiacheng-xu]

Hi,

thanks for the fast response, and for your insight on PTB! is it also possible to get LM paramaters in the standard setting ?

The configuration of the word language model example will help.

[pclucas14]

great! thanks again