-
Notifications
You must be signed in to change notification settings - Fork 10
/
Copy pathtransformer-training-logs.txt
6564 lines (6535 loc) · 708 KB
/
transformer-training-logs.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
WARNING: Logging before flag parsing goes to stderr.
W0804 19:23:44.963335 140200711067520 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/expert_utils.py:68: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.
W0804 19:23:47.298877 140200711067520 lazy_loader.py:50]
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
* https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
* https://github.com/tensorflow/addons
* https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.
W0804 19:23:48.913978 140200711067520 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/adafactor.py:27: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.
W0804 19:23:48.914525 140200711067520 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/multistep_optimizer.py:32: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.
W0804 19:23:48.933969 140200711067520 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/mesh_tensorflow/ops.py:4237: The name tf.train.CheckpointSaverListener is deprecated. Please use tf.estimator.CheckpointSaverListener instead.
W0804 19:23:48.934196 140200711067520 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/mesh_tensorflow/ops.py:4260: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.
W0804 19:23:48.985285 140200711067520 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/rl/gym_utils.py:219: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.
W0804 19:23:49.096743 140200711067520 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/trainer_lib.py:109: The name tf.OptimizerOptions is deprecated. Please use tf.compat.v1.OptimizerOptions instead.
W0804 19:23:49.880575 140200711067520 deprecation_wrapper.py:119] From /usr/local/bin/t2t-trainer:32: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.
W0804 19:23:49.880771 140200711067520 deprecation_wrapper.py:119] From /usr/local/bin/t2t-trainer:32: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.
W0804 19:23:49.880886 140200711067520 deprecation_wrapper.py:119] From /usr/local/bin/t2t-trainer:33: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.
I0804 19:23:49.881355 140200711067520 usr_dir.py:43] Importing user module t2t_paper_generation_problem from path /content/fake-academic-paper-generation
W0804 19:23:49.883288 140200711067520 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/hparams_lib.py:49: The name tf.gfile.Exists is deprecated. Please use tf.io.gfile.exists instead.
W0804 19:23:49.883560 140200711067520 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/trainer_lib.py:780: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.
W0804 19:23:49.890107 140200711067520 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/bin/t2t_trainer.py:282: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.
I0804 19:23:49.890470 140200711067520 t2t_trainer.py:286] Generating data for paper_generation_problem
W0804 19:23:49.892291 140200711067520 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/data_generators/generator_utils.py:164: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.
I0804 19:23:49.905993 140200711067520 generator_utils.py:232] Downloading https://github.com/lipanpanpanpan/fake-academic-paper-generation/raw/master/dataset/preprocessed_data.txt to experiment/transformer/transformer_small/tmp/paper_dataset.txt
W0804 19:23:49.906136 140200711067520 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/data_generators/generator_utils.py:234: The name tf.gfile.Copy is deprecated. Please use tf.io.gfile.copy instead.
100% completed
W0804 19:23:51.389821 140200711067520 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/data_generators/generator_utils.py:242: The name tf.gfile.Rename is deprecated. Please use tf.io.gfile.rename instead.
I0804 19:23:51.390216 140200711067520 generator_utils.py:247] Successfully downloaded paper_dataset.txt, 38382903 bytes.
I0804 19:23:51.463192 140200711067520 generator_utils.py:170] Generating case 0.
I0804 19:23:59.097003 140200711067520 generator_utils.py:170] Generating case 100000.
I0804 19:24:06.568216 140200711067520 generator_utils.py:170] Generating case 200000.
I0804 19:24:14.001129 140200711067520 generator_utils.py:193] Generated 299861 Examples
I0804 19:24:14.003109 140200711067520 generator_utils.py:527] Shuffling data...
W0804 19:24:14.003282 140200711067520 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/data_generators/generator_utils.py:469: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and:
`tf.data.TFRecordDataset(path)`
W0804 19:24:14.027454 140200711067520 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/data_generators/generator_utils.py:513: The name tf.gfile.Remove is deprecated. Please use tf.io.gfile.remove instead.
I0804 19:24:16.187735 140200711067520 generator_utils.py:530] Data shuffled.
W0804 19:24:16.190629 140200711067520 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/trainer_lib.py:121: The name tf.GraphOptions is deprecated. Please use tf.compat.v1.GraphOptions instead.
W0804 19:24:16.190868 140200711067520 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/trainer_lib.py:127: The name tf.GPUOptions is deprecated. Please use tf.compat.v1.GPUOptions instead.
W0804 19:24:16.191073 140200711067520 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/trainer_lib.py:240: RunConfig.__init__ (from tensorflow.contrib.learn.python.learn.estimators.run_config) is deprecated and will be removed in a future version.
Instructions for updating:
When switching to tf.estimator.Estimator, use tf.estimator.RunConfig instead.
I0804 19:24:16.191280 140200711067520 trainer_lib.py:263] Configuring DataParallelism to replicate the model.
I0804 19:24:16.191368 140200711067520 devices.py:76] schedule=continuous_train_and_eval
I0804 19:24:16.191456 140200711067520 devices.py:77] worker_gpu=1
I0804 19:24:16.191523 140200711067520 devices.py:78] sync=False
W0804 19:24:16.191632 140200711067520 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/devices.py:139: The name tf.logging.warn is deprecated. Please use tf.compat.v1.logging.warn instead.
W0804 19:24:16.191702 140200711067520 devices.py:141] Schedule=continuous_train_and_eval. Assuming that training is running on a single machine.
I0804 19:24:16.192488 140200711067520 devices.py:170] datashard_devices: ['gpu:0']
I0804 19:24:16.192576 140200711067520 devices.py:171] caching_devices: None
I0804 19:24:16.193089 140200711067520 devices.py:172] ps_devices: ['gpu:0']
I0804 19:24:16.193797 140200711067520 estimator.py:209] Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f82b6e3b470>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_train_distribute': None, '_eval_distribute': None, '_experimental_max_worker_delay_secs': None, '_device_fn': None, '_tf_config': gpu_options {
per_process_gpu_memory_fraction: 1.0
}
, '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_secs': None, '_log_step_count_steps': 100, '_protocol': None, '_session_config': gpu_options {
per_process_gpu_memory_fraction: 0.95
}
allow_soft_placement: true
graph_options {
optimizer_options {
global_jit_level: OFF
}
}
isolate_session_state: true
, '_save_checkpoints_steps': 1000, '_keep_checkpoint_max': 20, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': 'experiment/transformer/transformer_small/output', 'use_tpu': False, 't2t_device_info': {'num_async_replicas': 1}, 'data_parallelism': <tensor2tensor.utils.expert_utils.Parallelism object at 0x7f82b6e3b4e0>}
W0804 19:24:16.194043 140200711067520 model_fn.py:630] Estimator's model_fn (<function T2TModel.make_estimator_model_fn.<locals>.wrapping_model_fn at 0x7f82b6e3f7b8>) includes params argument, but params are not passed to Estimator.
W0804 19:24:16.194649 140200711067520 trainer_lib.py:724] ValidationMonitor only works with --schedule=train_and_evaluate
I0804 19:24:16.197000 140200711067520 estimator_training.py:186] Not using Distribute Coordinator.
I0804 19:24:16.197229 140200711067520 training.py:612] Running training and evaluation locally (non-distributed).
I0804 19:24:16.197569 140200711067520 training.py:700] Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 1000 or save_checkpoints_secs None.
W0804 19:24:16.206409 140200711067520 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
I0804 19:24:16.217053 140200711067520 problem.py:644] Reading data files from experiment/transformer/transformer_small/data/paper_generation_problem-train*
I0804 19:24:16.223310 140200711067520 problem.py:670] partition: 0 num_data_files: 100
W0804 19:24:16.225496 140200711067520 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/data_generators/problem.py:680: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_determinstic`.
W0804 19:24:16.431521 140200711067520 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/data_reader.py:37: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
W0804 19:24:16.471996 140200711067520 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/experimental/ops/grouping.py:193: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0804 19:24:16.536600 140200711067520 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/data_reader.py:231: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.
W0804 19:24:16.547811 140200711067520 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/data_reader.py:233: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
I0804 19:24:16.582735 140200711067520 estimator.py:1145] Calling model_fn.
I0804 19:24:16.595087 140200711067520 t2t_model.py:2172] Setting T2TModel mode to 'train'
W0804 19:24:16.675199 140200711067520 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/t2t_model.py:243: The name tf.summary.text is deprecated. Please use tf.compat.v1.summary.text instead.
I0804 19:24:17.482010 140200711067520 api.py:255] Using variable initializer: uniform_unit_scaling
I0804 19:24:17.951250 140200711067520 t2t_model.py:2172] Transforming feature 'targets' with symbol_modality_258_256.targets_bottom
I0804 19:24:18.102306 140200711067520 t2t_model.py:2172] Building model body
W0804 19:24:18.335082 140200711067520 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/models/transformer.py:156: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
W0804 19:24:18.372541 140200711067520 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/layers/common_layers.py:3106: The name tf.layers.Dense is deprecated. Please use tf.compat.v1.layers.Dense instead.
W0804 19:24:18.797479 140200711067520 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/layers/common_attention.py:1217: The name tf.summary.image is deprecated. Please use tf.compat.v1.summary.image instead.
I0804 19:24:19.331498 140200711067520 t2t_model.py:2172] Transforming body output with symbol_modality_258_256.top
W0804 19:24:19.461583 140200711067520 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/learning_rate.py:107: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.
I0804 19:24:19.462944 140200711067520 learning_rate.py:29] Base learning rate: 2.000000
I0804 19:24:19.473967 140200711067520 optimize.py:327] Trainable Variables Total size: 1644032
I0804 19:24:19.474202 140200711067520 optimize.py:327] Non-trainable variables Total size: 5
I0804 19:24:19.474358 140200711067520 optimize.py:182] Using optimizer adam
I0804 19:24:21.590090 140200711067520 estimator.py:1147] Done calling model_fn.
I0804 19:24:21.591542 140200711067520 basic_session_run_hooks.py:541] Create CheckpointSaverHook.
I0804 19:24:22.326212 140200711067520 monitored_session.py:240] Graph was finalized.
2019-08-04 19:24:22.340769: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2300000000 Hz
2019-08-04 19:24:22.342739: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x9512540 executing computations on platform Host. Devices:
2019-08-04 19:24:22.342773: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): <undefined>, <undefined>
2019-08-04 19:24:22.348671: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-08-04 19:24:22.573397: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-04 19:24:22.574025: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x95121c0 executing computations on platform CUDA. Devices:
2019-08-04 19:24:22.574064: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): Tesla T4, Compute Capability 7.5
2019-08-04 19:24:22.574401: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-04 19:24:22.575001: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:00:04.0
2019-08-04 19:24:22.588746: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-08-04 19:24:22.758388: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-08-04 19:24:22.838309: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2019-08-04 19:24:22.863166: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2019-08-04 19:24:23.062855: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2019-08-04 19:24:23.170580: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2019-08-04 19:24:23.517111: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-08-04 19:24:23.517378: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-04 19:24:23.517903: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-04 19:24:23.518245: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-08-04 19:24:23.520902: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-08-04 19:24:23.522597: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-08-04 19:24:23.522625: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2019-08-04 19:24:23.522643: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2019-08-04 19:24:23.525086: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-04 19:24:23.525589: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-04 19:24:23.525942: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:40] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2019-08-04 19:24:23.525985: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14325 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)
2019-08-04 19:24:23.873368: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0804 19:24:25.580031 140200711067520 session_manager.py:500] Running local_init_op.
I0804 19:24:25.614005 140200711067520 session_manager.py:502] Done running local_init_op.
I0804 19:24:27.651640 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 0 into experiment/transformer/transformer_small/output/model.ckpt.
2019-08-04 19:24:29.339574: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
I0804 19:24:32.483394 140200711067520 basic_session_run_hooks.py:262] loss = 8.155501, step = 0
I0804 19:24:36.518196 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 24.7793
I0804 19:24:36.519674 140200711067520 basic_session_run_hooks.py:260] loss = 5.2857046, step = 100 (4.036 sec)
I0804 19:24:39.498749 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.551
I0804 19:24:39.500139 140200711067520 basic_session_run_hooks.py:260] loss = 3.4506884, step = 200 (2.980 sec)
I0804 19:24:42.455612 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.8195
I0804 19:24:42.457046 140200711067520 basic_session_run_hooks.py:260] loss = 3.0842905, step = 300 (2.957 sec)
I0804 19:24:45.409548 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.8535
I0804 19:24:45.411277 140200711067520 basic_session_run_hooks.py:260] loss = 3.058373, step = 400 (2.954 sec)
I0804 19:24:48.360075 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.8921
I0804 19:24:48.361675 140200711067520 basic_session_run_hooks.py:260] loss = 2.8275805, step = 500 (2.950 sec)
I0804 19:24:51.305898 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.9463
I0804 19:24:51.307310 140200711067520 basic_session_run_hooks.py:260] loss = 2.579091, step = 600 (2.946 sec)
I0804 19:24:54.257405 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.8808
I0804 19:24:54.258950 140200711067520 basic_session_run_hooks.py:260] loss = 2.4020169, step = 700 (2.952 sec)
I0804 19:24:57.242395 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.501
I0804 19:24:57.243805 140200711067520 basic_session_run_hooks.py:260] loss = 2.4527538, step = 800 (2.985 sec)
I0804 19:25:00.233128 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.4369
I0804 19:25:00.234609 140200711067520 basic_session_run_hooks.py:260] loss = 2.389892, step = 900 (2.991 sec)
I0804 19:25:03.196714 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 1000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:25:03.523173 140200711067520 problem.py:644] Reading data files from experiment/transformer/transformer_small/data/paper_generation_problem-dev*
I0804 19:25:03.524591 140200711067520 problem.py:670] partition: 0 num_data_files: 1
I0804 19:25:03.679222 140200711067520 estimator.py:1145] Calling model_fn.
I0804 19:25:03.680652 140200711067520 t2t_model.py:2172] Setting T2TModel mode to 'eval'
I0804 19:25:03.681203 140200711067520 t2t_model.py:2172] Setting hparams.dropout to 0.0
I0804 19:25:03.681339 140200711067520 t2t_model.py:2172] Setting hparams.label_smoothing to 0.0
I0804 19:25:03.681461 140200711067520 t2t_model.py:2172] Setting hparams.layer_prepostprocess_dropout to 0.0
I0804 19:25:03.681557 140200711067520 t2t_model.py:2172] Setting hparams.symbol_dropout to 0.0
I0804 19:25:03.681661 140200711067520 t2t_model.py:2172] Setting hparams.attention_dropout to 0.0
I0804 19:25:03.681751 140200711067520 t2t_model.py:2172] Setting hparams.relu_dropout to 0.0
I0804 19:25:03.776337 140200711067520 api.py:255] Using variable initializer: uniform_unit_scaling
I0804 19:25:03.840514 140200711067520 t2t_model.py:2172] Transforming feature 'targets' with symbol_modality_258_256.targets_bottom
I0804 19:25:03.988373 140200711067520 t2t_model.py:2172] Building model body
I0804 19:25:04.936924 140200711067520 t2t_model.py:2172] Transforming body output with symbol_modality_258_256.top
W0804 19:25:05.102966 140200711067520 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/metrics.py:582: The name tf.metrics.mean is deprecated. Please use tf.compat.v1.metrics.mean instead.
W0804 19:25:05.505138 140200711067520 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/bleu_hook.py:151: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
options available in V2.
- tf.py_function takes a python function which manipulates tf eager
tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
an ndarray (just call tensor.numpy()) but having access to eager tensors
means `tf.py_function`s can use accelerators such as GPUs as well as
being differentiable using a gradient tape.
- tf.numpy_function maintains the semantics of the deprecated tf.py_func
(it is not differentiable, and manipulates numpy arrays). It drops the
stateful argument making all functions stateful.
W0804 19:25:05.688900 140200711067520 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/t2t_model.py:1670: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.
I0804 19:25:05.690661 140200711067520 estimator.py:1147] Done calling model_fn.
I0804 19:25:05.713266 140200711067520 evaluation.py:255] Starting evaluation at 2019-08-04T19:25:05Z
I0804 19:25:06.189876 140200711067520 monitored_session.py:240] Graph was finalized.
2019-08-04 19:25:06.190700: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-04 19:25:06.191100: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:00:04.0
2019-08-04 19:25:06.191209: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-08-04 19:25:06.191234: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-08-04 19:25:06.191259: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2019-08-04 19:25:06.191281: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2019-08-04 19:25:06.191306: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2019-08-04 19:25:06.191328: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2019-08-04 19:25:06.191352: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-08-04 19:25:06.191497: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-04 19:25:06.191925: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-04 19:25:06.192297: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-08-04 19:25:06.192339: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-08-04 19:25:06.192354: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2019-08-04 19:25:06.192365: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2019-08-04 19:25:06.192703: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-04 19:25:06.193130: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-04 19:25:06.193514: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14325 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)
W0804 19:25:06.193651 140200711067520 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
I0804 19:25:06.194862 140200711067520 saver.py:1280] Restoring parameters from experiment/transformer/transformer_small/output/model.ckpt-1000
I0804 19:25:06.383098 140200711067520 session_manager.py:500] Running local_init_op.
I0804 19:25:06.426431 140200711067520 session_manager.py:502] Done running local_init_op.
I0804 19:25:12.864027 140200711067520 evaluation.py:167] Evaluation [10/100]
I0804 19:25:18.602217 140200711067520 evaluation.py:167] Evaluation [20/100]
I0804 19:25:24.269555 140200711067520 evaluation.py:167] Evaluation [30/100]
I0804 19:25:30.000144 140200711067520 evaluation.py:167] Evaluation [40/100]
I0804 19:25:35.615264 140200711067520 evaluation.py:167] Evaluation [50/100]
I0804 19:25:41.510267 140200711067520 evaluation.py:167] Evaluation [60/100]
I0804 19:25:47.200354 140200711067520 evaluation.py:167] Evaluation [70/100]
I0804 19:25:52.884746 140200711067520 evaluation.py:167] Evaluation [80/100]
I0804 19:25:58.551951 140200711067520 evaluation.py:167] Evaluation [90/100]
I0804 19:26:03.668064 140200711067520 evaluation.py:275] Finished evaluation at 2019-08-04-19:26:03
I0804 19:26:03.668334 140200711067520 estimator.py:2039] Saving dict for global step 1000: global_step = 1000, loss = 2.7128453, metrics-paper_generation_problem/targets/accuracy = 0.26113552, metrics-paper_generation_problem/targets/accuracy_per_sequence = 0.0, metrics-paper_generation_problem/targets/accuracy_top5 = 0.6263957, metrics-paper_generation_problem/targets/approx_bleu_score = 0.12949093, metrics-paper_generation_problem/targets/neg_log_perplexity = -2.712865, metrics-paper_generation_problem/targets/rouge_2_fscore = 0.25634146, metrics-paper_generation_problem/targets/rouge_L_fscore = 0.3960722
I0804 19:26:03.668929 140200711067520 estimator.py:2099] Saving 'checkpoint_path' summary for global step 1000: experiment/transformer/transformer_small/output/model.ckpt-1000
I0804 19:26:03.720886 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 1.57511
I0804 19:26:03.722041 140200711067520 basic_session_run_hooks.py:260] loss = 2.409999, step = 1000 (63.487 sec)
I0804 19:26:06.732197 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.2086
I0804 19:26:06.733334 140200711067520 basic_session_run_hooks.py:260] loss = 2.379183, step = 1100 (3.011 sec)
I0804 19:26:09.694392 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.7588
I0804 19:26:09.695862 140200711067520 basic_session_run_hooks.py:260] loss = 2.2706354, step = 1200 (2.963 sec)
I0804 19:26:12.664976 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.6638
I0804 19:26:12.666488 140200711067520 basic_session_run_hooks.py:260] loss = 2.1597714, step = 1300 (2.971 sec)
I0804 19:26:15.605208 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 34.0104
I0804 19:26:15.606709 140200711067520 basic_session_run_hooks.py:260] loss = 2.1520514, step = 1400 (2.940 sec)
I0804 19:26:18.582639 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.5867
I0804 19:26:18.584065 140200711067520 basic_session_run_hooks.py:260] loss = 2.2127125, step = 1500 (2.977 sec)
I0804 19:26:21.566531 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.5128
I0804 19:26:21.567976 140200711067520 basic_session_run_hooks.py:260] loss = 2.0944536, step = 1600 (2.984 sec)
I0804 19:26:24.546823 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.5536
I0804 19:26:24.548582 140200711067520 basic_session_run_hooks.py:260] loss = 2.0609598, step = 1700 (2.981 sec)
I0804 19:26:27.561280 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.1735
I0804 19:26:27.562687 140200711067520 basic_session_run_hooks.py:260] loss = 1.9879534, step = 1800 (3.014 sec)
I0804 19:26:30.551955 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.4397
I0804 19:26:30.554009 140200711067520 basic_session_run_hooks.py:260] loss = 1.8525437, step = 1900 (2.991 sec)
I0804 19:26:33.533509 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 2000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:26:33.856554 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:26:33.890398 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 29.952
I0804 19:26:33.891387 140200711067520 basic_session_run_hooks.py:260] loss = 1.8864993, step = 2000 (3.337 sec)
I0804 19:26:36.900977 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.2162
I0804 19:26:36.902472 140200711067520 basic_session_run_hooks.py:260] loss = 1.8693491, step = 2100 (3.011 sec)
I0804 19:26:39.917082 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.1554
I0804 19:26:39.918612 140200711067520 basic_session_run_hooks.py:260] loss = 1.7641883, step = 2200 (3.016 sec)
I0804 19:26:42.932776 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.16
I0804 19:26:42.933911 140200711067520 basic_session_run_hooks.py:260] loss = 1.7817109, step = 2300 (3.015 sec)
I0804 19:26:45.979623 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.8208
I0804 19:26:45.980860 140200711067520 basic_session_run_hooks.py:260] loss = 1.6913946, step = 2400 (3.047 sec)
I0804 19:26:49.032759 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.7533
I0804 19:26:49.034281 140200711067520 basic_session_run_hooks.py:260] loss = 1.7186183, step = 2500 (3.053 sec)
I0804 19:26:52.098445 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.619
I0804 19:26:52.099894 140200711067520 basic_session_run_hooks.py:260] loss = 1.6929767, step = 2600 (3.066 sec)
I0804 19:26:55.139570 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.8828
I0804 19:26:55.141005 140200711067520 basic_session_run_hooks.py:260] loss = 1.5994068, step = 2700 (3.041 sec)
I0804 19:26:58.194659 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.7319
I0804 19:26:58.196120 140200711067520 basic_session_run_hooks.py:260] loss = 1.6039113, step = 2800 (3.055 sec)
I0804 19:27:01.242408 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.8113
I0804 19:27:01.243752 140200711067520 basic_session_run_hooks.py:260] loss = 1.6295178, step = 2900 (3.048 sec)
I0804 19:27:04.237113 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 3000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:27:04.550706 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:27:04.586644 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 29.9019
I0804 19:27:04.587828 140200711067520 basic_session_run_hooks.py:260] loss = 1.6565264, step = 3000 (3.344 sec)
I0804 19:27:07.640507 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.746
I0804 19:27:07.641699 140200711067520 basic_session_run_hooks.py:260] loss = 1.5527774, step = 3100 (3.054 sec)
I0804 19:27:10.707409 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6059
I0804 19:27:10.708856 140200711067520 basic_session_run_hooks.py:260] loss = 1.5831912, step = 3200 (3.067 sec)
I0804 19:27:13.774051 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6089
I0804 19:27:13.775332 140200711067520 basic_session_run_hooks.py:260] loss = 1.6109092, step = 3300 (3.066 sec)
I0804 19:27:16.842576 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.5891
I0804 19:27:16.843778 140200711067520 basic_session_run_hooks.py:260] loss = 1.4197326, step = 3400 (3.068 sec)
I0804 19:27:19.892964 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.7826
I0804 19:27:19.894052 140200711067520 basic_session_run_hooks.py:260] loss = 1.5294964, step = 3500 (3.050 sec)
I0804 19:27:22.921457 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.0201
I0804 19:27:22.923023 140200711067520 basic_session_run_hooks.py:260] loss = 1.5571033, step = 3600 (3.029 sec)
I0804 19:27:25.939731 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.1311
I0804 19:27:25.941386 140200711067520 basic_session_run_hooks.py:260] loss = 1.4533595, step = 3700 (3.018 sec)
I0804 19:27:28.964576 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.0598
I0804 19:27:28.965959 140200711067520 basic_session_run_hooks.py:260] loss = 1.5444443, step = 3800 (3.025 sec)
I0804 19:27:31.960324 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.3803
I0804 19:27:31.961987 140200711067520 basic_session_run_hooks.py:260] loss = 1.518613, step = 3900 (2.996 sec)
I0804 19:27:34.944219 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 4000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:27:35.268598 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:27:35.304935 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 29.8987
I0804 19:27:35.306002 140200711067520 basic_session_run_hooks.py:260] loss = 1.4616487, step = 4000 (3.344 sec)
I0804 19:27:38.312713 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.2474
I0804 19:27:38.313874 140200711067520 basic_session_run_hooks.py:260] loss = 1.507455, step = 4100 (3.008 sec)
I0804 19:27:41.351313 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.9102
I0804 19:27:41.353080 140200711067520 basic_session_run_hooks.py:260] loss = 1.4154464, step = 4200 (3.039 sec)
I0804 19:27:44.383909 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.975
I0804 19:27:44.385295 140200711067520 basic_session_run_hooks.py:260] loss = 1.5419605, step = 4300 (3.032 sec)
I0804 19:27:47.412206 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.0216
I0804 19:27:47.413871 140200711067520 basic_session_run_hooks.py:260] loss = 1.4361955, step = 4400 (3.029 sec)
I0804 19:27:50.440459 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.0227
I0804 19:27:50.442209 140200711067520 basic_session_run_hooks.py:260] loss = 1.3980746, step = 4500 (3.028 sec)
I0804 19:27:53.486541 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.8291
I0804 19:27:53.488100 140200711067520 basic_session_run_hooks.py:260] loss = 1.4099467, step = 4600 (3.046 sec)
I0804 19:27:56.535737 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.7951
I0804 19:27:56.537252 140200711067520 basic_session_run_hooks.py:260] loss = 1.4482095, step = 4700 (3.049 sec)
I0804 19:27:59.606945 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.5603
I0804 19:27:59.608086 140200711067520 basic_session_run_hooks.py:260] loss = 1.5307233, step = 4800 (3.071 sec)
I0804 19:28:02.622036 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.1666
I0804 19:28:02.623761 140200711067520 basic_session_run_hooks.py:260] loss = 1.4048406, step = 4900 (3.016 sec)
I0804 19:28:05.611516 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 5000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:28:05.921041 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:28:05.960367 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 29.9551
I0804 19:28:05.961374 140200711067520 basic_session_run_hooks.py:260] loss = 1.4705069, step = 5000 (3.338 sec)
I0804 19:28:08.978475 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.1339
I0804 19:28:08.979931 140200711067520 basic_session_run_hooks.py:260] loss = 1.4336276, step = 5100 (3.019 sec)
I0804 19:28:12.036247 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.7031
I0804 19:28:12.037760 140200711067520 basic_session_run_hooks.py:260] loss = 1.4010735, step = 5200 (3.058 sec)
I0804 19:28:15.064410 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.0234
I0804 19:28:15.065931 140200711067520 basic_session_run_hooks.py:260] loss = 1.415313, step = 5300 (3.028 sec)
I0804 19:28:18.098437 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.9596
I0804 19:28:18.099949 140200711067520 basic_session_run_hooks.py:260] loss = 1.4335316, step = 5400 (3.034 sec)
I0804 19:28:21.160341 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6592
I0804 19:28:21.162182 140200711067520 basic_session_run_hooks.py:260] loss = 1.3694171, step = 5500 (3.062 sec)
I0804 19:28:24.180371 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.1123
I0804 19:28:24.181669 140200711067520 basic_session_run_hooks.py:260] loss = 1.2803149, step = 5600 (3.019 sec)
I0804 19:28:27.215081 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.9522
I0804 19:28:27.216775 140200711067520 basic_session_run_hooks.py:260] loss = 1.4415474, step = 5700 (3.035 sec)
I0804 19:28:30.263624 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.8027
I0804 19:28:30.264824 140200711067520 basic_session_run_hooks.py:260] loss = 1.352975, step = 5800 (3.048 sec)
I0804 19:28:33.287676 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.0682
I0804 19:28:33.289037 140200711067520 basic_session_run_hooks.py:260] loss = 1.3370267, step = 5900 (3.024 sec)
I0804 19:28:36.283484 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 6000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:28:36.577485 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:28:36.610884 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 30.091
I0804 19:28:36.611997 140200711067520 basic_session_run_hooks.py:260] loss = 1.3969012, step = 6000 (3.323 sec)
I0804 19:28:39.640194 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.0115
I0804 19:28:39.641671 140200711067520 basic_session_run_hooks.py:260] loss = 1.3341318, step = 6100 (3.030 sec)
I0804 19:28:42.691244 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.7753
I0804 19:28:42.692683 140200711067520 basic_session_run_hooks.py:260] loss = 1.4407762, step = 6200 (3.051 sec)
I0804 19:28:45.721908 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.9962
I0804 19:28:45.723112 140200711067520 basic_session_run_hooks.py:260] loss = 1.3321823, step = 6300 (3.030 sec)
I0804 19:28:48.742534 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.1058
I0804 19:28:48.743758 140200711067520 basic_session_run_hooks.py:260] loss = 1.5399805, step = 6400 (3.021 sec)
I0804 19:28:51.820149 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.4923
I0804 19:28:51.821729 140200711067520 basic_session_run_hooks.py:260] loss = 1.3282262, step = 6500 (3.078 sec)
I0804 19:28:54.839002 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.1257
I0804 19:28:54.840555 140200711067520 basic_session_run_hooks.py:260] loss = 1.2764181, step = 6600 (3.019 sec)
I0804 19:28:57.865550 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.0407
I0804 19:28:57.867305 140200711067520 basic_session_run_hooks.py:260] loss = 1.3656299, step = 6700 (3.027 sec)
I0804 19:29:00.895458 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.0043
I0804 19:29:00.896995 140200711067520 basic_session_run_hooks.py:260] loss = 1.3575528, step = 6800 (3.030 sec)
I0804 19:29:03.941338 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.8309
I0804 19:29:03.942813 140200711067520 basic_session_run_hooks.py:260] loss = 1.3987968, step = 6900 (3.046 sec)
I0804 19:29:06.971069 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 7000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:29:07.254149 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:29:07.295687 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 29.8119
I0804 19:29:07.296850 140200711067520 basic_session_run_hooks.py:260] loss = 1.4891579, step = 7000 (3.354 sec)
I0804 19:29:10.359506 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6394
I0804 19:29:10.361052 140200711067520 basic_session_run_hooks.py:260] loss = 1.2564094, step = 7100 (3.064 sec)
I0804 19:29:13.477219 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.0746
I0804 19:29:13.478415 140200711067520 basic_session_run_hooks.py:260] loss = 1.3091213, step = 7200 (3.117 sec)
I0804 19:29:16.549064 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.5538
I0804 19:29:16.550181 140200711067520 basic_session_run_hooks.py:260] loss = 1.3494519, step = 7300 (3.072 sec)
I0804 19:29:19.606631 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.7057
I0804 19:29:19.607827 140200711067520 basic_session_run_hooks.py:260] loss = 1.1798173, step = 7400 (3.058 sec)
I0804 19:29:22.651954 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.8372
I0804 19:29:22.653059 140200711067520 basic_session_run_hooks.py:260] loss = 1.1869992, step = 7500 (3.045 sec)
I0804 19:29:25.736833 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.4166
I0804 19:29:25.738704 140200711067520 basic_session_run_hooks.py:260] loss = 1.3844441, step = 7600 (3.086 sec)
I0804 19:29:28.803191 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6115
I0804 19:29:28.804862 140200711067520 basic_session_run_hooks.py:260] loss = 1.3332542, step = 7700 (3.066 sec)
I0804 19:29:31.877873 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.5237
I0804 19:29:31.879033 140200711067520 basic_session_run_hooks.py:260] loss = 1.294771, step = 7800 (3.074 sec)
I0804 19:29:34.926853 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.7979
I0804 19:29:34.928403 140200711067520 basic_session_run_hooks.py:260] loss = 1.2601916, step = 7900 (3.049 sec)
I0804 19:29:37.965266 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 8000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:29:38.244294 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:29:38.277575 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 29.8443
I0804 19:29:38.278681 140200711067520 basic_session_run_hooks.py:260] loss = 1.3717446, step = 8000 (3.350 sec)
I0804 19:29:41.355464 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.4899
I0804 19:29:41.356830 140200711067520 basic_session_run_hooks.py:260] loss = 1.3131135, step = 8100 (3.078 sec)
I0804 19:29:44.388844 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.9664
I0804 19:29:44.390315 140200711067520 basic_session_run_hooks.py:260] loss = 1.2771983, step = 8200 (3.033 sec)
I0804 19:29:47.432667 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.8535
I0804 19:29:47.434068 140200711067520 basic_session_run_hooks.py:260] loss = 1.3089756, step = 8300 (3.044 sec)
I0804 19:29:50.496766 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6359
I0804 19:29:50.498517 140200711067520 basic_session_run_hooks.py:260] loss = 1.3441107, step = 8400 (3.064 sec)
I0804 19:29:53.582629 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.4058
I0804 19:29:53.584056 140200711067520 basic_session_run_hooks.py:260] loss = 1.2427619, step = 8500 (3.086 sec)
I0804 19:29:56.699201 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.0872
I0804 19:29:56.700674 140200711067520 basic_session_run_hooks.py:260] loss = 1.2500312, step = 8600 (3.117 sec)
I0804 19:29:59.801394 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.2347
I0804 19:29:59.802819 140200711067520 basic_session_run_hooks.py:260] loss = 1.2316062, step = 8700 (3.102 sec)
I0804 19:30:02.929850 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 31.9647
I0804 19:30:02.931274 140200711067520 basic_session_run_hooks.py:260] loss = 1.1864159, step = 8800 (3.128 sec)
I0804 19:30:06.015702 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.406
I0804 19:30:06.017162 140200711067520 basic_session_run_hooks.py:260] loss = 1.3271396, step = 8900 (3.086 sec)
I0804 19:30:09.002612 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 9000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:30:09.280321 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:30:09.314185 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 30.3167
I0804 19:30:09.315178 140200711067520 basic_session_run_hooks.py:260] loss = 1.2230386, step = 9000 (3.298 sec)
I0804 19:30:12.411708 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.2843
I0804 19:30:12.413269 140200711067520 basic_session_run_hooks.py:260] loss = 1.360263, step = 9100 (3.098 sec)
I0804 19:30:15.491156 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.4733
I0804 19:30:15.492901 140200711067520 basic_session_run_hooks.py:260] loss = 1.3862758, step = 9200 (3.080 sec)
I0804 19:30:18.592680 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.2422
I0804 19:30:18.593923 140200711067520 basic_session_run_hooks.py:260] loss = 1.2016011, step = 9300 (3.101 sec)
I0804 19:30:21.622977 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.0001
I0804 19:30:21.624153 140200711067520 basic_session_run_hooks.py:260] loss = 1.2682439, step = 9400 (3.030 sec)
I0804 19:30:24.661991 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.9054
I0804 19:30:24.663237 140200711067520 basic_session_run_hooks.py:260] loss = 1.3375763, step = 9500 (3.039 sec)
I0804 19:30:27.707390 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.8367
I0804 19:30:27.709131 140200711067520 basic_session_run_hooks.py:260] loss = 1.2132281, step = 9600 (3.046 sec)
I0804 19:30:30.771943 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.631
I0804 19:30:30.773294 140200711067520 basic_session_run_hooks.py:260] loss = 1.338763, step = 9700 (3.064 sec)
I0804 19:30:33.862114 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.3608
I0804 19:30:33.863699 140200711067520 basic_session_run_hooks.py:260] loss = 1.2979809, step = 9800 (3.090 sec)
I0804 19:30:36.965536 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.2227
I0804 19:30:36.966934 140200711067520 basic_session_run_hooks.py:260] loss = 1.2382991, step = 9900 (3.103 sec)
I0804 19:30:40.036624 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 10000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:30:40.334321 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:30:40.372535 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 29.351
I0804 19:30:40.373526 140200711067520 basic_session_run_hooks.py:260] loss = 1.1605426, step = 10000 (3.407 sec)
I0804 19:30:43.472177 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.262
I0804 19:30:43.473607 140200711067520 basic_session_run_hooks.py:260] loss = 1.180176, step = 10100 (3.100 sec)
I0804 19:30:46.511767 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.8992
I0804 19:30:46.513055 140200711067520 basic_session_run_hooks.py:260] loss = 1.3104894, step = 10200 (3.039 sec)
I0804 19:30:49.558764 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.8192
I0804 19:30:49.560128 140200711067520 basic_session_run_hooks.py:260] loss = 1.3073193, step = 10300 (3.047 sec)
I0804 19:30:52.609038 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.7838
I0804 19:30:52.610279 140200711067520 basic_session_run_hooks.py:260] loss = 1.2635038, step = 10400 (3.050 sec)
I0804 19:30:55.678632 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.5775
I0804 19:30:55.679919 140200711067520 basic_session_run_hooks.py:260] loss = 1.3505212, step = 10500 (3.070 sec)
I0804 19:30:58.745019 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6118
I0804 19:30:58.746312 140200711067520 basic_session_run_hooks.py:260] loss = 1.2462518, step = 10600 (3.066 sec)
I0804 19:31:01.830933 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.4053
I0804 19:31:01.832185 140200711067520 basic_session_run_hooks.py:260] loss = 1.2149142, step = 10700 (3.086 sec)
I0804 19:31:04.909625 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.4814
I0804 19:31:04.910925 140200711067520 basic_session_run_hooks.py:260] loss = 1.2501847, step = 10800 (3.079 sec)
I0804 19:31:07.989155 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.4725
I0804 19:31:07.990710 140200711067520 basic_session_run_hooks.py:260] loss = 1.2028272, step = 10900 (3.080 sec)
I0804 19:31:10.990052 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 11000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:31:11.271090 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:31:11.305317 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 30.155
I0804 19:31:11.306512 140200711067520 basic_session_run_hooks.py:260] loss = 1.2102648, step = 11000 (3.316 sec)
I0804 19:31:14.342254 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.9283
I0804 19:31:14.343698 140200711067520 basic_session_run_hooks.py:260] loss = 1.2601262, step = 11100 (3.037 sec)
I0804 19:31:17.368982 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.039
I0804 19:31:17.370527 140200711067520 basic_session_run_hooks.py:260] loss = 1.3274779, step = 11200 (3.027 sec)
I0804 19:31:20.392978 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.0688
I0804 19:31:20.394392 140200711067520 basic_session_run_hooks.py:260] loss = 1.1980853, step = 11300 (3.024 sec)
I0804 19:31:23.397733 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.2804
I0804 19:31:23.399140 140200711067520 basic_session_run_hooks.py:260] loss = 1.4138035, step = 11400 (3.005 sec)
I0804 19:31:26.441767 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.8514
I0804 19:31:26.443464 140200711067520 basic_session_run_hooks.py:260] loss = 1.1718758, step = 11500 (3.044 sec)
I0804 19:31:29.444451 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.3036
I0804 19:31:29.445904 140200711067520 basic_session_run_hooks.py:260] loss = 1.227898, step = 11600 (3.002 sec)
I0804 19:31:32.510579 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6145
I0804 19:31:32.511914 140200711067520 basic_session_run_hooks.py:260] loss = 1.2353508, step = 11700 (3.066 sec)
I0804 19:31:35.530181 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.117
I0804 19:31:35.531721 140200711067520 basic_session_run_hooks.py:260] loss = 1.2210552, step = 11800 (3.020 sec)
I0804 19:31:38.590711 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6735
I0804 19:31:38.591968 140200711067520 basic_session_run_hooks.py:260] loss = 1.2374473, step = 11900 (3.060 sec)
I0804 19:31:41.607785 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 12000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:31:41.888094 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:31:41.929655 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 29.9495
I0804 19:31:41.931216 140200711067520 basic_session_run_hooks.py:260] loss = 1.2551806, step = 12000 (3.339 sec)
I0804 19:31:44.980729 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.7756
I0804 19:31:44.981899 140200711067520 basic_session_run_hooks.py:260] loss = 1.159721, step = 12100 (3.051 sec)
I0804 19:31:48.044164 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.643
I0804 19:31:48.045541 140200711067520 basic_session_run_hooks.py:260] loss = 1.2198614, step = 12200 (3.064 sec)
I0804 19:31:51.111698 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.5994
I0804 19:31:51.113060 140200711067520 basic_session_run_hooks.py:260] loss = 1.2674105, step = 12300 (3.068 sec)
I0804 19:31:54.192655 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.4575
I0804 19:31:54.194410 140200711067520 basic_session_run_hooks.py:260] loss = 1.2431961, step = 12400 (3.081 sec)
I0804 19:31:57.306456 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.1153
I0804 19:31:57.308202 140200711067520 basic_session_run_hooks.py:260] loss = 1.1959051, step = 12500 (3.114 sec)
I0804 19:32:00.363126 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.7152
I0804 19:32:00.364523 140200711067520 basic_session_run_hooks.py:260] loss = 1.1595746, step = 12600 (3.056 sec)
I0804 19:32:03.410493 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.8154
I0804 19:32:03.411881 140200711067520 basic_session_run_hooks.py:260] loss = 1.2487422, step = 12700 (3.047 sec)
I0804 19:32:06.477142 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6086
I0804 19:32:06.478483 140200711067520 basic_session_run_hooks.py:260] loss = 1.2127666, step = 12800 (3.067 sec)
I0804 19:32:09.558259 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.4558
I0804 19:32:09.559898 140200711067520 basic_session_run_hooks.py:260] loss = 1.2425289, step = 12900 (3.081 sec)
I0804 19:32:12.591459 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 13000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:32:12.870286 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:32:12.910022 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 29.8349
I0804 19:32:12.911083 140200711067520 basic_session_run_hooks.py:260] loss = 1.2182281, step = 13000 (3.351 sec)
I0804 19:32:16.010016 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.2583
I0804 19:32:16.011257 140200711067520 basic_session_run_hooks.py:260] loss = 1.1973971, step = 13100 (3.100 sec)
I0804 19:32:19.125058 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.1023
I0804 19:32:19.126234 140200711067520 basic_session_run_hooks.py:260] loss = 1.1959229, step = 13200 (3.115 sec)
I0804 19:32:22.168682 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.8555
I0804 19:32:22.169960 140200711067520 basic_session_run_hooks.py:260] loss = 1.2638351, step = 13300 (3.044 sec)
I0804 19:32:25.177658 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.234
I0804 19:32:25.179447 140200711067520 basic_session_run_hooks.py:260] loss = 1.2654928, step = 13400 (3.009 sec)
I0804 19:32:28.211886 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.9573
I0804 19:32:28.213636 140200711067520 basic_session_run_hooks.py:260] loss = 1.1349357, step = 13500 (3.034 sec)
I0804 19:32:31.248620 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.9301
I0804 19:32:31.249807 140200711067520 basic_session_run_hooks.py:260] loss = 1.1719645, step = 13600 (3.036 sec)
I0804 19:32:34.282307 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.9634
I0804 19:32:34.283779 140200711067520 basic_session_run_hooks.py:260] loss = 1.2101492, step = 13700 (3.034 sec)
I0804 19:32:37.334640 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.7616
I0804 19:32:37.335915 140200711067520 basic_session_run_hooks.py:260] loss = 1.1115232, step = 13800 (3.052 sec)
I0804 19:32:40.407629 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.5416
I0804 19:32:40.408902 140200711067520 basic_session_run_hooks.py:260] loss = 1.1840647, step = 13900 (3.073 sec)
I0804 19:32:43.444034 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 14000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:32:43.723935 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:32:43.763708 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 29.7966
I0804 19:32:43.764799 140200711067520 basic_session_run_hooks.py:260] loss = 1.1738458, step = 14000 (3.356 sec)
I0804 19:32:46.862951 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.2662
I0804 19:32:46.864213 140200711067520 basic_session_run_hooks.py:260] loss = 1.2190872, step = 14100 (3.099 sec)
I0804 19:32:49.952946 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.3625
I0804 19:32:49.954496 140200711067520 basic_session_run_hooks.py:260] loss = 1.1474192, step = 14200 (3.090 sec)
I0804 19:32:53.020766 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.5962
I0804 19:32:53.022028 140200711067520 basic_session_run_hooks.py:260] loss = 1.2996737, step = 14300 (3.068 sec)
I0804 19:32:56.078085 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.7084
I0804 19:32:56.079457 140200711067520 basic_session_run_hooks.py:260] loss = 1.1575891, step = 14400 (3.057 sec)
I0804 19:32:59.134395 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.7193
I0804 19:32:59.135835 140200711067520 basic_session_run_hooks.py:260] loss = 1.1813397, step = 14500 (3.056 sec)
I0804 19:33:02.225768 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.348
I0804 19:33:02.227131 140200711067520 basic_session_run_hooks.py:260] loss = 1.1509129, step = 14600 (3.091 sec)
I0804 19:33:05.300163 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.5267
I0804 19:33:05.301403 140200711067520 basic_session_run_hooks.py:260] loss = 1.1458942, step = 14700 (3.074 sec)
I0804 19:33:08.390953 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.3543
I0804 19:33:08.392508 140200711067520 basic_session_run_hooks.py:260] loss = 1.2474288, step = 14800 (3.091 sec)
I0804 19:33:11.492274 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.2445
I0804 19:33:11.493993 140200711067520 basic_session_run_hooks.py:260] loss = 1.0441868, step = 14900 (3.101 sec)
I0804 19:33:14.511921 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 15000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:33:14.803041 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:33:14.842698 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 29.8466
I0804 19:33:14.843883 140200711067520 basic_session_run_hooks.py:260] loss = 1.1936878, step = 15000 (3.350 sec)
I0804 19:33:17.916839 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.5296
I0804 19:33:17.918240 140200711067520 basic_session_run_hooks.py:260] loss = 1.2710189, step = 15100 (3.074 sec)
I0804 19:33:20.985048 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.5924
I0804 19:33:20.986593 140200711067520 basic_session_run_hooks.py:260] loss = 1.2386189, step = 15200 (3.068 sec)
I0804 19:33:24.052024 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6054
I0804 19:33:24.053371 140200711067520 basic_session_run_hooks.py:260] loss = 1.1741124, step = 15300 (3.067 sec)
I0804 19:33:27.132543 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.4623
I0804 19:33:27.133707 140200711067520 basic_session_run_hooks.py:260] loss = 1.2481456, step = 15400 (3.080 sec)
I0804 19:33:30.199351 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6069
I0804 19:33:30.200798 140200711067520 basic_session_run_hooks.py:260] loss = 1.0897235, step = 15500 (3.067 sec)
I0804 19:33:33.301364 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.2373
I0804 19:33:33.302937 140200711067520 basic_session_run_hooks.py:260] loss = 1.1586254, step = 15600 (3.102 sec)
I0804 19:33:36.291994 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.4381
I0804 19:33:36.293476 140200711067520 basic_session_run_hooks.py:260] loss = 1.1332475, step = 15700 (2.991 sec)
I0804 19:33:39.290340 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.3513
I0804 19:33:39.291769 140200711067520 basic_session_run_hooks.py:260] loss = 1.1740195, step = 15800 (2.998 sec)
I0804 19:33:42.327117 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.9296
I0804 19:33:42.328668 140200711067520 basic_session_run_hooks.py:260] loss = 1.2191907, step = 15900 (3.037 sec)
I0804 19:33:45.334685 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 16000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:33:45.609776 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:33:45.646860 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 30.1225
I0804 19:33:45.647949 140200711067520 basic_session_run_hooks.py:260] loss = 1.0809509, step = 16000 (3.319 sec)
I0804 19:33:48.711970 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6256
I0804 19:33:48.713227 140200711067520 basic_session_run_hooks.py:260] loss = 1.1982739, step = 16100 (3.065 sec)
I0804 19:33:51.773390 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6646
I0804 19:33:51.774976 140200711067520 basic_session_run_hooks.py:260] loss = 1.2547654, step = 16200 (3.062 sec)
I0804 19:33:54.828641 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.7305
I0804 19:33:54.829794 140200711067520 basic_session_run_hooks.py:260] loss = 1.1052699, step = 16300 (3.055 sec)
I0804 19:33:57.903104 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.526
I0804 19:33:57.904279 140200711067520 basic_session_run_hooks.py:260] loss = 1.1795783, step = 16400 (3.074 sec)
I0804 19:34:00.935705 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.975
I0804 19:34:00.937056 140200711067520 basic_session_run_hooks.py:260] loss = 1.2374704, step = 16500 (3.033 sec)
I0804 19:34:03.978750 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.8621
I0804 19:34:03.980371 140200711067520 basic_session_run_hooks.py:260] loss = 1.2353446, step = 16600 (3.043 sec)
I0804 19:34:07.039579 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6709
I0804 19:34:07.041054 140200711067520 basic_session_run_hooks.py:260] loss = 1.1759075, step = 16700 (3.061 sec)
I0804 19:34:10.104204 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6301
I0804 19:34:10.105698 140200711067520 basic_session_run_hooks.py:260] loss = 1.1354991, step = 16800 (3.065 sec)
I0804 19:34:13.183482 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.4753
I0804 19:34:13.185091 140200711067520 basic_session_run_hooks.py:260] loss = 1.2012011, step = 16900 (3.079 sec)
I0804 19:34:16.225773 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 17000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:34:16.499884 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:34:16.540835 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 29.7851
I0804 19:34:16.541842 140200711067520 basic_session_run_hooks.py:260] loss = 1.1836874, step = 17000 (3.357 sec)
I0804 19:34:19.645821 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.2066
I0804 19:34:19.647203 140200711067520 basic_session_run_hooks.py:260] loss = 1.1589793, step = 17100 (3.105 sec)
I0804 19:34:22.735124 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.3697
I0804 19:34:22.736683 140200711067520 basic_session_run_hooks.py:260] loss = 1.1926513, step = 17200 (3.089 sec)
I0804 19:34:25.773269 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.9149
I0804 19:34:25.774763 140200711067520 basic_session_run_hooks.py:260] loss = 1.1059518, step = 17300 (3.038 sec)
I0804 19:34:28.807503 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.9573
I0804 19:34:28.809046 140200711067520 basic_session_run_hooks.py:260] loss = 1.2031003, step = 17400 (3.034 sec)
I0804 19:34:31.853115 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.8339
I0804 19:34:31.854511 140200711067520 basic_session_run_hooks.py:260] loss = 1.1429232, step = 17500 (3.045 sec)
I0804 19:34:34.910079 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.7122
I0804 19:34:34.911245 140200711067520 basic_session_run_hooks.py:260] loss = 1.212168, step = 17600 (3.057 sec)
I0804 19:34:37.976313 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6134
I0804 19:34:37.977970 140200711067520 basic_session_run_hooks.py:260] loss = 1.14036, step = 17700 (3.067 sec)
I0804 19:34:41.053347 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.4988
I0804 19:34:41.054639 140200711067520 basic_session_run_hooks.py:260] loss = 1.207542, step = 17800 (3.077 sec)
I0804 19:34:44.123361 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.573
I0804 19:34:44.125077 140200711067520 basic_session_run_hooks.py:260] loss = 1.2870129, step = 17900 (3.070 sec)
I0804 19:34:47.147558 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 18000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:34:47.425818 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:34:47.465058 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 29.9248
I0804 19:34:47.466052 140200711067520 basic_session_run_hooks.py:260] loss = 1.1035069, step = 18000 (3.341 sec)
I0804 19:34:50.530331 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6239
I0804 19:34:50.531792 140200711067520 basic_session_run_hooks.py:260] loss = 1.1633078, step = 18100 (3.066 sec)
I0804 19:34:53.584231 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.7448
I0804 19:34:53.585653 140200711067520 basic_session_run_hooks.py:260] loss = 1.1383585, step = 18200 (3.054 sec)
I0804 19:34:56.624778 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.889
I0804 19:34:56.626069 140200711067520 basic_session_run_hooks.py:260] loss = 1.200794, step = 18300 (3.040 sec)
I0804 19:34:59.670257 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.8355
I0804 19:34:59.671993 140200711067520 basic_session_run_hooks.py:260] loss = 1.2173938, step = 18400 (3.046 sec)
I0804 19:35:02.712268 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.873
I0804 19:35:02.714134 140200711067520 basic_session_run_hooks.py:260] loss = 1.1468053, step = 18500 (3.042 sec)
I0804 19:35:05.741981 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.0063
I0804 19:35:05.743324 140200711067520 basic_session_run_hooks.py:260] loss = 1.277636, step = 18600 (3.029 sec)
I0804 19:35:08.766356 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.0646
I0804 19:35:08.767856 140200711067520 basic_session_run_hooks.py:260] loss = 1.1514399, step = 18700 (3.025 sec)
I0804 19:35:11.811201 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.8426
I0804 19:35:11.812715 140200711067520 basic_session_run_hooks.py:260] loss = 1.1231668, step = 18800 (3.045 sec)
I0804 19:35:14.879994 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.5861
I0804 19:35:14.881193 140200711067520 basic_session_run_hooks.py:260] loss = 1.1923566, step = 18900 (3.068 sec)
I0804 19:35:17.894748 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 19000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:35:18.178933 140200711067520 problem.py:644] Reading data files from experiment/transformer/transformer_small/data/paper_generation_problem-dev*
I0804 19:35:18.180367 140200711067520 problem.py:670] partition: 0 num_data_files: 1
I0804 19:35:18.331510 140200711067520 estimator.py:1145] Calling model_fn.
I0804 19:35:18.332573 140200711067520 t2t_model.py:2172] Setting T2TModel mode to 'eval'
I0804 19:35:18.332980 140200711067520 t2t_model.py:2172] Setting hparams.dropout to 0.0
I0804 19:35:18.333073 140200711067520 t2t_model.py:2172] Setting hparams.label_smoothing to 0.0
I0804 19:35:18.333155 140200711067520 t2t_model.py:2172] Setting hparams.layer_prepostprocess_dropout to 0.0
I0804 19:35:18.333221 140200711067520 t2t_model.py:2172] Setting hparams.symbol_dropout to 0.0
I0804 19:35:18.333304 140200711067520 t2t_model.py:2172] Setting hparams.attention_dropout to 0.0
I0804 19:35:18.333377 140200711067520 t2t_model.py:2172] Setting hparams.relu_dropout to 0.0
I0804 19:35:18.421065 140200711067520 api.py:255] Using variable initializer: uniform_unit_scaling
I0804 19:35:18.481137 140200711067520 t2t_model.py:2172] Transforming feature 'targets' with symbol_modality_258_256.targets_bottom
I0804 19:35:18.619872 140200711067520 t2t_model.py:2172] Building model body
I0804 19:35:19.307488 140200711067520 t2t_model.py:2172] Transforming body output with symbol_modality_258_256.top
I0804 19:35:20.314753 140200711067520 estimator.py:1147] Done calling model_fn.
I0804 19:35:20.333621 140200711067520 evaluation.py:255] Starting evaluation at 2019-08-04T19:35:20Z
I0804 19:35:20.501849 140200711067520 monitored_session.py:240] Graph was finalized.
2019-08-04 19:35:20.502474: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-04 19:35:20.502866: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:00:04.0
2019-08-04 19:35:20.502975: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-08-04 19:35:20.503000: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-08-04 19:35:20.503025: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2019-08-04 19:35:20.503047: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2019-08-04 19:35:20.503068: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2019-08-04 19:35:20.503091: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2019-08-04 19:35:20.503115: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-08-04 19:35:20.503228: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-04 19:35:20.503694: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-04 19:35:20.504008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-08-04 19:35:20.504050: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-08-04 19:35:20.504063: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2019-08-04 19:35:20.504074: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2019-08-04 19:35:20.504353: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-04 19:35:20.504758: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-04 19:35:20.505083: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14325 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)
I0804 19:35:20.506659 140200711067520 saver.py:1280] Restoring parameters from experiment/transformer/transformer_small/output/model.ckpt-19000
I0804 19:35:20.711789 140200711067520 session_manager.py:500] Running local_init_op.
I0804 19:35:20.759563 140200711067520 session_manager.py:502] Done running local_init_op.
I0804 19:35:26.769286 140200711067520 evaluation.py:167] Evaluation [10/100]
I0804 19:35:32.076843 140200711067520 evaluation.py:167] Evaluation [20/100]
I0804 19:35:37.385871 140200711067520 evaluation.py:167] Evaluation [30/100]
I0804 19:35:42.723874 140200711067520 evaluation.py:167] Evaluation [40/100]
I0804 19:35:48.036636 140200711067520 evaluation.py:167] Evaluation [50/100]
I0804 19:35:53.423947 140200711067520 evaluation.py:167] Evaluation [60/100]
I0804 19:35:58.816497 140200711067520 evaluation.py:167] Evaluation [70/100]
I0804 19:36:04.211121 140200711067520 evaluation.py:167] Evaluation [80/100]
I0804 19:36:09.521650 140200711067520 evaluation.py:167] Evaluation [90/100]
I0804 19:36:14.369156 140200711067520 evaluation.py:275] Finished evaluation at 2019-08-04-19:36:14
I0804 19:36:14.369442 140200711067520 estimator.py:2039] Saving dict for global step 19000: global_step = 19000, loss = 1.2819614, metrics-paper_generation_problem/targets/accuracy = 0.6460767, metrics-paper_generation_problem/targets/accuracy_per_sequence = 0.0, metrics-paper_generation_problem/targets/accuracy_top5 = 0.8694033, metrics-paper_generation_problem/targets/approx_bleu_score = 0.4516523, metrics-paper_generation_problem/targets/neg_log_perplexity = -1.2820032, metrics-paper_generation_problem/targets/rouge_2_fscore = 0.5532905, metrics-paper_generation_problem/targets/rouge_L_fscore = 0.67241085
I0804 19:36:14.370100 140200711067520 estimator.py:2099] Saving 'checkpoint_path' summary for global step 19000: experiment/transformer/transformer_small/output/model.ckpt-19000
I0804 19:36:14.424113 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 1.67943
I0804 19:36:14.425105 140200711067520 basic_session_run_hooks.py:260] loss = 1.0774114, step = 19000 (59.544 sec)
I0804 19:36:17.545263 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.04
I0804 19:36:17.546634 140200711067520 basic_session_run_hooks.py:260] loss = 1.2865647, step = 19100 (3.122 sec)
I0804 19:36:20.643223 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.2794
I0804 19:36:20.644990 140200711067520 basic_session_run_hooks.py:260] loss = 1.177287, step = 19200 (3.098 sec)
I0804 19:36:23.667823 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 33.0622
I0804 19:36:23.669291 140200711067520 basic_session_run_hooks.py:260] loss = 1.2077243, step = 19300 (3.024 sec)
I0804 19:36:26.740880 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.5409
I0804 19:36:26.742123 140200711067520 basic_session_run_hooks.py:260] loss = 1.1630546, step = 19400 (3.073 sec)
I0804 19:36:29.852913 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.1333
I0804 19:36:29.854687 140200711067520 basic_session_run_hooks.py:260] loss = 1.1744058, step = 19500 (3.113 sec)
I0804 19:36:32.968632 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.0952
I0804 19:36:32.970390 140200711067520 basic_session_run_hooks.py:260] loss = 1.1371635, step = 19600 (3.116 sec)
I0804 19:36:36.103107 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 31.9034
I0804 19:36:36.104585 140200711067520 basic_session_run_hooks.py:260] loss = 1.1755279, step = 19700 (3.134 sec)
I0804 19:36:39.256193 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 31.7149
I0804 19:36:39.257328 140200711067520 basic_session_run_hooks.py:260] loss = 1.1414742, step = 19800 (3.153 sec)
I0804 19:36:42.421108 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 31.5964
I0804 19:36:42.422265 140200711067520 basic_session_run_hooks.py:260] loss = 1.16572, step = 19900 (3.165 sec)
I0804 19:36:45.534163 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 20000 into experiment/transformer/transformer_small/output/model.ckpt.
W0804 19:36:45.561821 140200711067520 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:960: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to delete files with this prefix.
I0804 19:36:45.824675 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:36:45.857781 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 29.0977
I0804 19:36:45.858798 140200711067520 basic_session_run_hooks.py:260] loss = 1.1964328, step = 20000 (3.437 sec)
I0804 19:36:48.946603 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.3753
I0804 19:36:48.947993 140200711067520 basic_session_run_hooks.py:260] loss = 1.2031322, step = 20100 (3.089 sec)
I0804 19:36:52.029267 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.4391
I0804 19:36:52.030838 140200711067520 basic_session_run_hooks.py:260] loss = 1.1104207, step = 20200 (3.083 sec)
I0804 19:36:55.129518 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.2557
I0804 19:36:55.131067 140200711067520 basic_session_run_hooks.py:260] loss = 1.1560361, step = 20300 (3.100 sec)
I0804 19:36:58.233558 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.2161
I0804 19:36:58.235221 140200711067520 basic_session_run_hooks.py:260] loss = 1.2182827, step = 20400 (3.104 sec)
I0804 19:37:01.309588 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.5094
I0804 19:37:01.310895 140200711067520 basic_session_run_hooks.py:260] loss = 1.1767564, step = 20500 (3.076 sec)
I0804 19:37:04.398446 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.3744
I0804 19:37:04.399903 140200711067520 basic_session_run_hooks.py:260] loss = 1.1665094, step = 20600 (3.089 sec)
I0804 19:37:07.504444 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.1957
I0804 19:37:07.506119 140200711067520 basic_session_run_hooks.py:260] loss = 1.0595956, step = 20700 (3.106 sec)
I0804 19:37:10.581554 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.4981
I0804 19:37:10.583051 140200711067520 basic_session_run_hooks.py:260] loss = 1.1754962, step = 20800 (3.077 sec)
I0804 19:37:13.612342 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.9945
I0804 19:37:13.613761 140200711067520 basic_session_run_hooks.py:260] loss = 1.142103, step = 20900 (3.031 sec)
I0804 19:37:16.630002 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 21000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:37:16.931900 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:37:16.965882 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 29.819
I0804 19:37:16.966959 140200711067520 basic_session_run_hooks.py:260] loss = 1.1589005, step = 21000 (3.353 sec)
I0804 19:37:20.068833 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.2276
I0804 19:37:20.070288 140200711067520 basic_session_run_hooks.py:260] loss = 1.0709862, step = 21100 (3.103 sec)
I0804 19:37:23.135121 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6128
I0804 19:37:23.136287 140200711067520 basic_session_run_hooks.py:260] loss = 1.1429499, step = 21200 (3.066 sec)
I0804 19:37:26.225683 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.3565
I0804 19:37:26.227053 140200711067520 basic_session_run_hooks.py:260] loss = 1.214007, step = 21300 (3.091 sec)
I0804 19:37:29.324099 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.2746
I0804 19:37:29.325542 140200711067520 basic_session_run_hooks.py:260] loss = 1.1842749, step = 21400 (3.098 sec)
I0804 19:37:32.401624 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.4936
I0804 19:37:32.403098 140200711067520 basic_session_run_hooks.py:260] loss = 1.1561172, step = 21500 (3.078 sec)
I0804 19:37:35.475774 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.5293
I0804 19:37:35.477524 140200711067520 basic_session_run_hooks.py:260] loss = 1.2352748, step = 21600 (3.074 sec)
I0804 19:37:38.517554 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.8757
I0804 19:37:38.518861 140200711067520 basic_session_run_hooks.py:260] loss = 1.18102, step = 21700 (3.041 sec)
I0804 19:37:41.585565 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.5947
I0804 19:37:41.586823 140200711067520 basic_session_run_hooks.py:260] loss = 1.1425862, step = 21800 (3.068 sec)
I0804 19:37:44.673245 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.3862
I0804 19:37:44.674494 140200711067520 basic_session_run_hooks.py:260] loss = 1.2044507, step = 21900 (3.088 sec)
I0804 19:37:47.703058 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 22000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:37:47.998501 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:37:48.041297 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 29.6907
I0804 19:37:48.042533 140200711067520 basic_session_run_hooks.py:260] loss = 1.1608398, step = 22000 (3.368 sec)
I0804 19:37:51.162552 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.0387
I0804 19:37:51.164051 140200711067520 basic_session_run_hooks.py:260] loss = 1.1650263, step = 22100 (3.122 sec)
I0804 19:37:54.292370 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 31.9506
I0804 19:37:54.293650 140200711067520 basic_session_run_hooks.py:260] loss = 1.1885363, step = 22200 (3.130 sec)
I0804 19:37:57.417355 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.0004
I0804 19:37:57.418799 140200711067520 basic_session_run_hooks.py:260] loss = 1.185166, step = 22300 (3.125 sec)
I0804 19:38:00.482517 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6247
I0804 19:38:00.484014 140200711067520 basic_session_run_hooks.py:260] loss = 1.1100837, step = 22400 (3.065 sec)
I0804 19:38:03.524507 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.8732
I0804 19:38:03.525793 140200711067520 basic_session_run_hooks.py:260] loss = 1.2033346, step = 22500 (3.042 sec)
I0804 19:38:06.566694 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.8708
I0804 19:38:06.568071 140200711067520 basic_session_run_hooks.py:260] loss = 1.142504, step = 22600 (3.042 sec)
I0804 19:38:09.629569 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6496
I0804 19:38:09.631026 140200711067520 basic_session_run_hooks.py:260] loss = 1.1963096, step = 22700 (3.063 sec)
I0804 19:38:12.732398 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.2283
I0804 19:38:12.733744 140200711067520 basic_session_run_hooks.py:260] loss = 1.2043911, step = 22800 (3.103 sec)
I0804 19:38:15.832314 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.259
I0804 19:38:15.833613 140200711067520 basic_session_run_hooks.py:260] loss = 1.2078202, step = 22900 (3.100 sec)
I0804 19:38:18.877968 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 23000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:38:19.182590 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:38:19.225291 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 29.4723
I0804 19:38:19.226340 140200711067520 basic_session_run_hooks.py:260] loss = 1.0467908, step = 23000 (3.393 sec)
I0804 19:38:22.314800 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.3679
I0804 19:38:22.316230 140200711067520 basic_session_run_hooks.py:260] loss = 1.1481651, step = 23100 (3.090 sec)
I0804 19:38:25.392672 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.49
I0804 19:38:25.394031 140200711067520 basic_session_run_hooks.py:260] loss = 1.2228857, step = 23200 (3.078 sec)
I0804 19:38:28.470298 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.4927
I0804 19:38:28.471825 140200711067520 basic_session_run_hooks.py:260] loss = 1.1864159, step = 23300 (3.078 sec)
I0804 19:38:31.560724 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.3578
I0804 19:38:31.562395 140200711067520 basic_session_run_hooks.py:260] loss = 1.1314461, step = 23400 (3.091 sec)
I0804 19:38:34.628766 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.5941
I0804 19:38:34.630047 140200711067520 basic_session_run_hooks.py:260] loss = 1.1422839, step = 23500 (3.068 sec)
I0804 19:38:37.723471 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.3135
I0804 19:38:37.724975 140200711067520 basic_session_run_hooks.py:260] loss = 1.0633789, step = 23600 (3.095 sec)
I0804 19:38:40.854739 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 31.9358
I0804 19:38:40.855888 140200711067520 basic_session_run_hooks.py:260] loss = 1.1601166, step = 23700 (3.131 sec)
I0804 19:38:43.948783 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.3203
I0804 19:38:43.950376 140200711067520 basic_session_run_hooks.py:260] loss = 1.1460757, step = 23800 (3.094 sec)
I0804 19:38:47.045925 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.2877
I0804 19:38:47.047343 140200711067520 basic_session_run_hooks.py:260] loss = 1.0903977, step = 23900 (3.097 sec)
I0804 19:38:50.107812 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 24000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:38:50.392560 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:38:50.429239 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 29.5566
I0804 19:38:50.430280 140200711067520 basic_session_run_hooks.py:260] loss = 1.1522293, step = 24000 (3.383 sec)
I0804 19:38:53.500026 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.5653
I0804 19:38:53.501578 140200711067520 basic_session_run_hooks.py:260] loss = 1.0931782, step = 24100 (3.071 sec)
I0804 19:38:56.591674 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.3451
I0804 19:38:56.593028 140200711067520 basic_session_run_hooks.py:260] loss = 1.1558337, step = 24200 (3.091 sec)
I0804 19:38:59.696580 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.2073
I0804 19:38:59.698130 140200711067520 basic_session_run_hooks.py:260] loss = 1.1288643, step = 24300 (3.105 sec)
I0804 19:39:02.803099 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.1903
I0804 19:39:02.804646 140200711067520 basic_session_run_hooks.py:260] loss = 1.2165997, step = 24400 (3.107 sec)
I0804 19:39:05.914798 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.1367
I0804 19:39:05.916656 140200711067520 basic_session_run_hooks.py:260] loss = 1.1749858, step = 24500 (3.112 sec)
I0804 19:39:09.025446 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.1478
I0804 19:39:09.026866 140200711067520 basic_session_run_hooks.py:260] loss = 1.2127006, step = 24600 (3.110 sec)
I0804 19:39:12.174772 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 31.7528
I0804 19:39:12.176238 140200711067520 basic_session_run_hooks.py:260] loss = 1.086857, step = 24700 (3.149 sec)
I0804 19:39:15.254091 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.4747
I0804 19:39:15.255303 140200711067520 basic_session_run_hooks.py:260] loss = 1.1870553, step = 24800 (3.079 sec)
I0804 19:39:18.315655 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6631
I0804 19:39:18.317138 140200711067520 basic_session_run_hooks.py:260] loss = 1.2003901, step = 24900 (3.062 sec)
I0804 19:39:21.360117 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 25000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:39:21.663019 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:39:21.697321 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 29.5709
I0804 19:39:21.698580 140200711067520 basic_session_run_hooks.py:260] loss = 1.290784, step = 25000 (3.381 sec)
I0804 19:39:24.784930 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.3877
I0804 19:39:24.786276 140200711067520 basic_session_run_hooks.py:260] loss = 1.1820152, step = 25100 (3.088 sec)
I0804 19:39:27.891233 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.1927
I0804 19:39:27.892799 140200711067520 basic_session_run_hooks.py:260] loss = 1.0839361, step = 25200 (3.107 sec)
I0804 19:39:30.992761 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.2421
I0804 19:39:30.994261 140200711067520 basic_session_run_hooks.py:260] loss = 1.2427635, step = 25300 (3.101 sec)
I0804 19:39:34.075691 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.4366
I0804 19:39:34.077037 140200711067520 basic_session_run_hooks.py:260] loss = 1.2150134, step = 25400 (3.083 sec)
I0804 19:39:37.195538 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.0532
I0804 19:39:37.197069 140200711067520 basic_session_run_hooks.py:260] loss = 1.150935, step = 25500 (3.120 sec)
I0804 19:39:40.262788 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6024
I0804 19:39:40.264772 140200711067520 basic_session_run_hooks.py:260] loss = 1.1871003, step = 25600 (3.068 sec)
I0804 19:39:43.327203 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6325
I0804 19:39:43.328678 140200711067520 basic_session_run_hooks.py:260] loss = 1.1857219, step = 25700 (3.064 sec)
I0804 19:39:46.411812 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.419
I0804 19:39:46.413258 140200711067520 basic_session_run_hooks.py:260] loss = 1.0898191, step = 25800 (3.085 sec)
I0804 19:39:49.502558 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.3549
I0804 19:39:49.504561 140200711067520 basic_session_run_hooks.py:260] loss = 1.2438653, step = 25900 (3.091 sec)
I0804 19:39:52.550063 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 26000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:39:52.839120 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:39:52.879206 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 29.6148
I0804 19:39:52.880406 140200711067520 basic_session_run_hooks.py:260] loss = 1.1750416, step = 26000 (3.376 sec)
I0804 19:39:55.994803 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.097
I0804 19:39:55.996100 140200711067520 basic_session_run_hooks.py:260] loss = 1.1264752, step = 26100 (3.116 sec)
I0804 19:39:59.091130 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.2962
I0804 19:39:59.092327 140200711067520 basic_session_run_hooks.py:260] loss = 1.122388, step = 26200 (3.096 sec)
I0804 19:40:02.208163 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.0819
I0804 19:40:02.209790 140200711067520 basic_session_run_hooks.py:260] loss = 1.2204032, step = 26300 (3.117 sec)
I0804 19:40:05.263779 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.7265
I0804 19:40:05.265557 140200711067520 basic_session_run_hooks.py:260] loss = 1.2088444, step = 26400 (3.056 sec)
I0804 19:40:08.357367 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.3249
I0804 19:40:08.358909 140200711067520 basic_session_run_hooks.py:260] loss = 1.1157737, step = 26500 (3.093 sec)
I0804 19:40:11.458297 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.2484
I0804 19:40:11.459793 140200711067520 basic_session_run_hooks.py:260] loss = 1.1500334, step = 26600 (3.101 sec)
I0804 19:40:14.575665 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.0791
I0804 19:40:14.576990 140200711067520 basic_session_run_hooks.py:260] loss = 1.1873708, step = 26700 (3.117 sec)
I0804 19:40:17.696856 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.0386
I0804 19:40:17.697938 140200711067520 basic_session_run_hooks.py:260] loss = 1.1324512, step = 26800 (3.121 sec)
I0804 19:40:20.828170 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 31.9375
I0804 19:40:20.829654 140200711067520 basic_session_run_hooks.py:260] loss = 1.0946645, step = 26900 (3.132 sec)
I0804 19:40:23.903733 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 27000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:40:24.199718 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:40:24.240202 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 29.3061
I0804 19:40:24.241343 140200711067520 basic_session_run_hooks.py:260] loss = 1.1957022, step = 27000 (3.412 sec)
I0804 19:40:27.355068 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.1045
I0804 19:40:27.356534 140200711067520 basic_session_run_hooks.py:260] loss = 1.0480574, step = 27100 (3.115 sec)
I0804 19:40:30.416153 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6679
I0804 19:40:30.417474 140200711067520 basic_session_run_hooks.py:260] loss = 1.2231941, step = 27200 (3.061 sec)
I0804 19:40:33.458863 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.8656
I0804 19:40:33.460330 140200711067520 basic_session_run_hooks.py:260] loss = 1.1976117, step = 27300 (3.043 sec)
I0804 19:40:36.518341 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6852
I0804 19:40:36.520019 140200711067520 basic_session_run_hooks.py:260] loss = 1.0975667, step = 27400 (3.060 sec)
I0804 19:40:39.598225 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.4688
I0804 19:40:39.599837 140200711067520 basic_session_run_hooks.py:260] loss = 1.0454144, step = 27500 (3.080 sec)
I0804 19:40:42.659654 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6644
I0804 19:40:42.661021 140200711067520 basic_session_run_hooks.py:260] loss = 1.1412435, step = 27600 (3.061 sec)
I0804 19:40:45.733619 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.5314
I0804 19:40:45.735571 140200711067520 basic_session_run_hooks.py:260] loss = 1.1361846, step = 27700 (3.075 sec)
I0804 19:40:48.811731 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.4874
I0804 19:40:48.813244 140200711067520 basic_session_run_hooks.py:260] loss = 1.1252506, step = 27800 (3.078 sec)
I0804 19:40:51.903401 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.3449
I0804 19:40:51.905001 140200711067520 basic_session_run_hooks.py:260] loss = 1.2447684, step = 27900 (3.092 sec)
I0804 19:40:54.919029 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 28000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:40:55.202318 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:40:55.238695 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 29.9821
I0804 19:40:55.239888 140200711067520 basic_session_run_hooks.py:260] loss = 1.1910185, step = 28000 (3.335 sec)
I0804 19:40:58.289263 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.781
I0804 19:40:58.290777 140200711067520 basic_session_run_hooks.py:260] loss = 1.1956398, step = 28100 (3.051 sec)
I0804 19:41:01.331097 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.875
I0804 19:41:01.332472 140200711067520 basic_session_run_hooks.py:260] loss = 1.0986398, step = 28200 (3.042 sec)
I0804 19:41:04.409526 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.4843
I0804 19:41:04.410885 140200711067520 basic_session_run_hooks.py:260] loss = 1.1499906, step = 28300 (3.078 sec)
I0804 19:41:07.469053 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6846
I0804 19:41:07.470686 140200711067520 basic_session_run_hooks.py:260] loss = 1.1346446, step = 28400 (3.060 sec)
I0804 19:41:10.508963 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.8957
I0804 19:41:10.510234 140200711067520 basic_session_run_hooks.py:260] loss = 1.1012393, step = 28500 (3.040 sec)
I0804 19:41:13.556344 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.8152
I0804 19:41:13.557551 140200711067520 basic_session_run_hooks.py:260] loss = 1.1290898, step = 28600 (3.047 sec)
I0804 19:41:16.640304 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.426
I0804 19:41:16.641870 140200711067520 basic_session_run_hooks.py:260] loss = 1.0827153, step = 28700 (3.084 sec)
I0804 19:41:19.798603 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 31.6626
I0804 19:41:19.799752 140200711067520 basic_session_run_hooks.py:260] loss = 1.2034243, step = 28800 (3.158 sec)
I0804 19:41:22.937272 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 31.8606
I0804 19:41:22.938701 140200711067520 basic_session_run_hooks.py:260] loss = 1.1859727, step = 28900 (3.139 sec)
I0804 19:41:26.041780 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 29000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:41:26.325041 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:41:26.366203 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 29.1635
I0804 19:41:26.367583 140200711067520 basic_session_run_hooks.py:260] loss = 1.1626228, step = 29000 (3.429 sec)
I0804 19:41:29.550554 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 31.4037
I0804 19:41:29.551915 140200711067520 basic_session_run_hooks.py:260] loss = 1.1033005, step = 29100 (3.184 sec)
I0804 19:41:32.649246 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.2715
I0804 19:41:32.650592 140200711067520 basic_session_run_hooks.py:260] loss = 1.0244342, step = 29200 (3.099 sec)
I0804 19:41:35.724768 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.5149
I0804 19:41:35.726013 140200711067520 basic_session_run_hooks.py:260] loss = 1.1233537, step = 29300 (3.075 sec)
I0804 19:41:38.855039 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 31.9461
I0804 19:41:38.856492 140200711067520 basic_session_run_hooks.py:260] loss = 1.1355093, step = 29400 (3.130 sec)
I0804 19:41:41.988116 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 31.9176
I0804 19:41:41.989664 140200711067520 basic_session_run_hooks.py:260] loss = 1.1635525, step = 29500 (3.133 sec)
I0804 19:41:45.132560 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 31.8022
I0804 19:41:45.133991 140200711067520 basic_session_run_hooks.py:260] loss = 1.1484761, step = 29600 (3.144 sec)
I0804 19:41:48.272463 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 31.8482
I0804 19:41:48.273803 140200711067520 basic_session_run_hooks.py:260] loss = 1.1287494, step = 29700 (3.140 sec)
I0804 19:41:51.417184 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 31.799
I0804 19:41:51.418999 140200711067520 basic_session_run_hooks.py:260] loss = 1.2534288, step = 29800 (3.145 sec)
I0804 19:41:54.538943 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.0332
I0804 19:41:54.540071 140200711067520 basic_session_run_hooks.py:260] loss = 1.1392552, step = 29900 (3.121 sec)
I0804 19:41:57.571832 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 30000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:41:57.864303 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:41:57.904655 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 29.7113
I0804 19:41:57.906008 140200711067520 basic_session_run_hooks.py:260] loss = 1.2355024, step = 30000 (3.366 sec)
I0804 19:42:00.968649 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.6374
I0804 19:42:00.969892 140200711067520 basic_session_run_hooks.py:260] loss = 1.1847318, step = 30100 (3.064 sec)
I0804 19:42:04.054337 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.4077
I0804 19:42:04.055823 140200711067520 basic_session_run_hooks.py:260] loss = 1.2119321, step = 30200 (3.086 sec)
I0804 19:42:07.164514 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.1527
I0804 19:42:07.165837 140200711067520 basic_session_run_hooks.py:260] loss = 1.1538389, step = 30300 (3.110 sec)
I0804 19:42:10.285569 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.0404
I0804 19:42:10.286929 140200711067520 basic_session_run_hooks.py:260] loss = 1.1768385, step = 30400 (3.121 sec)
I0804 19:42:13.418894 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 31.9149
I0804 19:42:13.420933 140200711067520 basic_session_run_hooks.py:260] loss = 1.1893463, step = 30500 (3.134 sec)
I0804 19:42:16.553886 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 31.898
I0804 19:42:16.555094 140200711067520 basic_session_run_hooks.py:260] loss = 1.228784, step = 30600 (3.134 sec)
I0804 19:42:19.660362 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.1908
I0804 19:42:19.661844 140200711067520 basic_session_run_hooks.py:260] loss = 1.1657281, step = 30700 (3.107 sec)
I0804 19:42:22.716851 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.7174
I0804 19:42:22.718348 140200711067520 basic_session_run_hooks.py:260] loss = 1.0922602, step = 30800 (3.057 sec)
I0804 19:42:25.808853 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.3414
I0804 19:42:25.810248 140200711067520 basic_session_run_hooks.py:260] loss = 1.0505708, step = 30900 (3.092 sec)
I0804 19:42:28.861890 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 31000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:42:29.144351 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:42:29.179987 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 29.6633
I0804 19:42:29.181079 140200711067520 basic_session_run_hooks.py:260] loss = 1.1450275, step = 31000 (3.371 sec)
I0804 19:42:32.264278 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.4226
I0804 19:42:32.265774 140200711067520 basic_session_run_hooks.py:260] loss = 1.2153137, step = 31100 (3.085 sec)
I0804 19:42:35.357343 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.3305
I0804 19:42:35.358888 140200711067520 basic_session_run_hooks.py:260] loss = 1.0850574, step = 31200 (3.093 sec)
I0804 19:42:38.470263 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.124
I0804 19:42:38.471723 140200711067520 basic_session_run_hooks.py:260] loss = 1.1064955, step = 31300 (3.113 sec)
I0804 19:42:41.562155 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.3429
I0804 19:42:41.563751 140200711067520 basic_session_run_hooks.py:260] loss = 1.1050696, step = 31400 (3.092 sec)
I0804 19:42:44.660566 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.2746
I0804 19:42:44.661798 140200711067520 basic_session_run_hooks.py:260] loss = 1.1563405, step = 31500 (3.098 sec)
I0804 19:42:47.734446 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.5323
I0804 19:42:47.735753 140200711067520 basic_session_run_hooks.py:260] loss = 1.0993637, step = 31600 (3.074 sec)
I0804 19:42:50.858160 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.013
I0804 19:42:50.859621 140200711067520 basic_session_run_hooks.py:260] loss = 1.1257668, step = 31700 (3.124 sec)
I0804 19:42:53.947921 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.365
I0804 19:42:53.949494 140200711067520 basic_session_run_hooks.py:260] loss = 1.1356064, step = 31800 (3.090 sec)
I0804 19:42:57.064991 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 32.0814
I0804 19:42:57.066840 140200711067520 basic_session_run_hooks.py:260] loss = 1.2098117, step = 31900 (3.117 sec)
I0804 19:43:00.166999 140200711067520 basic_session_run_hooks.py:606] Saving checkpoints for 32000 into experiment/transformer/transformer_small/output/model.ckpt.
I0804 19:43:00.463842 140200711067520 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
I0804 19:43:00.505716 140200711067520 basic_session_run_hooks.py:692] global_step/sec: 29.0634