We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Program crashing when beginning the epoch1? like this, Epoch 1/300 tcmalloc: large alloc 3623936000 bytes == 0x55d811920000 @ 0x7fe4a6f961e7 0x7fe473f84902 0x7fe476b62110 0x7fe476bc6bd1 0x7fe476bc86d8 0x7fe476bcac83 0x7fe4700a000c 0x7fe470092775 0x7fe4701531c1 0x7fe4701508b8 0x7fe4a58786df 0x7fe4a6d4b6db 0x7fe4a5e8071f tcmalloc: large alloc 3623936000 bytes == 0x55d8f24ae000 @ 0x7fe4a6f961e7 0x7fe473f84902 0x7fe476b62110 0x7fe476bc6bd1 0x7fe476bc86d8 0x7fe476bcac83 0x7fe4700a000c 0x7fe470092775 0x7fe4701531c1 0x7fe4701508b8 0x7fe4a58786df 0x7fe4a6d4b6db 0x7fe4a5e8071f tcmalloc: large alloc 4831920128 bytes == 0x55d8f24ae000 @ 0x7fe4a6f961e7 0x7fe473f84902 0x7fe476b62110 0x7fe476bc6bd1 0x7fe476bc86d8 0x7fe476bcac83 0x7fe4700a000c 0x7fe470092775 0x7fe4701531c1 0x7fe4701508b8 0x7fe4a58786df 0x7fe4a6d4b6db 0x7fe4a5e8071f tcmalloc: large alloc 4831920128 bytes == 0x55d9324ae000 @ 0x7fe4a6f961e7 0x7fe473f84902 0x7fe476b62110 0x7fe476bc6bd1 0x7fe476bc86d8 0x7fe476bcac83 0x7fe4700a000c 0x7fe470092775 0x7fe4701531c1 0x7fe4701508b8 0x7fe4a58786df 0x7fe4a6d4b6db 0x7fe4a5e8071f 2021-03-07 11:35:48.388501: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 7247757312 exceeds 10% of system memory. 2021-03-07 11:35:48.388501: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 7247757312 exceeds 10% of system memory. tcmalloc: large alloc 7247757312 bytes == 0x55d9324ae000 @ 0x7fe4a6f78b6b 0x7fe4a6f98379 0x7fe470171087 0x7fe46ff6d59f 0x7fe46fe37f1b 0x7fe46fdfd8f6 0x7fe46fe005f1 0x7fe476aa1c00 0x7fe4700a000c 0x7fe470092775 0x7fe4701531c1 0x7fe4701508b8 0x7fe4a58786df 0x7fe4a6d4b6db 0x7fe4a5e8071f tcmalloc: large alloc 7247757312 bytes == 0x55daeacd0000 @ 0x7fe4a6f78b6b 0x7fe4a6f98379 0x7fe470171087 0x7fe46ff6d59f 0x7fe46fe37f1b 0x7fe46fdfd8f6 0x7fe46fe005f1 0x7fe476aa2c3d 0x7fe4700a000c 0x7fe470092775 0x7fe4701531c1 0x7fe4701508b8 0x7fe4a58786df 0x7fe4a6d4b6db 0x7fe4a5e8071f ^C
The text was updated successfully, but these errors were encountered:
That's an out of memory error. $ nvidia-smi to check you GPU status.
$ nvidia-smi
Sorry, something went wrong.
No branches or pull requests
Program crashing when beginning the epoch1?
like this,
Epoch 1/300
tcmalloc: large alloc 3623936000 bytes == 0x55d811920000 @ 0x7fe4a6f961e7 0x7fe473f84902 0x7fe476b62110 0x7fe476bc6bd1 0x7fe476bc86d8 0x7fe476bcac83 0x7fe4700a000c 0x7fe470092775 0x7fe4701531c1 0x7fe4701508b8 0x7fe4a58786df 0x7fe4a6d4b6db 0x7fe4a5e8071f
tcmalloc: large alloc 3623936000 bytes == 0x55d8f24ae000 @ 0x7fe4a6f961e7 0x7fe473f84902 0x7fe476b62110 0x7fe476bc6bd1 0x7fe476bc86d8 0x7fe476bcac83 0x7fe4700a000c 0x7fe470092775 0x7fe4701531c1 0x7fe4701508b8 0x7fe4a58786df 0x7fe4a6d4b6db 0x7fe4a5e8071f
tcmalloc: large alloc 4831920128 bytes == 0x55d8f24ae000 @ 0x7fe4a6f961e7 0x7fe473f84902 0x7fe476b62110 0x7fe476bc6bd1 0x7fe476bc86d8 0x7fe476bcac83 0x7fe4700a000c 0x7fe470092775 0x7fe4701531c1 0x7fe4701508b8 0x7fe4a58786df 0x7fe4a6d4b6db 0x7fe4a5e8071f
tcmalloc: large alloc 4831920128 bytes == 0x55d9324ae000 @ 0x7fe4a6f961e7 0x7fe473f84902 0x7fe476b62110 0x7fe476bc6bd1 0x7fe476bc86d8 0x7fe476bcac83 0x7fe4700a000c 0x7fe470092775 0x7fe4701531c1 0x7fe4701508b8 0x7fe4a58786df 0x7fe4a6d4b6db 0x7fe4a5e8071f
2021-03-07 11:35:48.388501: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 7247757312 exceeds 10% of system memory.
2021-03-07 11:35:48.388501: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 7247757312 exceeds 10% of system memory.
tcmalloc: large alloc 7247757312 bytes == 0x55d9324ae000 @ 0x7fe4a6f78b6b 0x7fe4a6f98379 0x7fe470171087 0x7fe46ff6d59f 0x7fe46fe37f1b 0x7fe46fdfd8f6 0x7fe46fe005f1 0x7fe476aa1c00 0x7fe4700a000c 0x7fe470092775 0x7fe4701531c1 0x7fe4701508b8 0x7fe4a58786df 0x7fe4a6d4b6db 0x7fe4a5e8071f
tcmalloc: large alloc 7247757312 bytes == 0x55daeacd0000 @ 0x7fe4a6f78b6b 0x7fe4a6f98379 0x7fe470171087 0x7fe46ff6d59f 0x7fe46fe37f1b 0x7fe46fdfd8f6 0x7fe46fe005f1 0x7fe476aa2c3d 0x7fe4700a000c 0x7fe470092775 0x7fe4701531c1 0x7fe4701508b8 0x7fe4a58786df 0x7fe4a6d4b6db 0x7fe4a5e8071f
^C
The text was updated successfully, but these errors were encountered: