Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hominid script ends with error #2

Open
ankit4035 opened this issue Nov 14, 2019 · 31 comments
Open

Hominid script ends with error #2

ankit4035 opened this issue Nov 14, 2019 · 31 comments

Comments

@ankit4035
Copy link

Hi,
I have used Hominid where it was working well. However, recently it is throwing an error in a new file.
The error seems to be something like:
ValueError: Input contains NaN, infinity or a value too large for dtype('float64')
and
TypeError: 'NoneType' object is not iterable

After throwing the error, scripts stops execution. Prior to this faulty SNP (where script halts), all other SNPs were processed and written in output file as well.

Please help me where I could be wrong.

I am attaching the scrrenshots of the script as well as error message.

Thanks,

Ankit

Screenshot from 2019-11-14 10:18:10
Screenshot from 2019-11-14 10:19:41
@jklynch
Copy link
Collaborator

jklynch commented Nov 14, 2019 via email

@ankit4035
Copy link
Author

Hi Josh,

I checked that row, there is no "NA" or "NaN" in it.
Screenshot from 2019-11-14 17:44:48

I have also reinstalled the hominid and still facing the same issue.

@jklynch
Copy link
Collaborator

jklynch commented Nov 14, 2019 via email

@ankit4035
Copy link
Author

ankit4035 commented Nov 15, 2019 •
edited
Loading

Hi Josh,

Here is the row above and below it.
Screenshot from 2019-11-15 09:12:22

As a matter of fact, I checked and confirmed that there isn't a single "NA" or "NaN" in the entire file.

I have also tried to remove that SNP, the same error was encountered with other SNP. I then also removed that SNP and tried, again same error was encountered. I continued this for four times and each time different SNP was giving error.

Ankit

@jklynch
Copy link
Collaborator

jklynch commented Nov 15, 2019 via email

@ankit4035
Copy link
Author

Hi Josh,

Yes, -1 is used for missing/Undetermined call. As per your suggestion, I replaced all -1 with "NA" in the file and run it again. The result is still the same, error at the same SNP.

Screenshot from 2019-11-16 09:52:46
Screenshot from 2019-11-16 09:50:18

If possible, can you send me that script you were talking about.

Ankit

@jklynch
Copy link
Collaborator

jklynch commented Nov 16, 2019 via email

@ankit4035
Copy link
Author

Hi Josh,

I think you forgot to attach the script. Can you please send it again.

Ankit

@jklynch
Copy link
Collaborator

jklynch commented Nov 18, 2019

I had to change the file extension to .txt and attach it directly to the issue.

align_input_files.txt

@ankit4035
Copy link
Author

ankit4035 commented Nov 19, 2019 •
edited
Loading

Hi Josh,

The script executed perfectly without any error. All the SNPs were finite and no issues were encountered.

Ankit

@ankit4035 ankit4035 reopened this Nov 19, 2019
@ankit4035
Copy link
Author

Sorry I closed the issue by mistake..

@jklynch
Copy link
Collaborator

jklynch commented Nov 20, 2019 via email

@ankit4035
Copy link
Author

Hi Josh,

Your instructions are very clear. I installed a new hominid and created a new virtual env. for it as well. I run the first five SNPs (problematic SNP AXCM75183727 is the second SNP) and the script ended up with the error.

I am attaching here :

  1. Subsetted rvcf file (I have to change the filename and extension)
    test-input-rvcf.txt

  2. Messages on the screen (I redirected them to a file)
    test-run-message.txt

  3. Result file that was created (changed the name and its extension).
    test-result-rvcf.txt

Ankit

@jklynch
Copy link
Collaborator

jklynch commented Nov 21, 2019

Hi Ankit,

I was not expecting to see that all elements of y_true are nan. My first thought is that none of the SNP data is aligning with the taxa data. I added another print to see what the taxa data looks like. Would you please run git pull from the hominid directory and then run hominid again? Sorry this is taking so long.

Josh

@ankit4035
Copy link
Author

Hi Josh,

I did as you asked. Here are the files.

test_result-rvcf.txt
test-run-message.txt

And thanks for sorting this out. Let's finish this.

Ankit

@jklynch
Copy link
Collaborator

jklynch commented Nov 22, 2019

Hi Ankit,

I added some more prints to show the data before it is sent to a worker. Would you please repeat the git pull and run it again?

Josh

@ankit4035
Copy link
Author

Hi Josh,

Did as you asked and here are the results file.

test-run-message.txt
test-result-rvcf.txt

Ankit

@jklynch
Copy link
Collaborator

jklynch commented Dec 1, 2019

Hi Ankit,

Sorry I have been out of touch lately. I just added some prints to the training loop since the data looks ok before that point. This could generate a lot of output. Please give it a try if you like.

Josh

@ankit4035
Copy link
Author

Hi Josh,

Here are the output files.

test-run-message.txt
test-result-rvcf.txt

Ankit

@jklynch
Copy link
Collaborator

jklynch commented Dec 26, 2019

Hello Ankit,
I have not been able to figure out why you are having this problem, and I don't think I can do it without the actual data. If you could send just two rows of the snp data (one good and one bad) plus a few rows of the taxa data I might be able to do it.

Josh

@ankit4035
Copy link
Author

Hi Josh,

Here are the files:

  1. Input OTU table (truncated)
    test-ASV.txt

  2. Good SNP data
    test.rvcf.txt

  3. Bad SNP data
    test-input-rvcf.txt

I tried with combined good and bad SNP, received the error in SNP AXCM75183727. I removed that line and tried again, received the error in SNP AXCM75183728. After removing both SNP, the file is running without any errors.

Merry Christmas and happy holidays.

Ankit

@jklynch
Copy link
Collaborator

jklynch commented Jan 21, 2020

Hi Ankit,

I am not getting any errors from the bad SNP file. Do you get errors from it?

Josh

@ankit4035
Copy link
Author

Hi Josh,

This is the output from the terminal screen.

test-bad-message.txt

Ankit

@jklynch
Copy link
Collaborator

jklynch commented Jan 22, 2020

Maybe there is something wrong with the formatting of the input file. When I run the bad SNP data in test-input-rvcf.txt I get no errors. This is the output:

(hom) jlynch@minty:~/local/hominid/issue-2/scripts$ ./test_bad_hominid.sh 
Namespace(cv_count=100, input_taxon_table_fp='../data/issue-2-test-ASV.txt', input_vcf_fp='../data/issue-2-bad-test-input.rvcf', maf_lower_cutoff=0.2, output_vcf_fp='../issue-2-bad-test-output.rvcf', permutation_method='no_permutation', single_process=False, snp_limit=-1, transform='arcsinsqrt')
building MPI lasso
[worker 1] running on minty
[worker 1] sending request for work
[worker 1] waiting for work
Namespace(cv_count=100, input_taxon_table_fp='../data/issue-2-test-ASV.txt', input_vcf_fp='../data/issue-2-bad-test-input.rvcf', maf_lower_cutoff=0.2, output_vcf_fp='../issue-2-bad-test-output.rvcf', permutation_method='no_permutation', single_process=False, snp_limit=-1, transform='arcsinsqrt')
building MPI lasso
loading taxon table file ../data/issue-2-test-ASV.txt
Namespace(cv_count=100, input_taxon_table_fp='../data/issue-2-test-ASV.txt', input_vcf_fp='../data/issue-2-bad-test-input.rvcf', maf_lower_cutoff=0.2, output_vcf_fp='../issue-2-bad-test-output.rvcf', permutation_method='no_permutation', single_process=False, snp_limit=-1, transform='arcsinsqrt')
building MPI lasso
[worker 2] running on minty
[worker 2] sending request for work
[worker 2] waiting for work
  taxon table has 195 rows
  taxon table has 283 columns
applying arcsin sqrt transformation
      AECK01CM1B  ...  AECK30CM8B
Sex          0.0  ...    1.570796
ASV1         0.0  ...    0.000000
ASV2         0.0  ...    0.000000
ASV3         0.0  ...    0.000000
ASV4         0.0  ...    0.000000

[5 rows x 283 columns]
[controller] recv message from worker 1 with tag 0
283 aligned samples for 10 ENSGALT00000002226 AXCM75183727
  maf 0.13 is too low for ENSGALT00000002226 AXCM75183727
*** write_snp_to_file:
   aligned_sample_count  ...  AECK23CM10B
0                   283  ...            2

[1 rows x 310 columns]
280 aligned samples for 1 ENSGALT00000050393 AXCM75183728
[controller] sending a task to worker 1 with SNP ENSGALT00000050393 AXCM75183728
[worker 1] waited 1.01s for work
[worker 1] received message with tag 3
testing SNP ENSGALT00000050393 AXCM75183728
[worker 2] waited 0.87s for work
[worker 2] received message with tag 2
[worker 2] received exit message
[worker 2] exiting after 0 tasks in    0.87s
[controller] recv message from worker 2 with tag 0
[controller] sending exit message to worker 2
[controller] recv message from worker 2 with tag 2
[controller] received exit message from worker 2
ENSGALT00000050393 AXCM75183728 rsq_mean 95% (pi)  : 0.1501 <-- 0.1646 --> 0.1790
ENSGALT00000050393 AXCM75183728 rsq_median 95% (pi): 0.1558 <-- 0.1811 --> 0.1888
ENSGALT00000050393 AXCM75183728 rsq_mean 99% (pi)  : 0.1455 <-- 0.1646 --> 0.1833
ENSGALT00000050393 AXCM75183728 rsq_median 99% (pi): 0.1460 <-- 0.1811 --> 0.1912
[worker 1] task time: 11.25s
[worker 1] sending request for work
[worker 1] waiting for work
[controller] recv message from worker 1 with tag 1
*** write_snp_to_file:
   aligned_sample_count  ...  AECK23CM10B
1                   280  ...            2

[1 rows x 310 columns]
[controller] received a processed task from worker 1
[controller] recv message from worker 1 with tag 0
[controller] sending exit message to worker 1
[worker 1] waited 0.30s for work
[worker 1] received message with tag 2
[worker 1] received exit message
[worker 1] exiting after 1 tasks in   12.56s
[controller] recv message from worker 1 with tag 2
[controller] received exit message from worker 1
[controller] all workers have exited

@ankit4035
Copy link
Author

I am using the same files, the command gets completed but shows error at the end.

message.txt

When I was running with complete files, the script stops on showing this error. However, it don't terminate the command and keeps using all the resources. Also, no new lines are added in the output file.

I will try preparing the files again and see what happens. Thanks.

@ankit4035
Copy link
Author

Hi Josh,

I reprepared both input files and reinstalled Hominid, but the same error persists. I also tried installing Hominid on a new system and the same error remains.

Would it possible for you to run it if I can share the complete file with you. Can I contact you by email for the same?

Ankit

@jklynch
Copy link
Collaborator

jklynch commented Jul 3, 2020

Hi Ankit,

I have released hominid version 1.1.0. It has better exception handling so a worker will not die if an exception is raised. Also in the case of the numerical error you have experienced the worker will retry up to three times. This is because the error seems to be caused when the random selection of the 0, 1, and 2 classes results in a testing or validation set that is smaller than it should be. If three attempts all fail the worker will give up on that snp and request a new one. Those are the only changes.

Josh

@ankit4035
Copy link
Author

HI Josh,

Thanks for the update. I will work with new version and let you know hot it goes..

Ankit

@ankit4035
Copy link
Author

Hi Josh,

I used the new version, and the file completed without any error. Also, the file with "cv_scores.txt" contained many entries which was missing in previous version.

I also wanted to ask/clarify few things:

  • Does running the same file again with same parameters result in identical output, or there will be some minor changes in both?
  • Also, if I run a subset of SNPs, should I be getting the same result for those SNPs as original file?

Thanks for your help.

Ankit

@jklynch
Copy link
Collaborator

jklynch commented Jul 20, 2020 via email

@ankit4035
Copy link
Author

Hi Josh,

I think I got your point.
I was asking these as I subsetted files with 1000 SNPs and run it with previous version. I obtained similar error, but running it again seemed to solve the issue for me.

Anyways, thanks again for you help.

Ankit

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jklynch @ankit4035