Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SSL verification #54

Open
adam-80 opened this issue Sep 2, 2024 · 2 comments
Open

SSL verification #54

adam-80 opened this issue Sep 2, 2024 · 2 comments

Comments

@adam-80
Copy link

adam-80 commented Sep 2, 2024

I am experiencing SSL errors when I am attempting to use the tap to call internal APIs. The certificates are signed by an internal CA, the bundle is deployed to the machine hosting the tap, using Meltano.

I have set the relevant environment variables to point Python to the certificate bundle.

The tap is able to hit the endpoint and infer the schema with the appropriate fields being collected. So that is successful. The issue is experienced once the schema has been inferred and the next page token has been found. The tap loops through the backoff process with the error:

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)

It seems that it can get the endpoint and the fields, then errors on the sync.

It appears that the tap uses a different method to infer the schema from actually querying the data, and that is not respecting the CA bundle variable.

I am not sure how to proceed, I hope you can offer some advice.

@jlloyd-widen
Copy link
Contributor

I can confirm that the methods used to infer the schema is different from those used to sync. Pagination might not run at all on schema inference. This would be hard for me to develop a fix for because I have nothing test against without being able to hit your internal APIs. My best suggestion is to attempt to fix it yourself and submit a PR. In case it's something managed by the singer-sdk and not this tap, I have just released a new version you might be able to update to and see if the issue persists. Sorry I can't give more insight 😢

@adam-80
Copy link
Author

adam-80 commented Sep 4, 2024

Thanks for the response!

Unfortunately, I am no developer, so my ability to create a PR is very limited. I've been digging around the code, but my limited ability is making this fairly useless.

Testing wouldn't be difficult, you would just need an endpoint that is secured by a self-signed certificate and have the CA bundle available. A possible solution would be to include an environment variable for users to be able to point to a specific CA bundle, which is passed to whatever python libraries are being called, or perhaps an option to disable SSL verification.

I'll update to the new version and report back.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants