We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Is there any way to reduce the http request/responses being printed to the logs? This was for a single table with two rows.
2023-05-18T12:32:39.201092Z [info ] Environment 'dev' is active 2023-05-18T12:32:42.127043Z [warning ] No state was found, complete import. 2023-05-18T12:32:42.528555Z [info ] INFO Using supplied catalog /home/andycarter/vscode_projects/empiric_elt/empiric_meltano/.meltano/run/tap-spreadsheets-anywhere/tap.properties.json. cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.528749Z [info ] INFO Processing 1 selected streams from Catalog cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.528866Z [info ] INFO Syncing stream:test cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.528960Z [info ] INFO Request URL: 'https://devstorefmzyvn6gotqas.blob.core.windows.net/blobcontainer?restype=REDACTED&comp=REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.529053Z [info ] Request method: 'GET' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.529154Z [info ] Request headers: cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.529258Z [info ] 'x-ms-version': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.529364Z [info ] 'Accept': 'application/xml' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.529473Z [info ] 'User-Agent': 'azsdk-python-storage-blob/12.16.0 Python/3.10.6 (Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.35)' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.529576Z [info ] 'x-ms-date': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.529690Z [info ] 'x-ms-client-request-id': '15924b0a-f578-11ed-a56a-00155d6836b4' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.529840Z [info ] 'Authorization': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.530106Z [info ] No body was attached to the request cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.714693Z [info ] time=2023-05-18 13:32:42 name=target_postgres level=INFO message=Schema 'raw__tap_spreadsheets_anywhere' does not exist. Creating... CREATE SCHEMA IF NOT EXISTS raw__tap_spreadsheets_anywhere cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres 2023-05-18T12:32:42.717406Z [info ] INFO Response status: 200 cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.717761Z [info ] Response headers: cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.718086Z [info ] 'Transfer-Encoding': 'chunked' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.718241Z [info ] 'Content-Type': 'application/xml' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.718404Z [info ] 'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.719016Z [info ] 'x-ms-request-id': 'fc5c41ae-701e-0014-2d84-89af86000000' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.719263Z [info ] 'x-ms-client-request-id': '15924b0a-f578-11ed-a56a-00155d6836b4' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.719416Z [info ] 'x-ms-version': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.719569Z [info ] 'Date': 'Thu, 18 May 2023 12:32:38 GMT' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.719833Z [info ] INFO Checking 2 resolved objects for any that match regular expression "in/emp.txt" and were modified since 2017-05-01 00:00:00+00:00 cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.720014Z [info ] INFO Processing 1 resolved objects that met our criteria. Enable debug verbosity logging for more details. cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.720178Z [info ] INFO Syncing file "in/emp.txt". cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.721358Z [info ] INFO Request URL: 'https://devstorefmzyvn6gotqas.blob.core.windows.net/blobcontainer/in/emp.txt' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.722021Z [info ] Request method: 'HEAD' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.722222Z [info ] Request headers: cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.722348Z [info ] 'x-ms-version': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.722473Z [info ] 'Accept': 'application/xml' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.722585Z [info ] 'User-Agent': 'azsdk-python-storage-blob/12.16.0 Python/3.10.6 (Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.35)' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.722705Z [info ] 'x-ms-date': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.722902Z [info ] 'x-ms-client-request-id': '15b1cf70-f578-11ed-a56a-00155d6836b4' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.723121Z [info ] 'Authorization': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.723812Z [info ] No body was attached to the request cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.776114Z [info ] time=2023-05-18 13:32:42 name=target_postgres level=INFO message=Table '"test"' does not exist. Creating... CREATE TABLE IF NOT EXISTS raw__tap_spreadsheets_anywhere."test" ("_smart_source_bucket" character varying, "_smart_source_file" character varying, "_smart_source_lineno" numeric, "personidlastnamefirstnameaddresscity" character varying) cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres 2023-05-18T12:32:42.884202Z [info ] INFO Response status: 200 cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.884708Z [info ] Response headers: cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.885120Z [info ] 'Content-Length': '86' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.885423Z [info ] 'Content-Type': 'text/plain' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.885700Z [info ] 'Content-MD5': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.885974Z [info ] 'Last-Modified': 'Fri, 12 May 2023 14:36:14 GMT' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.886300Z [info ] 'Accept-Ranges': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.886755Z [info ] 'ETag': '"0x8DB52F63D10EF6A"' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.887060Z [info ] 'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.887367Z [info ] 'x-ms-request-id': '5d901676-301e-003a-0b84-89fd91000000' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.887883Z [info ] 'x-ms-client-request-id': '15b1cf70-f578-11ed-a56a-00155d6836b4' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.888275Z [info ] 'x-ms-version': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.888610Z [info ] 'x-ms-creation-time': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.889107Z [info ] 'x-ms-lease-status': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.890025Z [info ] 'x-ms-lease-state': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.890645Z [info ] 'x-ms-blob-type': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.891199Z [info ] 'x-ms-server-encrypted': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.891704Z [info ] 'x-ms-access-tier': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.892250Z [info ] 'x-ms-access-tier-inferred': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.892839Z [info ] 'Date': 'Thu, 18 May 2023 12:32:39 GMT' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.893863Z [info ] INFO Request URL: 'https://devstorefmzyvn6gotqas.blob.core.windows.net/blobcontainer/in/emp.txt' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.894584Z [info ] Request method: 'GET' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.895417Z [info ] Request headers: cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.898002Z [info ] 'x-ms-range': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.899292Z [info ] 'x-ms-version': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.899755Z [info ] 'Accept': 'application/xml' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.900309Z [info ] 'User-Agent': 'azsdk-python-storage-blob/12.16.0 Python/3.10.6 (Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.35)' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.901188Z [info ] 'x-ms-date': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.903005Z [info ] 'x-ms-client-request-id': '15cae50a-f578-11ed-a56a-00155d6836b4' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.903593Z [info ] 'Authorization': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.904158Z [info ] No body was attached to the request cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.917702Z [info ] INFO Response status: 206 cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.918325Z [info ] Response headers: cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.925455Z [info ] 'Content-Length': '86' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.926205Z [info ] 'Content-Type': 'text/plain' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.926697Z [info ] 'Content-Range': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.927164Z [info ] 'Last-Modified': 'Fri, 12 May 2023 14:36:14 GMT' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.927599Z [info ] 'Accept-Ranges': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.928037Z [info ] 'ETag': '"0x8DB52F63D10EF6A"' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.928536Z [info ] 'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.929027Z [info ] 'x-ms-request-id': '5d901692-301e-003a-2384-89fd91000000' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.929394Z [info ] 'x-ms-client-request-id': '15cae50a-f578-11ed-a56a-00155d6836b4' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.929803Z [info ] 'x-ms-version': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.930234Z [info ] 'x-ms-creation-time': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.930665Z [info ] 'x-ms-blob-content-md5': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.933171Z [info ] 'x-ms-lease-status': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.934547Z [info ] 'x-ms-lease-state': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.935142Z [info ] 'x-ms-blob-type': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.935642Z [info ] 'x-ms-server-encrypted': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.936053Z [info ] 'Date': 'Thu, 18 May 2023 12:32:39 GMT' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:42.936905Z [info ] INFO Wrote 2 records for stream "test". cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere 2023-05-18T12:32:43.049936Z [info ] time=2023-05-18 13:32:43 name=target_postgres level=INFO message=Loading 2 rows into 'raw__tap_spreadsheets_anywhere."test"' cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres 2023-05-18T12:32:43.080390Z [info ] time=2023-05-18 13:32:43 name=target_postgres level=INFO message=Loading into raw__tap_spreadsheets_anywhere."test": {"inserts": 2, "updates": 0, "size_bytes": 126} cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres 2023-05-18T12:32:43.139285Z [info ] Incremental state has been updated at 2023-05-18 12:32:43.139165. 2023-05-18T12:32:43.165229Z [info ] Block run completed. block_type=ExtractLoadBlocks err=None set_number=0 success=True
The text was updated successfully, but these errors were encountered:
After investigation this is due to the logging policy in azure.core, closing.
Sorry, something went wrong.
I was able to silence the info from Azure Blob with this added the top of file_utils.py
file_utils.py
logger=logging.getLogger('azure.core.pipeline.policies.http_logging_policy') logger.setLevel(logging.WARNING)
It makes for much cleaner logs now.
No branches or pull requests
Is there any way to reduce the http request/responses being printed to the logs? This was for a single table with two rows.
The text was updated successfully, but these errors were encountered: