Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FDS-2525] Authenticated export of telemetry data #1527

Merged
merged 53 commits into from
Nov 13, 2024
Merged

[FDS-2525] Authenticated export of telemetry data #1527

merged 53 commits into from
Nov 13, 2024

Conversation

BryanFauble
Copy link
Collaborator

@BryanFauble BryanFauble commented Oct 29, 2024 •
edited
Loading

Problem:

  1. When creating telemetry data where an open-telemetry collector is not available to perform an oauth2 token exchange the only way to attach an auth token on outbound requests was to define a static header an environment variables.

Solution:

  1. Adding the requests-oauth2client library that will handle the oauth2 token exchange with Auth0 for us if the environment variables are set.
  2. Attach the auth mechanism on the requests.Session() object that the opentelemetry exporter uses to send data out.
  3. This is the first portion of https://sagebionetworks.jira.com/browse/FDS-2525 - Which allows developers to export telemetry data locally. It also verifies that all we need to do for FDS-2525 is set a few environment variables and allow schematic running within AWS to export it's data to the OTEL collector, which will then forward the data to the DPE K8s cluster.

Testing:

  1. I ran this code locally with the environment variables set and confirmed that both traces AND log were exported and viewable within SigNoz on the sandbox kubernetes cluster.
  2. I also verified I could point this at the https://dev.sagedpe.org URL too

TODO:

  • Remove hacky code around verify=False for the HTTP POST logic when non-self signed certs can be used.

@BryanFauble
Adding changes for synapse entity tracking
b369ef4
@BryanFauble
Create unique test suites for each run
30def7e
@BryanFauble
Remove extra try-catch
093e179
@BryanFauble
Remove todo
05e936c
@BryanFauble
Remove un-needed code
57b8408
@BryanFauble
mypy flagged issues
2ef297f
@BryanFauble
mypy flagged issues
476b627
@BryanFauble
pylint patching
9b08554
@BryanFauble
pylint issue resolution
e4eff7a
@BryanFauble
Remove py3.10 only feature
6226efa
@BryanFauble
Merge branch 'develop' into fds-2386-synapse-refactoring
6b77396
@BryanFauble
Patching for issues
906a298
@BryanFauble
Merge remote-tracking branch 'origin/develop-fix-gh-workflow' into fd…
a7fab56 
…s-2386-synapse-refactoring
@BryanFauble
Bug patching
fe9b27e
@BryanFauble
Merge branch 'develop' into fds-2386-synapse-refactoring
b445853
@BryanFauble
Adding tests for io_utils changes
f10640e
@BryanFauble
Log tests to CLI (Temp)
38eb3ec
@BryanFauble
Create dir if not exsists
51bc8fc
@BryanFauble
Merge branch 'develop' into fds-2386-synapse-refactoring
058eed4
@BryanFauble
Don't log the cli
14fdc96
@BryanFauble
Update to released version of synapse python client
cd60d3c
@BryanFauble
Add docstrings
662461d
@BryanFauble
Code review feedback
ff02757
@BryanFauble
Don't build data docs if not logging for debug
f84fad0
@BryanFauble
assertion that entity_type is not None
8cc688b
@BryanFauble
Attach package version
a92a27f
@BryanFauble
Include the file size of the manifest used in the operation
6489df4
@BryanFauble
Move version of package to Resource definition
07afa48
@BryanFauble
Move to grabbing owner_id and exporting everything over GRPC lib
8f10eaf
@BryanFauble
Add examples
3a01976
Base automatically changed from attach-additional-telemetry-data to develop November 1, 2024 19:01
@BryanFauble BryanFauble marked this pull request as ready for review November 4, 2024 23:21
@BryanFauble BryanFauble requested a review from a team as a code owner November 4, 2024 23:21
thomasyu888
thomasyu888 approved these changes Nov 5, 2024
View reviewed changes
Copy link
Member

@thomasyu888 thomasyu888 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔥 LGTM! Thanks for doing this. Make sense to me, I'll leave it to at least one other developer to review this.

@GiaJordan
Copy link
Contributor

@BryanFauble The changes look good to me!
I just see that a couple tests are failing and there are a couple merge conflicts.

@sonarcloud SonarCloud
Copy link

sonarcloud bot commented Nov 13, 2024

Quality Gate Passed Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
45.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@BryanFauble BryanFauble merged commit 778bf54 into develop Nov 13, 2024
7 checks passed
@BryanFauble BryanFauble deleted the fds-2525-authenticated-exports branch November 13, 2024 16:35
GiaJordan pushed a commit that referenced this pull request Nov 13, 2024
@BryanFauble @GiaJordan
[FDS-2525] Authenticated export of telemetry data (#1527)
65fb55d 
* Authenticated export of telemetry data, updating to HTTP otel library
andrewelamb added a commit that referenced this pull request Nov 14, 2024
@andrewelamb @BryanFauble
@jaymedina @thomasyu888 @linglp @GiaJordan
V24.11.1 (#1543)
2f40625 
* add new tests

* add unit tests

* ran black

* Update schematic/models/validate_attribute.py

Co-authored-by: BryanFauble <[email protected]>

* added tests

* Update README.md

* Update README.md

* add unit tests

* run black

* Update README.md

* temp commit

* remove old tests

* [FDS-2386] Synapse entity tracking and code concurrency updates (#1505)

* [FDS-2386] Synapse entity tracking and code concurrency updates

* ran black

* Update CODEOWNERS

* updated data model type rules to include error param

* fix validate type attribute to use msg level param

* added error handling

* run black

* create Node class

* sat up Node class so that nodes with no displayName fields cause an error on creation

* ran black

* ran mypy

* added new configs for CLI tests

* added new manifests for testing CLI commands

* automate manual CLI tests

* ran black

* Update CODEOWNERS

* Update scan_repo.yml

* Update .github/CODEOWNERS

* Update .github/workflows/scan_repo.yml

* Attach additional telemetry data to OTEL traces (#1519)

* Attach additional telemetry data to OTEL traces

* feat: added tracing for cross manifest validation and file name validation  (#1509)

* add tracing for GX validation

* temp commit

* Updating contribution doc to expect squash and merge (#1534)

* [FDS-2491] Integration tests for Schematic API Test plan (#1512)

Integration tests for Schematic API Test plan

* [FDS-2500] Add Integration Tests for: Manifest Validation (#1516)

* Add Integration Tests for: Manifest Validation

* [FDS-2449] Lock `sphinx` version and update `poetry.lock` (#1530)

Also install `typing-extensions` in the build

* manual test files now being saved in manifests folder

* manual test files now being saved in manifests folder

* remove lines to delete json files that were under git control

* ran black

* add try finally blocks to remove created files

* ran black

* add lines to remove created json files

* Update file annotation store process to require filename be present in order to annotate file

* add lines to remove created json files

* Revert "Update file annotation store process to require filename be present in order to annotate file"

This reverts commit f57c718.

* Don't attempt to annotate the table

* add code in finally blocks to reset config to default values, when tests change them

* complete submit manifest command test

* ran black

* add test for bug case

* update test for table tidyness

* remove unused import

* remove etag column if already present when building temp file view

* catch all exceptions to switch to sequential mode

* update test for updated data

* Revert "update test for updated data"

This reverts commit 255e3c0.

* Revert "catch all exceptions to switch to sequential mode"

This reverts commit 68b0b24.

* catch ValueErrors as well

* Updates for integration test failures (#1537)

* Updates for integration test failures, Config file reset and scope changes

* add todos for removing config resets

* [FDS-2525] Authenticated export of telemetry data (#1527)

* Authenticated export of telemetry data, updating to HTTP otel library

* temp reduce tests

* restore tests

* uncomment tests

* redid how files are deleted, manual tests values are set

* ran black

* [SCHEMATIC-157] Make some dependencies required to avoid `schematic CLI` commands from potentially erroring when doing a pip install (#1540)

* Make otel flash non-optional

* Add dependencies as non-optional

* Update pyproject.toml

---------

Co-authored-by: BryanFauble <[email protected]>
Co-authored-by: Jenny V Medina <[email protected]>
Co-authored-by: Thomas Yu <[email protected]>
Co-authored-by: Lingling <[email protected]>
Co-authored-by: GiaJordan <[email protected]>
andrewelamb added a commit that referenced this pull request Nov 15, 2024
@andrewelamb @BryanFauble
@jaymedina @thomasyu888 @linglp @GiaJordan
V24.11.2 (#1549)
816534a 
* add new tests

* add unit tests

* ran black

* Update schematic/models/validate_attribute.py

Co-authored-by: BryanFauble <[email protected]>

* added tests

* Update README.md

* Update README.md

* add unit tests

* run black

* Update README.md

* temp commit

* remove old tests

* [FDS-2386] Synapse entity tracking and code concurrency updates (#1505)

* [FDS-2386] Synapse entity tracking and code concurrency updates

* ran black

* Update CODEOWNERS

* updated data model type rules to include error param

* fix validate type attribute to use msg level param

* added error handling

* run black

* create Node class

* sat up Node class so that nodes with no displayName fields cause an error on creation

* ran black

* ran mypy

* added new configs for CLI tests

* added new manifests for testing CLI commands

* automate manual CLI tests

* ran black

* Update CODEOWNERS

* Update scan_repo.yml

* Update .github/CODEOWNERS

* Update .github/workflows/scan_repo.yml

* Attach additional telemetry data to OTEL traces (#1519)

* Attach additional telemetry data to OTEL traces

* feat: added tracing for cross manifest validation and file name validation  (#1509)

* add tracing for GX validation

* temp commit

* Updating contribution doc to expect squash and merge (#1534)

* [FDS-2491] Integration tests for Schematic API Test plan (#1512)

Integration tests for Schematic API Test plan

* [FDS-2500] Add Integration Tests for: Manifest Validation (#1516)

* Add Integration Tests for: Manifest Validation

* [FDS-2449] Lock `sphinx` version and update `poetry.lock` (#1530)

Also install `typing-extensions` in the build

* manual test files now being saved in manifests folder

* manual test files now being saved in manifests folder

* remove lines to delete json files that were under git control

* ran black

* add try finally blocks to remove created files

* ran black

* add lines to remove created json files

* Update file annotation store process to require filename be present in order to annotate file

* add lines to remove created json files

* Revert "Update file annotation store process to require filename be present in order to annotate file"

This reverts commit f57c718.

* Don't attempt to annotate the table

* add code in finally blocks to reset config to default values, when tests change them

* complete submit manifest command test

* ran black

* add test for bug case

* update test for table tidyness

* remove unused import

* remove etag column if already present when building temp file view

* catch all exceptions to switch to sequential mode

* update test for updated data

* Revert "update test for updated data"

This reverts commit 255e3c0.

* Revert "catch all exceptions to switch to sequential mode"

This reverts commit 68b0b24.

* catch ValueErrors as well

* Updates for integration test failures (#1537)

* Updates for integration test failures, Config file reset and scope changes

* add todos for removing config resets

* [FDS-2525] Authenticated export of telemetry data (#1527)

* Authenticated export of telemetry data, updating to HTTP otel library

* temp reduce tests

* restore tests

* uncomment tests

* redid how files are deleted, manual tests values are set

* ran black

* [SCHEMATIC-157] Make some dependencies required to avoid `schematic CLI` commands from potentially erroring when doing a pip install (#1540)

* Make otel flash non-optional

* Add dependencies as non-optional

* Include schematic_api for now (#1547)

* update toml version to 24.11.1

---------

Co-authored-by: BryanFauble <[email protected]>
Co-authored-by: Jenny V Medina <[email protected]>
Co-authored-by: Thomas Yu <[email protected]>
Co-authored-by: Lingling <[email protected]>
Co-authored-by: GiaJordan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@GiaJordan GiaJordan GiaJordan approved these changes

@thomasyu888 thomasyu888 thomasyu888 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@BryanFauble @GiaJordan @thomasyu888 @linglp