Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: issue 2066 - updated sample MySQL data loader file. Signed-off-by: Malavika <[email protected]> #2067

Closed
wants to merge 3 commits into from
Closed

fix: issue 2066 - updated sample MySQL data loader file. Signed-off-by: Malavika <[email protected]> #2067

wants to merge 3 commits into from

Conversation

MalavikaN1
Copy link

@MalavikaN1 MalavikaN1 commented Dec 29, 2022 •
edited
Loading

fixes: #2066

Summary of Changes

Changes made to:
https://github.com/amundsen-io/amundsen/blob/main/databuilder/example/scripts/sample_mysql_loader.py

import pymysql
pymysql.install_as_MySQLdb()

def connection_string():
    user = 'root'
    password='root'
    host = 'localhost'
    port = '3307'
    db = 'test_db'
    return "mysql+pymysql://%s:%s@%s:%s/%s" % (user,password, host, port, db)

job_config = ConfigFactory.from_dict({
        f'extractor.mysql_metadata.{MysqlMetadataExtractor.WHERE_CLAUSE_SUFFIX_KEY}': where_clause_suffix,
        f'extractor.mysql_metadata.{MysqlMetadataExtractor.USE_CATALOG_AS_CLUSTER_NAME}': True,
        f'extractor.mysql_metadata.extractor.sqlalchemy.{SQLAlchemyExtractor.CONN_STRING}': connection_string(),
        f'loader.filesystem_csv_neo4j.{FsNeo4jCSVLoader.NODE_DIR_PATH}': node_files_folder,
        f'loader.filesystem_csv_neo4j.{FsNeo4jCSVLoader.RELATION_DIR_PATH}': relationship_files_folder,
        f'publisher.neo4j.{neo4j_csv_publisher.NODE_FILES_DIR}': node_files_folder,
        f'publisher.neo4j.{neo4j_csv_publisher.RELATION_FILES_DIR}': relationship_files_folder,
        f'publisher.neo4j.{neo4j_csv_publisher.NEO4J_END_POINT_KEY}': neo4j_endpoint,
        f'publisher.neo4j.{neo4j_csv_publisher.NEO4J_USER}': neo4j_user,
        f'publisher.neo4j.{neo4j_csv_publisher.NEO4J_PASSWORD}': neo4j_password,
        f'publisher.neo4j.{neo4j_csv_publisher.NEO4J_ENCRYPTED}': False,
        f'publisher.neo4j.{neo4j_csv_publisher.JOB_PUBLISH_TAG}': 'unique_tag',  # should use unique tag here like {ds}
    })

Changes made to :
https://github.com/amundsen-io/amundsen/blob/main/databuilder/setup.py

requirements_path = os.path.join(os.path.dirname(os.path.realpath(__file__)),
                                 '../requirements.txt')

CheckList

Make sure you have checked all steps below to ensure a timely review.

  • PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2"
  • PR includes a summary of changes.
  • PR adds unit tests, updates existing unit tests, OR documents why no test additions or modifications are needed.
  • In case of new functionality, my PR adds documentation that describes how to use it.
    • All the public functions and the classes in the PR contain docstrings that explain what it does

@boring-cyborg boring-cyborg bot added the area:databuilder From databuilder folder label Dec 29, 2022
@boring-cyborg
Copy link

boring-cyborg bot commented Dec 29, 2022

Congratulations on your first Pull Request and welcome to Amundsen community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/amundsen-io/amundsen/blob/main/CONTRIBUTING.md)

@feng-tao
Copy link
Member

feng-tao commented Jan 4, 2023

databuilder/dist/amundsen_databuilder-7.4.3-py3.10.egg is not needed but checked in this pr.

@MalavikaN1 MalavikaN1 changed the title fix: issue 2066 - updated sample MySQL data loader file. fix: issue 2066 - updated sample MySQL data loader file. Signed-off-by: Malavika <[email protected]> Jan 17, 2023
@Golodhros
Copy link
Member

@MalavikaN1 could you review the PR with Tao's comment?

@stale
Copy link

stale bot commented Feb 18, 2023

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

@stale stale bot added the stale label Feb 18, 2023
@stale
Copy link

stale bot commented Mar 12, 2023

This issue has been automatically closed for inactivity. If you still wish to make these changes, please open a new pull request or reopen this one.

@stale stale bot closed this Mar 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@feng-tao feng-tao Awaiting requested review from feng-tao feng-tao is a code owner

@jinhyukchang jinhyukchang Awaiting requested review from jinhyukchang jinhyukchang is a code owner

@allisonsuarez allisonsuarez Awaiting requested review from allisonsuarez allisonsuarez is a code owner

@dorianj dorianj Awaiting requested review from dorianj

@verdan verdan Awaiting requested review from verdan verdan is a code owner

@mgorsk1 mgorsk1 Awaiting requested review from mgorsk1 mgorsk1 is a code owner

@youngyjd youngyjd Awaiting requested review from youngyjd youngyjd is a code owner

@dechoma dechoma Awaiting requested review from dechoma dechoma is a code owner

@sewardgw sewardgw Awaiting requested review from sewardgw sewardgw is a code owner

@dkunitsk dkunitsk Awaiting requested review from dkunitsk dkunitsk is a code owner

@kristenarmes kristenarmes Awaiting requested review from kristenarmes kristenarmes is a code owner

Assignees
No one assigned
Labels
area:databuilder From databuilder folder stale
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Amundsen is unable to import MYSQL data
3 participants
@MalavikaN1 @feng-tao @Golodhros