-
Notifications
You must be signed in to change notification settings - Fork 53
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Refresh README, examples and changelog with recent changes (#78)
This PR refreshes README, examples of our and changelog to better emphasise recent changes in the validation/open API of ours clarifying read-only properties in our API. Specifically DAGRun dag_id has not been marked as read-only so far which was a bug, and some of our "dev" examples set the read only properties, which could mislead our users. We've recently clarified that those properties are read-only and we are now updating the API documentation and examples to show that the good way of using related APIs. Also the examples have been refrehsed and modernized a bit as well as the dev example has been synchronized automatically (via pre-commit) to make sure it is the same in the README and in dev where we are using it to run the tests with the API. Minimum version of Python have been set to 3.7 and CHANGELOG was updated to reflect all the changes, as well as it has been exposed in Project_URL of the package in PyPI. Link in README have been changed to URL so that they are properly rendered in README in `pypi` package documentation (currently following the links to other filess in the repo there fails.
- Loading branch information
Showing
6 changed files
with
224 additions
and
56 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -19,16 +19,12 @@ | |
|
|
||
| # Apache Airflow Python Client | ||
|
|
||
| > **_NOTE:_** The Apache Airflow Client is still under active development and some methods | ||
| > or APIs might be broken. Please raise an issue in github if you encounter any such issues. | ||
|
|
||
|
|
||
| ## Requirements. | ||
|
|
||
| Python >= 3.6 | ||
| Python >= 3.7 | ||
|
|
||
| ## Installation & Usage | ||
|
|
||
| ### pip install | ||
|
|
||
| You can install directly using pip: | ||
|
|
@@ -41,7 +37,7 @@ pip install apache-airflow-client | |
|
|
||
| Or install via [Setuptools](http://pypi.python.org/pypi/setuptools). | ||
|
|
||
| ```sh | ||
| ```shell | ||
| git clone [email protected]:apache/airflow-client-python.git | ||
| cd airflow-client-python | ||
| python setup.py install --user | ||
|
|
@@ -53,40 +49,116 @@ Then import the package: | |
| import airflow_client.client | ||
| ``` | ||
|
|
||
| ## Changelog | ||
|
|
||
| See [CHANGELOG.md](https://github.com/apache/airflow-client-python/blob/main/CHANGELOG.md) for keeping | ||
| track on what has changed in the client. | ||
|
|
||
|
|
||
| ## Getting Started | ||
|
|
||
| Please follow the [installation procedure](#installation--usage) and then run the following: | ||
| Please follow the [installation procedure](#installation--usage) and then run the following | ||
| example python script: | ||
|
|
||
| ```python | ||
| import airflow_client.client | ||
| from pprint import pprint | ||
| from airflow_client.client.api import config_api | ||
| import uuid | ||
| import airflow_client.client | ||
| try: | ||
| # If you have rich installed, you will have nice colored output of the API responses | ||
| from rich import print | ||
| except ImportError: | ||
| print("Output will not be colored. Please install rich to get colored output: `pip install rich`") | ||
| pass | ||
| from airflow_client.client.api import config_api, dag_api, dag_run_api | ||
| from airflow_client.client.model.dag_run import DAGRun | ||
| # The client must use the authentication and authorization parameters | ||
| # in accordance with the API server security policy. | ||
| # Examples for each auth method are provided below, use the example that | ||
| # satisfies your auth use case. | ||
| # | ||
| # In case of the basic authentication below. Make sure: | ||
| # - Airflow is configured with the basic_auth as backend: | ||
| # auth_backend = airflow.api.auth.backend.basic_auth | ||
| # - Make sure that the client has been generated with securitySchema Basic. | ||
| # In case of the basic authentication below, make sure that Airflow is | ||
| # configured also with the basic_auth as backend additionally to regular session backend needed | ||
| # by the UI. In the `[api]` section of your `airflow.cfg` set: | ||
| # | ||
| # auth_backend = airflow.api.auth.backend.session,airflow.api.auth.backend.basic_auth | ||
| # | ||
| # Make sure that your user/name are configured properly - using the user/password that has admin | ||
| # privileges in Airflow | ||
| # Configure HTTP basic authorization: Basic | ||
| configuration = airflow_client.client.Configuration( | ||
| host="http://localhost/api/v1", | ||
| host="http://localhost:8080/api/v1", | ||
| username='admin', | ||
| password='admin' | ||
| ) | ||
| # Make sure in the [core] section, the `load_examples` config is set to True in your airflow.cfg | ||
| # or AIRFLOW__CORE__LOAD_EXAMPLES environment variable set to True | ||
| DAG_ID = "example_bash_operator" | ||
| # Enter a context with an instance of the API client | ||
| with airflow_client.client.ApiClient(configuration) as api_client: | ||
| # Create an instance of the API class | ||
| api_instance = config_api.ConfigApi(api_client) | ||
| errors = False | ||
| print('[blue]Getting DAG list') | ||
| dag_api_instance = dag_api.DAGApi(api_client) | ||
| try: | ||
| api_response = dag_api_instance.get_dags() | ||
| print(api_response) | ||
| except airflow_client.client.OpenApiException as e: | ||
| print("[red]Exception when calling DagAPI->get_dags: %s\n" % e) | ||
| errors = True | ||
| else: | ||
| print('[green]Getting DAG list successful') | ||
| print('[blue]Getting Tasks for a DAG') | ||
| try: | ||
| api_response = dag_api_instance.get_tasks(DAG_ID) | ||
| print(api_response) | ||
| except airflow_client.client.exceptions.OpenApiException as e: | ||
| print("[red]Exception when calling DagAPI->get_tasks: %s\n" % e) | ||
| errors = True | ||
| else: | ||
| print('[green]Getting Tasks successful') | ||
| print('[blue]Triggering a DAG run') | ||
| dag_run_api_instance = dag_run_api.DAGRunApi(api_client) | ||
| try: | ||
| # Create a DAGRun object (no dag_id should be specified because it is read-only property of DAGRun) | ||
| # dag_run id is generated randomly to allow multiple executions of the script | ||
| dag_run = DAGRun( | ||
| dag_run_id='some_test_run_' + uuid.uuid4().hex, | ||
| ) | ||
| api_response = dag_run_api_instance.post_dag_run(DAG_ID, dag_run) | ||
| print(api_response) | ||
| except airflow_client.client.exceptions.OpenApiException as e: | ||
| print("[red]Exception when calling DAGRunAPI->post_dag_run: %s\n" % e) | ||
| errors = True | ||
| else: | ||
| print('[green]Posting DAG Run successful') | ||
| # Get current configuration. Note, this is disabled by default with most installation. | ||
| # You need to set `expose_config = True` in Airflow configuration in order to retrieve configuration. | ||
| conf_api_instance = config_api.ConfigApi(api_client) | ||
| try: | ||
| # Get current configuration | ||
| api_response = api_instance.get_config() | ||
| pprint(api_response) | ||
| except airflow_client.client.ApiException as e: | ||
| print("Exception when calling ConfigApi->get_config: %s\n" % e) | ||
| api_response = conf_api_instance.get_config() | ||
| print(api_response) | ||
| except airflow_client.client.OpenApiException as e: | ||
| print("[red]Exception when calling ConfigApi->get_config: %s\n" % e) | ||
| errors = True | ||
| else: | ||
| print('[green]Config retrieved successfully') | ||
| if errors: | ||
| print ('\n[red]There were errors while running the script - see above for details') | ||
| else: | ||
| print ('\n[green]Everything went well') | ||
| ``` | ||
| See [README](./airflow_client/README.md#documentation-for-api-endpoints) for full client API documentation. | ||
| See [README](https://github.com/apache/airflow-client-python/blob/main/README.md#documentation-for-api-endpoints) | ||
| for full client API documentation. | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| #!/usr/bin/env python3 | ||
| # Licensed to the Apache Software Foundation (ASF) under one | ||
| # or more contributor license agreements. See the NOTICE file | ||
| # distributed with this work for additional information | ||
| # regarding copyright ownership. The ASF licenses this file | ||
| # to you under the Apache License, Version 2.0 (the | ||
| # "License"); you may not use this file except in compliance | ||
| # with the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, | ||
| # software distributed under the License is distributed on an | ||
| # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| # KIND, either express or implied. See the License for the | ||
| # specific language governing permissions and limitations | ||
| # under the License. | ||
|
|
||
| from pathlib import Path | ||
|
|
||
| BASE_DIR = Path(__file__).resolve().parent.parent | ||
|
|
||
| if __name__ == '__main__': | ||
| example = (BASE_DIR / "dev" / "test_python_client.py").read_text().splitlines() | ||
| example = example[example.index("# under the License.")+1:] # Remove license header | ||
| readme_lines = (BASE_DIR / "README.md").read_text().splitlines() | ||
| result_lines = [] | ||
| skip_lines = False | ||
| changes_made = False | ||
| for line in readme_lines: | ||
| if line.strip().startswith("example python script:"): | ||
| result_lines.append(line) | ||
| result_lines.append("") | ||
| result_lines.append("```python") | ||
| result_lines.extend(example) | ||
| result_lines.append("```") | ||
| skip_lines = True | ||
| changes_made = True | ||
| else: | ||
| if not skip_lines: | ||
| result_lines.append(line) | ||
| if line.strip() == "```": | ||
| skip_lines = False | ||
| (BASE_DIR / "README.md").write_text("\n".join(result_lines)) | ||
| if not changes_made: | ||
| raise Exception("Could not find example python script in README.md to replace. " | ||
| "Please make sure `example python script:` line is in README.md") |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters