Docs: review and update connector form scratch documentation and rela…

…ted guides (airbytehq#33822)

Co-authored-by: Marcos Marx <[email protected]>

.prettierrc

@@ -0,0 +1,11 @@
+{
+  "overrides": [
+    {
+      "files": "*.md",
+      "options": {
+        "printWidth": 100,
+        "proseWrap": "always"
+      }
+    }
+  ]
+}

airbyte-cdk/python/README.md

@@ -1,10 +1,9 @@
 # Connector Development Kit \(Python\)
 
-The Airbyte Python CDK is a framework for rapidly developing production-grade Airbyte connectors. The CDK currently offers helpers specific for creating Airbyte source connectors for:
+The Airbyte Python CDK is a framework for rapidly developing production-grade Airbyte connectors.The CDK currently offers helpers specific for creating Airbyte source connectors for:
 
-* HTTP APIs \(REST APIs, GraphQL, etc..\)
-* Singer Taps
-* Generic Python sources \(anything not covered by the above\)
+- HTTP APIs \(REST APIs, GraphQL, etc..\)
+- Generic Python sources \(anything not covered by the above\)
 
 The CDK provides an improved developer experience by providing basic implementation structure and abstracting away low-level glue boilerplate.
 
@@ -14,14 +13,14 @@ This document is a general introduction to the CDK. Readers should have basic fa
 
 Generate an empty connector using the code generator. First clone the Airbyte repository then from the repository root run
 
-```text
+```bash
 cd airbyte-integrations/connector-templates/generator
 ./generate.sh
 ```
 
 then follow the interactive prompt. Next, find all `TODO`s in the generated project directory -- they're accompanied by lots of comments explaining what you'll need to do in order to implement your connector. Upon completing all TODOs properly, you should have a functioning connector.
 
-Additionally, you can follow [this tutorial](https://docs.airbyte.io/connector-development/tutorials/cdk-tutorial-python-http) for a complete walkthrough of creating an HTTP connector using the Airbyte CDK.
+Additionally, you can follow [this tutorial](https://docs.airbyte.com/connector-development/cdk-python/) for a complete walkthrough of creating an HTTP connector using the Airbyte CDK.
 
 ### Concepts & Documentation
 
@@ -31,47 +30,44 @@ See the [concepts docs](docs/concepts/) for a tour through what the API offers.
 
 **HTTP Connectors**:
 
-* [Exchangerates API](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-exchange-rates/source_exchange_rates/source.py)
-* [Stripe](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-stripe/source_stripe/source.py)
-* [Slack](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-slack/source_slack/source.py)
-
-**Singer connectors**:
-
-* [Salesforce](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-salesforce-singer/source_salesforce_singer/source.py)
-* [Github](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-github-singer/source_github_singer/source.py)
+- [Stripe](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-stripe/source_stripe/source.py)
+- [Slack](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-slack/source_slack/source.py)
 
 **Simple Python connectors using the barebones `Source` abstraction**:
 
-* [Google Sheets](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-google-sheets/google_sheets_source/google_sheets_source.py)
-* [Mailchimp](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-mailchimp/source_mailchimp/source.py)
+- [Google Sheets](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-google-sheets/google_sheets_source/google_sheets_source.py)
+- [Mailchimp](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-mailchimp/source_mailchimp/source.py)
 
 ## Contributing
 
 ### First time setup
 
-We assume `python` points to python >=3.8.
+We assume `python` points to Python 3.9 or higher.
 
 Setup a virtual env:
 
-```text
+```bash
 python -m venv .venv
 source .venv/bin/activate
 pip install -e ".[dev]" # [dev] installs development-only dependencies
 ```
 
 #### Iteration
 
-* Iterate on the code locally
-* Run tests via `python -m pytest -s unit_tests`
-* Perform static type checks using `mypy airbyte_cdk`. `MyPy` configuration is in `mypy.ini`.
- * Run `mypy <files to check>` to only check specific files. This is useful as the CDK still contains code that is not compliant.
-* The `type_check_and_test.sh` script bundles both type checking and testing in one convenient command. Feel free to use it!
+- Iterate on the code locally
+- Run tests via `python -m pytest -s unit_tests`
+- Perform static type checks using `mypy airbyte_cdk`. `MyPy` configuration is in `mypy.ini`.
+- Run `mypy <files to check>` to only check specific files. This is useful as the CDK still contains code that is not compliant.
+- The `type_check_and_test.sh` script bundles both type checking and testing in one convenient command. Feel free to use it!
 
 ##### Autogenerated files
+
 If the iteration you are working on includes changes to the models, you might want to regenerate them. In order to do that, you can run:
-```commandline
+
+```bash
 ./gradlew :airbyte-cdk:python:format
 ```
+
 This will generate the files based on the schemas, add the license information and format the code. If you want to only do the former and rely on
 pre-commit to the others, you can run the appropriate generation command i.e. `./gradlew generateComponentManifestClassFiles`.
 
@@ -82,14 +78,16 @@ All tests are located in the `unit_tests` directory. Run `python -m pytest --cov
 #### Building and testing a connector with your local CDK
 
 When developing a new feature in the CDK, you may find it helpful to run a connector that uses that new feature. You can test this in one of two ways:
-* Running a connector locally
-* Building and running a source via Docker
+
+- Running a connector locally
+- Building and running a source via Docker
 
 ##### Installing your local CDK into a local Python connector
 
 In order to get a local Python connector running your local CDK, do the following.
 
 First, make sure you have your connector's virtual environment active:
+
 ```bash
 # from the `airbyte/airbyte-integrations/connectors/<connector-directory>` directory
 source .venv/bin/activate
@@ -99,6 +97,7 @@ pip install -e .
 ```
 
 Then, navigate to the CDK and install it in editable mode:
+
 ```bash
 cd ../../../airbyte-cdk/python
 pip install -e .
@@ -107,28 +106,35 @@ pip install -e .
 You should see that `pip` has uninstalled the version of `airbyte-cdk` defined by your connector's `setup.py` and installed your local CDK. Any changes you make will be immediately reflected in your editor, so long as your editor's interpreter is set to your connector's virtual environment.
 
 ##### Building a Python connector in Docker with your local CDK installed
+
 _Pre-requisite: Install the [`airbyte-ci` CLI](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md)_
 
 You can build your connector image with the local CDK using
+
 ```bash
 # from the airbytehq/airbyte base directory
 airbyte-ci connectors --use-local-cdk --name=<CONNECTOR> build
 ```
+
 Note that the local CDK is injected at build time, so if you make changes, you will have to run the build command again to see them reflected.
 
 ##### Running Connector Acceptance Tests for a single connector in Docker with your local CDK installed
+
 _Pre-requisite: Install the [`airbyte-ci` CLI](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md)_
 
 To run acceptance tests for a single connectors using the local CDK, from the connector directory, run
+
 ```bash
 airbyte-ci connectors --use-local-cdk --name=<CONNECTOR> test
 ```
 
 #### When you don't have access to the API
+
 There can be some time where you do not have access to the API (either because you don't have the credentials, network access, etc...) You will probably still want to do end-to-end testing at least once. In order to do so, you can emulate the server you would be reaching using a server stubbing tool.
 
 For example, using [mockserver](https://www.mock-server.com/), you can set up an expectation file like this:
-```
+
+```json
 {
   "httpRequest": {
     "method": "GET",
@@ -141,19 +147,14 @@ For example, using [mockserver](https://www.mock-server.com/), you can set up an
 ```
 
 Assuming this file has been created at `secrets/mock_server_config/expectations.json`, running the following command will allow to match any requests on path `/data` to return the response defined in the expectation file:
-`docker run -d --rm -v $(pwd)/secrets/mock_server_config:/config -p 8113:8113 --env MOCKSERVER_LOG_LEVEL=TRACE --env MOCKSERVER_SERVER_PORT=8113 --env MOCKSERVER_WATCH_INITIALIZATION_JSON=true --env MOCKSERVER_PERSISTED_EXPECTATIONS_PATH=/config/expectations.json --env MOCKSERVER_INITIALIZATION_JSON_PATH=/config/expectations.json mockserver/mockserver:5.15.0`
+
+```bash
+docker run -d --rm -v $(pwd)/secrets/mock_server_config:/config -p 8113:8113 --env MOCKSERVER_LOG_LEVEL=TRACE --env MOCKSERVER_SERVER_PORT=8113 --env MOCKSERVER_WATCH_INITIALIZATION_JSON=true --env MOCKSERVER_PERSISTED_EXPECTATIONS_PATH=/config/expectations.json --env MOCKSERVER_INITIALIZATION_JSON_PATH=/config/expectations.json mockserver/mockserver:5.15.0
+```
 
 HTTP requests to `localhost:8113/data` should now return the body defined in the expectations file. To test this, the implementer either has to change the code which defines the base URL for Python source or update the `url_base` from low-code. With the Connector Builder running in docker, you will have to use domain `host.docker.internal` instead of `localhost` as the requests are executed within docker.
 
 #### Publishing a new version to PyPi
 
 1. Open a PR
 2. Once it is approved and **merged**, an Airbyte member must run the `Publish CDK Manually` workflow from master using `release-type=major|manor|patch` and setting the changelog message.
-
-## Coming Soon
-
-* Full OAuth 2.0 support \(including refresh token issuing flow via UI or CLI\)
-* Airbyte Java HTTP CDK
-* CDK for Async HTTP endpoints \(request-poll-wait style endpoints\)
-* CDK for other protocols
-* Don't see a feature you need? [Create an issue and let us know how we can help!](https://github.com/airbytehq/airbyte/issues/new?assignees=&labels=type%2Fenhancement&template=feature-request.md&title=)

airbyte-integrations/connector-templates/generator/plopfile.js

@@ -24,7 +24,6 @@ ${additionalMessage || ""}
 
 module.exports = function (plop) {
   const docRoot = '../../../docs/integrations';
-  const definitionRoot = '../../../airbyte-config-oss/init-oss/src/main/resources';
 
   const connectorAcceptanceTestFilesInputRoot = '../connector_acceptance_test_files';
 

docs/connector-development/cdk-python/README.md

@@ -1,44 +1,54 @@
 # Connector Development Kit
 
 :::info
-Over the next few months, the project will only accept connector contributions that are made using the [Low-Code CDK](https://docs.airbyte.com/connector-development/config-based/low-code-cdk-overview) or the [Connector Builder](https://docs.airbyte.com/connector-development/connector-builder-ui/overview).
+Over the next few months, the project will only accept connector contributions that are made using the
+[Low-Code CDK](https://docs.airbyte.com/connector-development/config-based/low-code-cdk-overview) or the
+[Connector Builder](https://docs.airbyte.com/connector-development/connector-builder-ui/overview).
 
-Contributions made with the Python CDK will be closed, but we will inquire to understand why it wasn't done with Low-Code/Connector Builder so we can address missing features.
-This decision is aimed at improving maintenance and providing a larger catalog with high-quality connectors.
+New pull requests made with the Python CDK will be closed, but we will inquire to understand why it wasn't done with
+Low-Code/Connector Builder so we can address missing features. This decision is aimed at improving maintenance and
+providing a larger catalog with high-quality connectors.
 
 You can continue to use the Python CDK to build connectors to help your company or projects.
 :::
 
 :::info
-Developer updates will be announced via our #help-connector-development Slack channel. If you are using the CDK, please join to stay up to date on changes and issues.
+Developer updates will be announced via
+[#help-connector-development](https://airbytehq.slack.com/archives/C027KKE4BCZ) Slack channel. If you are using the
+CDK, please join to stay up to date on changes and issues.
 :::
 
 :::info
-This section is for the Python CDK. See our [community-maintained CDKs section](../README.md#community-maintained-cdks)
-if you want to write connectors in other languages.
+This section is for the Python CDK. See our
+[community-maintained CDKs section](../README.md#community-maintained-cdks) if you want to write connectors in other
+languages.
 :::
-The Airbyte Python CDK is a framework for rapidly developing production-grade Airbyte connectors. The CDK currently offers helpers specific for creating Airbyte source connectors for:
+
+The Airbyte Python CDK is a framework for rapidly developing production-grade Airbyte connectors. The CDK currently
+offers helpers specific for creating Airbyte source connectors for:
 
 - HTTP APIs \(REST APIs, GraphQL, etc..\)
 - Generic Python sources \(anything not covered by the above\)
-- Singer Taps (Note: The CDK supports building Singer taps but Airbyte no longer access contributions of this type)
-
-The CDK provides an improved developer experience by providing basic implementation structure and abstracting away low-level glue boilerplate.
 
-This document is a general introduction to the CDK. Readers should have basic familiarity with the [Airbyte Specification](https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/) before proceeding.
+This document is a general introduction to the CDK. Readers should have basic familiarity with the
+[Airbyte Specification](https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/) before proceeding.
 
-If you have any issues with troubleshooting or want to learn more about the CDK from the Airbyte team, head to [the Connector Development section of our Airbyte Forum](https://github.com/airbytehq/airbyte/discussions) to inquire further!
+If you have any issues with troubleshooting or want to learn more about the CDK from the Airbyte team, head to 
+[the Connector Development section of our Airbyte Forum](https://github.com/airbytehq/airbyte/discussions) to
+inquire further!
 
 ## Getting Started
 
-Generate an empty connector using the code generator. First clone the Airbyte repository then from the repository root run
+Generate an empty connector using the code generator. First clone the Airbyte repository, then from the repository
+root run
 
-```text
+```bash
 cd airbyte-integrations/connector-templates/generator
 ./generate.sh
 ```
 
-then follow the interactive prompt. Next, find all `TODO`s in the generated project directory -- they're accompanied by lots of comments explaining what you'll need to do in order to implement your connector. Upon completing all TODOs properly, you should have a functioning connector.
+Next, find all `TODO`s in the generated project directory. They're accompanied by comments explaining what you'll
+need to do in order to implement your connector. Upon completing all TODOs properly, you should have a functioning connector.
 
 Additionally, you can follow [this tutorial](../tutorials/cdk-tutorial-python-http/getting-started.md) for a complete walkthrough of creating an HTTP connector using the Airbyte CDK.
 
@@ -68,7 +78,6 @@ You can find a complete tutorial for implementing an HTTP source connector in [t
 
 **HTTP Connectors**:
 
-- [Exchangerates API](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-exchange-rates/source_exchange_rates/source.py)
 - [Stripe](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-stripe/source_stripe/source.py)
 - [Slack](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-slack/source_slack/source.py)
 
@@ -81,11 +90,11 @@ You can find a complete tutorial for implementing an HTTP source connector in [t
 
 ### First time setup
 
-We assume `python` points to python >=3.9.
+We assume `python` points to Python 3.9 or higher.
 
 Setup a virtual env:
 
-```text
+```bash
 python -m venv .venv
 source .venv/bin/activate
 pip install -e ".[tests]" # [tests] installs test-only dependencies
@@ -102,7 +111,7 @@ pip install -e ".[tests]" # [tests] installs test-only dependencies
 
 While developing your connector, you can print detailed debug information during a sync by specifying the `--debug` flag. This allows you to get a better picture of what is happening during each step of your sync.
 
-```text
+```bash
 python main.py read --config secrets/config.json --catalog sample_files/configured_catalog.json --debug
 ```
 
@@ -120,11 +129,3 @@ All tests are located in the `unit_tests` directory. Run `pytest --cov=airbyte_c
 
 1. Open a PR
 2. Once it is approved and merge, an Airbyte member must run the `Publish CDK Manually` workflow using `release-type=major|manor|patch` and setting the changelog message.
-
-## Coming Soon
-
-- Full OAuth 2.0 support \(including refresh token issuing flow via UI or CLI\)
-- Airbyte Java HTTP CDK
-- CDK for Async HTTP endpoints \(request-poll-wait style endpoints\)
-- CDK for other protocols
-- Don't see a feature you need? [Create an issue and let us know how we can help!](https://github.com/airbytehq/airbyte/issues/new?assignees=&labels=type%2Fenhancement&template=feature-request.md&title=)

docs/connector-development/tutorials/adding-incremental-sync.md

@@ -2,7 +2,7 @@
 
 ## Overview
 
-This tutorial will assume that you already have a working source. If you do not, feel free to refer to the [Building a Toy Connector](building-a-python-source.md) tutorial. This tutorial will build directly off the example from that article. We will also assume that you have a basic understanding of how Airbyte's Incremental-Append replication strategy works. We have a brief explanation of it [here](/using-airbyte/core-concepts/sync-modes/incremental-append.md).
+This tutorial will assume that you already have a working source. If you do not, feel free to refer to the [Building a Toy Connector](build-a-connector-the-hard-way.md) tutorial. This tutorial will build directly off the example from that article. We will also assume that you have a basic understanding of how Airbyte's Incremental-Append replication strategy works. We have a brief explanation of it [here](/using-airbyte/core-concepts/sync-modes/incremental-append.md).
 
 ## Update Catalog in `discover`