Add examples and update constraints for Kafka API #13492
Conversation
github-actions
bot
added
the
documentation
|
⚪ |
|
⚪ |
❌ Documentation buildRevision build failed Build logsErrors (35)❌ /en/reference/kafka-api/examples.md: 201: YFM001/inline-code-length Inline code length [Expected: 140; Actual: 212] [Context: "Unexpected error in join group response: This most likely occurs because of a request being malformed by the client library or the message was sent to an incompatible broker. See the broker logs for more details."] ❌ /en/reference/kafka-api/examples.md: 203: MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1] ❌ /en/reference/kafka-api/examples.md: 36: MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1] ❌ /en/reference/kafka-api/examples.md: 385: MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "- ❌ /en/reference/kafka-api/examples.md: 390: MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "- <path_to_database> is the pa..."] ❌ /en/reference/kafka-api/examples.md: 409: MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1] ❌ /en/reference/kafka-api/examples.md: 43: MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "- absence of support for the [..."] ❌ /en/reference/kafka-api/examples.md: 463: MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "1. Create a user. [How to do t..."] ❌ /en/reference/kafka-api/examples.md: 464: MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "2. Connect to the Kafka API as..."] ❌ /en/reference/kafka-api/examples.md: 465: MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "- YDB is running locally with ..."] ❌ /en/reference/kafka-api/examples.md: 482: MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1] ❌ /en/reference/kafka-api/examples.md: 48: MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "- ❌ /ru/reference/kafka-api/constraints.md: 12: MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1] ❌ /ru/reference/kafka-api/constraints.md: 13: MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1] ❌ /ru/reference/kafka-api/read-write.md: 201: YFM001/inline-code-length Inline code length [Expected: 140; Actual: 212] [Context: "Unexpected error in join group response: This most likely occurs because of a request being malformed by the client library or the message was sent to an incompatible broker. See the broker logs for more details."] Log was truncated. (20 records) |
…o kafka-api-docs-improvements
|
⚪ |
|
⚪ |
❌ Documentation buildRevision build failed Build logsErrors (35)❌ /en/reference/kafka-api/examples.md: 201: YFM001/inline-code-length Inline code length [Expected: 140; Actual: 212] [Context: "Unexpected error in join group response: This most likely occurs because of a request being malformed by the client library or the message was sent to an incompatible broker. See the broker logs for more details."] ❌ /en/reference/kafka-api/examples.md: 203: MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1] ❌ /en/reference/kafka-api/examples.md: 36: MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1] ❌ /en/reference/kafka-api/examples.md: 385: MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "- ❌ /en/reference/kafka-api/examples.md: 390: MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "- <path_to_database> is the pa..."] ❌ /en/reference/kafka-api/examples.md: 409: MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1] ❌ /en/reference/kafka-api/examples.md: 43: MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "- absence of support for the [..."] ❌ /en/reference/kafka-api/examples.md: 463: MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "1. Create a user. [How to do t..."] ❌ /en/reference/kafka-api/examples.md: 464: MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "2. Connect to the Kafka API as..."] ❌ /en/reference/kafka-api/examples.md: 465: MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "- YDB is running locally with ..."] ❌ /en/reference/kafka-api/examples.md: 482: MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1] ❌ /en/reference/kafka-api/examples.md: 48: MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "- ❌ /ru/reference/kafka-api/constraints.md: 12: MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1] ❌ /ru/reference/kafka-api/constraints.md: 13: MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1] ❌ /ru/reference/kafka-api/read-write.md: 201: YFM001/inline-code-length Inline code length [Expected: 140; Actual: 212] [Context: "Unexpected error in join group response: This most likely occurs because of a request being malformed by the client library or the message was sent to an incompatible broker. See the broker logs for more details."] Log was truncated. (20 records) |
|
|
||
| Аутентификация всегда включена при использовании [Kafka API в Yandex Cloud](https://yandex.cloud/ru/docs/data-streams/kafkaapi/auth) | ||
|
|
||
| Аутентификация по умолчанию выключена при использовании локальной базы (например из Docker образа). |
There was a problem hiding this comment.
Не совсем так. Работа по kafka api возможна с указанием логина\пароля, а так же без аутентификации. Для этого надо указать .... в логине-пароле. При использовании [Kafka API в Yandex Cloud] запросы без аутентификации запрещены. При разворачивании YDB самостоятельно доступны оба варианта. Для включения требования работы только с аутентификацией укажите опцию ....
There was a problem hiding this comment.
Может быть так: "при самостоятельном развертывании YDB аутентификация по умолчанию выключена. Для включения укажите опцию ..."? Мне нравится, что тезисы про cloud и dedicated разнесены по строчкам, чтобы можно было на оба из них обратить внимание.
There was a problem hiding this comment.
Я недавно это место обновлял в основной документации https://ydb.tech/docs/ru/security/authentication#anonymous
| 1. Поддержана только SASL/PLAIN-аутентификация. | ||
| 1. Не поддержаны [компатифицированные топики](https://docs.confluent.io/kafka/design/log_compaction.html). | ||
| В связи с этим поверх Kafka API в YDB Topics не работают Kafka Connect, Schema Registry и Kafka Streams. |
There was a problem hiding this comment.
уточни что schema registry не работает с нами как со стораджем, а писать схематизированные данные и читать их можно.
There was a problem hiding this comment.
Как бы ты это написал? Я читаю текущий вариант как "Приложение Schema registry не работает с YDB"
Добавить что-то типа "При этом писать во внешние Schema Registry, которые уже работают поверх другого Kafka кластера, схему и писать в YDB сообщения возможно"?
| 1. Создайте [базу данных YDB](https://yandex.cloud/ru/docs/ydb/quickstart), если у вас ее еще нет | ||
| 2. Создайте [очередь Yandex Data Streams](https://yandex.cloud/ru/docs/data-streams/quickstart) | ||
| 3. Создайте [сервисный аккаунт](https://yandex.cloud/ru/docs/iam/operations/sa/create), если у вас его еще нет | ||
| и добавьте этому сервисному аккаунту роли ydb.viewer (для чтения данных из потока), ydb.editor (для записи данных в поток) и ydb.kafkaApi.client (для доступа к потоку данных по Kafka API) |
There was a problem hiding this comment.
ydb.kafkaApi.client очень скоро уже не будет нужно, надо не забыть будет удалить, и докатить это по-быстрее.
There was a problem hiding this comment.
А этой мой фикс вот тут удаляет эту проверку?
https://a.yandex-team.ru/arcadia/contrib/ydb/services/persqueue_v1/actors/persqueue_utils.h?rev=r15535601#L20
| - Spark | ||
|
|
||
| Apache Spark при работе с Kafka не использует ничего из существующих ограничений Kafka API в YDB Topics. Благодаря этому | ||
| работа с Kafka API из Spark должна работать в полном объеме. |
There was a problem hiding this comment.
работа должна работать. Благодаря этому использование Spark с Ydb topics возможно в полном объеме.
| - Exactly once работа по Kafka API сейчас не поддержана, так как поддержка транзакций в Kafka API сейчас в разработке; | ||
| - Подписка на топики через паттерн сейчас недоступна; | ||
| - Использование CreateTime сообщения в качестве watermark сейчас недоступно, так как вместо CreateTime используется текущее время вычитки | ||
| (баг будет пофикшен в следующих версиях). |
There was a problem hiding this comment.
баг будет пофикшен - почему сразу баг-то? Не надо это писать. Исправим, уберем.
There was a problem hiding this comment.
Ну так совсем не пишут в документации "баг будет пофикшен"
| @@ -63,6 +259,80 @@ | |||
| producer.close(); | |||
| ``` | |||
|
|
|||
| - Spark | |||
|
|
|||
| Apache Spark при работе с Kafka не использует ничего из существующих ограничений Kafka API в YDB Topics. Благодаря этому | |||
There was a problem hiding this comment.
работа должна работать опять. Так же надо поправить
|
|
||
| {% endlist %} | ||
|
|
||
| #### Примеры с аутентификацией в локальной базе |
There was a problem hiding this comment.
что такое "локальная база"? разве так у нас везде называется? К самостоятельно развернутому YDB? и только в докере? а как подключиться к моему YDB на кластере?
There was a problem hiding this comment.
Хороший поинт. Может быть "примеры аутентификации в самостоятельно развернутом YDB"?
|
⚪ |
❌ Documentation buildRevision build failed Build logsErrors (9)❌ /en/reference/kafka-api/examples.md: 203: YFM001/inline-code-length Inline code length [Expected: 140; Actual: 212] [Context: "Unexpected error in join group response: ❌ /ru/reference/kafka-api/read-write.md: 116: MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1] ❌ /ru/reference/kafka-api/read-write.md: 151: MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1] ❌ /ru/reference/kafka-api/read-write.md: 153: MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1] ❌ /ru/reference/kafka-api/read-write.md: 165: MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1] ❌ /ru/reference/kafka-api/read-write.md: 195: MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1] ❌ /ru/reference/kafka-api/read-write.md: 204: YFM001/inline-code-length Inline code length [Expected: 140; Actual: 212] [Context: "Unexpected error in join group response: This most likely occurs because of a request being malformed by the client library or the message was sent to an incompatible broker. See the broker logs for more details."] ❌ /ru/reference/kafka-api/read-write.md: 60: MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1] ❌ Link is unreachable: ./auth.md in /en/reference/kafka-api/examples.md |
|
⚪ |
❌ Documentation buildRevision build failed Build logsErrors (4)❌ /en/reference/kafka-api/examples.md: 203: YFM001/inline-code-length Inline code length [Expected: 140; Actual: 212] [Context: "Unexpected error in join group response: ❌ /ru/reference/kafka-api/read-write.md: 165: MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1] ❌ /ru/reference/kafka-api/read-write.md: 205: YFM001/inline-code-length Inline code length [Expected: 140; Actual: 212] [Context: "Unexpected error in join group response: This most likely occurs because of a request being malformed ❌ Link is unreachable: ./auth.md in /en/reference/kafka-api/examples.md |
|
⚪ |
|
⚪ |
|
⚪ |
|
⚪ |
❌ Documentation buildRevision build failed Build logsErrors (1)❌ Link is unreachable: ./auth.md in /en/reference/kafka-api/examples.md |
|
⚪ Test history | Ya make output | Test bloat
🟢 |
|
⚪ Test history | Ya make output | Test bloat
🟢 |
✅ Documentation buildRevision built successfully |
✅ Documentation buildRevision built successfully |
|
⚪ |
❌ Documentation buildRevision build failed Build logsErrors (9)❌ Link is unreachable: ../../../_includes/go/auth-static-database-sql.md in /en/recipes/ydb-sdk/auth-static.md ❌ Link is unreachable: ../../../_includes/go/auth-static-database-sql.md in /ru/recipes/ydb-sdk/auth-static.md ❌ Link is unreachable: ../../../_includes/go/auth-static-with-database-sql.md in /en/recipes/ydb-sdk/auth-static.md ❌ Link is unreachable: ../../../_includes/go/auth-static-with-database-sql.md in /ru/recipes/ydb-sdk/auth-static.md ❌ Link is unreachable: ../../../_includes/go/auth-static-with-native.md in /en/recipes/ydb-sdk/auth-static.md ❌ Link is unreachable: ../../../_includes/go/auth-static-with-native.md in /ru/recipes/ydb-sdk/auth-static.md ❌ Skip error: Error: ENOENT: no such file or directory, open '/_includes/go/auth-static-database-sql.md' in /_includes/go/auth-static-database-sql.md ❌ Skip error: Error: ENOENT: no such file or directory, open '/_includes/go/auth-static-with-database-sql.md' in /_includes/go/auth-static-with-database-sql.md ❌ Skip error: Error: ENOENT: no such file or directory, open '/_includes/go/auth-static-with-native.md' in /_includes/go/auth-static-with-native.md |
|
⚪ |
❌ Documentation buildRevision build failed Build logsErrors (10)❌ /en/reference/kafka-api/examples.md: 10: MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "This example shows a code snip..."] ❌ /en/reference/kafka-api/examples.md: 15: MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "1. [Create a topic](../ydb-cli..."] ❌ /en/reference/kafka-api/examples.md: 31: MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "1. Create a [YDB database](htt..."] ❌ /en/reference/kafka-api/examples.md: 33: MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "1. Create a [service account](..."] ❌ /en/reference/kafka-api/examples.md: 35: MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "1. Create an [API key](https:/..."] ❌ /en/reference/kafka-api/examples.md: 8: MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines ❌ /ru/reference/kafka-api/examples.md: 13: MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "1. [Создайте топик](../ydb-cli..."] ❌ /ru/reference/kafka-api/examples.md: 14: MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "1. [Добавьте читателя](../ydb-..."] ❌ /ru/reference/kafka-api/examples.md: 33: MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "- ydb.viewer (для чтения данны..."] ❌ /ru/reference/kafka-api/examples.md: 8: MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines |
|
⚪ |
|
⚪ |
|
⚪ Test history | Ya make output | Test bloat
🟢 |
✅ Documentation buildRevision built successfully |
|
⚪ Test history | Ya make output | Test bloat
🟢 |
Co-authored-by: anton-bobkov <[email protected]>
✅ Documentation buildRevision built successfully |
ydb/docs/en/core/reference/kafka-api/_includes/flink-constraints.md
Outdated
Show resolved
Hide resolved
ydb/docs/en/core/reference/kafka-api/_includes/flink-version-notice.md
Outdated
Show resolved
Hide resolved
ydb/docs/en/core/reference/kafka-api/_includes/kafka-console-utillities-java23-fix.md
Outdated
Show resolved
Hide resolved
ydb/docs/en/core/reference/kafka-api/_includes/spark-constraints.md
Outdated
Show resolved
Hide resolved
ydb/docs/en/core/reference/kafka-api/_includes/spark-version-notice.md
Outdated
Show resolved
Hide resolved
|
|
||
| Sample JDBCSink connector `/etc/kafka-connect-worker/jdbc-sink.properties` configuration file for moving data from a topic to a PostgreSQL table. The [Kafka Connect JDBC Connector](https://github.com/confluentinc/kafka-connect-jdbc) is used. | ||
| See instructions on how to start working with the Kafka API over YDB Topics in Yandex Cloud in the section [above](#how-to-try-kafka-api-in-cloud). |
|
|
||
| {% endlist %} | ||
|
|
||
| #### Authentication examples in on-prem YDB |
There was a problem hiding this comment.
The primary deployment options for self-managed YDB are Ansible and K8s. Let's refer to them instead of generic "on-prem".
There was a problem hiding this comment.
Personally I tried this locally and assume that users will attempt to set up authentication in local deployment before commiting it to the working environment (like Ansible or Kubernetes). That's why I used more generic term "on-prem", as these examples could be used locally/in docker/in k8s/on bare metal etc. Is this ok?
Co-authored-by: Ivan Blinkov <[email protected]>
✅ Documentation buildRevision built successfully |
…kiy/ydb into kafka-api-docs-improvements
✅ Documentation buildRevision built successfully |
|
⚪ Test history | Ya make output | Test bloat
🟢 |
|
⚪ Test history | Ya make output | Test bloat
🟢 |
✅ Documentation buildRevision built successfully |
Changelog entry
...
Changelog category
Additional information
...