testing: updated docker-compose.yml (MINOR) (#5245)

* testing: updated docker-compose.yml Commit adds utility `up` script to bring up docker images, wait for ksql to be responsive, and then load the cli. `down` to bring everything down again. `.env` file added with environment vars for setting docker version. updated some outdated links too. Co-authored-by: Andy Coates <[email protected]>
confluentinc · May 4, 2020 · b252fae · b252fae
1 parent 4c351d2
commit b252fae
Show file tree

Hide file tree

Showing 6 changed files with 63 additions and 16 deletions.
diff --git a/.env b/.env
@@ -0,0 +1,13 @@
+# Sets environment variables used in docker-compose.yml
+
+# Set to specific version needed, e.g. 5.5.0
+CP_VERSION=latest
+
+# Set to the specific version needed, e.g. 0.9.1, or local.build
+KSQL_VERSION=latest
+
+KSQL_IMAGE_BASE=
+
+# For locally built images the following settings should be uncommented
+#KSQL_VERSION=local.build
+#KSQL_IMAGE_BASE=placeholder/
diff --git a/docker-compose.yml b/docker-compose.yml
@@ -15,22 +15,22 @@
 #
 # If you wish to run with locally built ksqlDB docker images then:
 #
-# 1. Follow the steps in https://github.com/confluentinc/ksql/blob/master/ksql-docker/README.md
+# 1. Follow the steps in https://github.com/confluentinc/ksql/blob/master/ksqldb-docker/README.md
 # to build a ksqlDB docker image with local changes.
 #
-# 2. Update this file below replacing all references to "confluentinc/ksqldb-server:latest" with your image, e.g. "placeholder/confluentinc/ksql-docker:local.build"
+# 2. Update .env file to use your local images by setting KSQL_IMAGE_BASE=placeholder/ and KSQL_VERSION=local.build.
 
 ---
 version: '2'
 services:
   zookeeper:
-    image: confluentinc/cp-zookeeper:latest
+    image: confluentinc/cp-zookeeper:${CP_VERSION}
     environment:
       ZOOKEEPER_CLIENT_PORT: 32181
       ZOOKEEPER_TICK_TIME: 2000
 
   kafka:
-    image: confluentinc/cp-enterprise-kafka:latest
+    image: confluentinc/cp-enterprise-kafka:${CP_VERSION}
     ports:
       - "29092:29092"
     depends_on:
@@ -47,7 +47,7 @@ services:
       KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 100
 
   schema-registry:
-    image: confluentinc/cp-schema-registry:latest
+    image: confluentinc/cp-schema-registry:${CP_VERSION}
     depends_on:
       - zookeeper
       - kafka
@@ -56,7 +56,7 @@ services:
       SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: zookeeper:32181
 
   primary-ksqldb-server:
-    image: confluentinc/ksqldb-server:latest
+    image: ${KSQL_IMAGE_BASE}confluentinc/ksqldb-server:${KSQL_VERSION}
     hostname: primary-ksqldb-server
     container_name: primary-ksqldb-server
     depends_on:
@@ -72,7 +72,7 @@ services:
       KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
 
   additional-ksqldb-server:
-    image: confluentinc/ksqldb-server:latest
+    image: ${KSQL_IMAGE_BASE}confluentinc/ksqldb-server:${KSQL_VERSION}
     hostname: additional-ksqldb-server
     container_name: additional-ksqldb-server
     depends_on:
@@ -87,7 +87,7 @@ services:
   # Access the cli by running:
   # > docker-compose exec ksqldb-cli  ksql http://primary-ksqldb-server:8088
   ksqldb-cli:
-    image: confluentinc/ksqldb-cli:latest
+    image: ${KSQL_IMAGE_BASE}confluentinc/ksqldb-cli:${KSQL_VERSION}
     container_name: ksqldb-cli
     depends_on:
       - primary-ksqldb-server

diff --git a/docs/concepts/ksqldb-architecture.md b/docs/concepts/ksqldb-architecture.md
@@ -34,20 +34,20 @@ builds and runs the application on available ksqlDB servers. Each ksqlDB
 Server instance runs a ksqlDB engine. Under the hood, the engine
 parses your SQL statements and builds corresponding Kafka Streams
 topologies. The ksqlDB engine is implemented in the
-[KsqlEngine.java](https://github.com/confluentinc/ksql/blob/master/ksql-engine/src/main/java/io/confluent/ksql/engine/KsqlEngine.java)
+[KsqlEngine.java](https://github.com/confluentinc/ksql/blob/master/ksqldb-engine/src/main/java/io/confluent/ksql/engine/KsqlEngine.java)
 class.
 - ksqlDB CLI: The ksqlDB CLI provides a console with a command-line interface
 for the ksqlDB engine. Use the ksqlDB CLI to interact with ksqlDB Server
 instances and develop your streaming applications. The ksqlDB CLI is
 designed to be familiar to users of MySQL, Postgres, and similar
 applications. The ksqlDB CLI is implemented in the
-[io.confluent.ksql.cli](https://github.com/confluentinc/ksql/tree/master/ksql-cli/src/main/java/io/confluent/ksql/cli)
+[io.confluent.ksql.cli](https://github.com/confluentinc/ksql/tree/master/ksqldb-cli/src/main/java/io/confluent/ksql/cli)
 package.
 - REST Interface: The REST server interface enables communicating with the
 ksqlDB engine from the CLI, {{ site.c3 }}, or from any other REST client. For
 more information, see [ksqlDB REST API Reference](../developer-guide/api.md).
 The ksqlDB REST server is implemented in the
-[KsqlRestApplication.java](https://github.com/confluentinc/ksql/blob/master/ksql-rest-app/src/main/java/io/confluent/ksql/rest/server/KsqlRestApplication.java)
+[KsqlRestApplication.java](https://github.com/confluentinc/ksql/blob/master/ksqldb-rest-app/src/main/java/io/confluent/ksql/rest/server/KsqlRestApplication.java)
 class.
 
 When you deploy your ksqlDB application, it runs on ksqlDB Server instances
@@ -301,7 +301,7 @@ statement might resemble:
 |                        |                                                                                                                                                         |     |
 
 The ksqlDB metastore is implemented in the
-[io.confluent.ksql.metastore](https://github.com/confluentinc/ksql/tree/master/ksql-metastore/src/main/java/io/confluent/ksql/metastore)
+[io.confluent.ksql.metastore](https://github.com/confluentinc/ksql/tree/master/ksqldb-metastore/src/main/java/io/confluent/ksql/metastore)
 package.
 
 ### Express Your Application as a SQL Statement
@@ -334,7 +334,7 @@ syntax tree (AST). The ksqlDB engine uses the AST to plan the query.
 
 The SQL statement parser is based on [ANTLR](https://www.antlr.org/)
 and is implemented in the
-[io.confluent.ksql.parser](https://github.com/confluentinc/ksql/tree/master/ksql-parser/src/main)
+[io.confluent.ksql.parser](https://github.com/confluentinc/ksql/tree/master/ksqldb-parser/src/main)
 package.
 
 ### ksqlDB Creates the Logical Plan
@@ -362,13 +362,13 @@ The generated code is based on the ksqlDB classes, `SchemaKStream` and
 `SchemaKTable`:
 
 -   A ksqlDB stream is rendered as a
-    [SchemaKStream](https://github.com/confluentinc/ksql/blob/master/ksql-engine/src/main/java/io/confluent/ksql/structured/SchemaKStream.java)
+    [SchemaKStream](https://github.com/confluentinc/ksql/blob/master/ksqldb-engine/src/main/java/io/confluent/ksql/structured/SchemaKStream.java)
     instance, which is a
     [KStream](https://docs.confluent.io/current/streams/javadocs/org/apache/kafka/streams/kstream/KStream.html)
     with a
     [Schema](https://kafka.apache.org/20/javadoc/org/apache/kafka/connect/data/Schema.html).
 -   A ksqlDB table is rendered as a
-    [SchemaKTable](https://github.com/confluentinc/ksql/blob/master/ksql-engine/src/main/java/io/confluent/ksql/structured/SchemaKTable.java)
+    [SchemaKTable](https://github.com/confluentinc/ksql/blob/master/ksqldb-engine/src/main/java/io/confluent/ksql/structured/SchemaKTable.java)
     instance, which is a
     [KTable](https://docs.confluent.io/current/streams/javadocs/org/apache/kafka/streams/kstream/KTable.html)
     with a

diff --git a/down b/down
@@ -0,0 +1,7 @@
+#!/bin/bash
+
+# exit when any command fails
+set -e
+
+echo -e "## Bringing down containers:\n"
+docker-compose down -v
diff --git a/ksqldb-docker/README.md b/ksqldb-docker/README.md
@@ -21,4 +21,8 @@ To build a new image with local changes:
     ```
     > docker image ls | grep local.build
     ```
-    You should see the new image listed.
+    You should see the new image listed. For example:
+
+    ```
+    placeholder/confluentinc/ksqldb-docker       local.build   94210cd14384   About an hour ago   716MB
+    ```
diff --git a/up b/up
@@ -0,0 +1,23 @@
+#!/bin/bash
+
+# exit when any command fails
+set -e
+
+echo -e "## Bringing up containers:\n"
+docker-compose up -d
+
+echo -e "\n## Waiting for ksqlDB to start:"
+until curl -s --fail -X "POST" "http://localhost:8088/ksql" \
+  -H "Content-Type: application/json; charset=utf-8" \
+  -d '{"ksql":"SHOW STREAMS;"}'
+do
+   sleep 5
+   echo -n "."
+done
+
+echo -e "\n## Starting ksqlDB CLI:\n"
+docker exec -it ksqldb-cli ksql http://primary-ksqldb-server:8088
+
+
+
+