Expose test-infra services via Camel JBang #16773

Croway · 2025-01-10T11:13:23Z

This PR adds two commands to Camel JBang that expose test-infra services via CLI

camel infra list - Lists all the available services

➜  camel git:(execute-test-infra-camel-cli) CamelJBang infra list
---
ignite: []
azure-storage-queue:
- "storage-queue"
openldap: []
event-bridge: []
xmpp: []
kinesis: []
redis: []
aws-lambda: []
aws-config: []
lambda: []
elastic-search: []
aws-secrets-manager: []
elasticsearch: []
hazelcast: []
arangodb: []
aws-kinesis: []
chatscript: []
azure:
- "storage-queue"
- "storage-blob"
aws-dynamo-db: []
...

camel infra run $SERVICE - Run any services available in the list

➜  camel git:(execute-test-infra-camel-cli) CamelJBang infra run redis
Starting service redis
---
port: 32771
host: "localhost"
getServiceAddress: "localhost:32771"

To stop the execution press q

In order to do so, after decoupling junit from test-infra, the following was done:

Added a new marker annotation @InfraService, the annotation is used to describe the services (ala google AutoService with more informations related to the test service)
A maven Mojo CamelTestInfraGenerateMetadataMojo.java that gathers all the annotations via jandex index and create a json file containing all the information, service class, aliases, implementation class, and implementation alias
A new test-infra module, camel-test-infra-all that gathers all the test-infra dependencies and execute the Mojo to generate the json file
The actual implementation in Camel JBang that reads the generated JSON file and visualizes it (let's discuss how, right now aliases and implementationAliases are exposed as yaml) or, in case of run, uses the reflection to run the service and print the informations provided by the Service Interface

github-actions · 2025-01-10T11:13:37Z

🌟 Thank you for your contribution to the Apache Camel project! 🌟

🤖 CI automation will test this PR automatically.

🐫 Apache Camel Committers, please review the following items:

First-time contributors require MANUAL approval for the GitHub Actions to run
You can use the command /component-test (camel-)component-name1 (camel-)component-name2.. to request a test from the test bot.
You can label PRs using build-all, build-dependents, skip-tests and test-dependents to fine-tune the checks executed by this PR.
Build and test logs are available in the Summary page. Only Apache Camel committers have access to the summary.
⚠️ Be careful when sharing logs. Review their contents before sharing them publicly.

Croway · 2025-01-10T11:16:50Z

Documentation is missing, I'll work on it, moreover, I'd like to discuss the output of the list and run command, any idea is welcomed, right now, I am just dumping some object to yaml

oscerd

Looks impressive. Thanks for this. Just a missing license found.

components/camel-test/camel-test-junit5/src/main/java/org/apache/camel/test/AvailablePort.java

davsclaus · 2025-01-10T11:49:22Z

It would be good to make the list dump in a table like (camel catalog component) and have a description to go with.
And maybe some of the options like --gav --sort and --filter
Also it can have a --json dump that 3rd party tooling can use.

And maybe we can have since version as well when we add new test-infra modules then the tool knows what version of Camel includes that.

davsclaus · 2025-01-10T11:50:15Z

Also I wonder if the test-infra is executed before camel-catalog, then we can maybe include the json dump in the catalog and include APIs there too. It makes it easier for other tooling to know without the need for the jbang commands.

the json file can be copied into the src/generated/..../test-infra folder (see what else is there already)

davsclaus · 2025-01-10T11:51:38Z

this can be done in new PRs I think a first cut is to make the run command work great. The list/dump/ and show nice is 2nd priority

davsclaus · 2025-01-10T11:55:21Z

...el-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/infra/InfraCommand.java

+
+@CommandLine.Command(name = "infra",
+                     description = "List and Run external services for testing and prototyping")
+public class InfraCommand extends CamelCommand {


In the future this command should not be yaml, but as in the comments more like camel catalog component and we should not have jackson-yaml or other dependencies directly in camel-jbang-core

davsclaus · 2025-01-10T11:56:09Z

...camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/infra/InfraList.java

+        List<InfraCommand.TestInfraService> metadata;
+
+        try (InputStream is
+                = this.getClass().getClassLoader().getResourceAsStream("META-INF/test-infra-metadata.json")) {


In the future we should have this in camel-catalog, and in camel-tooling-model so we can use this API directly instead of jackson

davsclaus · 2025-01-10T11:57:03Z

Before we cut 4.10 LTS it would be good to avoid jackson yaml and make this first class in camel-catalog and camel-tooling-model

orpiske

This is really cool! I can barely wait to see all the cool things we'll be able to do with it.

orpiske · 2025-01-10T12:21:37Z

.../camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/infra/InfraRun.java

+        InfrastructureService actualService = (InfrastructureService) Class.forName(serviceImpl)
+                .getDeclaredConstructor(null)
+                .newInstance(null);


Food for thought: I think it would be interesting if, in the future, we could pass arguments to the service we are starting, so that it would be possible to configure it in different ways (i.e.: think of something like providing a configuration class for a message broker that configures different protocols, auth mechanisms, behaviors, etc).

yeah, that would be nice, so far most of the Services have an empty constructor with a default configuration, but some of them, sftp for example, need a security configuration.

I guess that we can add some information into the annotation, or the Service itself

orpiske · 2025-01-10T12:22:19Z

.../src/main/java/org/apache/camel/test/infra/xmpp/services/XmppLocalContainerInfraService.java

 import org.apache.camel.test.infra.common.LocalPropertyResolver;
 import org.apache.camel.test.infra.common.services.ContainerService;
 import org.apache.camel.test.infra.xmpp.common.XmppProperties;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;

+@InfraService(service = XmppInfraService.class, serviceAlias = { "xmpp" })


We may need to add a note about this in the test infra documentation (if we don't generate it automatically).

orpiske · 2025-01-10T12:24:40Z

...rc/main/java/org/apache/camel/test/infra/ftp/services/embedded/FtpsEmbeddedInfraService.java

 import org.apache.ftpserver.FtpServerFactory;
 import org.apache.ftpserver.listener.ListenerFactory;
 import org.apache.ftpserver.ssl.SslConfigurationFactory;

+@InfraService(service = FtpInfraService.class, serviceAlias = { "ftps" })
 public class FtpsEmbeddedInfraService extends FtpEmbeddedInfraService {


This is probably the only one I am a bit concerned about: by using an embedded service, we'll end up bringing all the dependencies used for that (this, for instance may expose us to a higher incidence of CVEs).

I don't think we must solve this now, but it's something I think we should consider for the future versions.

good point, same for Artemis, but I do not think we have many embedded implementations, most probably it can be refactored in the future to use testcontainers

Croway · 2025-01-10T15:02:07Z

Thanks for your inputs @davsclaus @orpiske @oscerd , in the latest commit I removed the jackson-yaml dependency in favour of asciitable for the list and json serialization for the Service properties, we do not know a priori what's inside a Service object (in some case can be a complex object), this is why a table cannot be used and I am still using json.

But I added a flag --json for the tooling
list output (I'll review some annotations to avoid cases like azure-storage-queue storage-queue)

 ALIAS                   IMPLEMENTATION              
 ignite                                              
 azure-storage-queue     storage-queue               
 openldap                                            
 event-bridge                                        
 kinesis                                             
 xmpp                                                
 redis                                               
 aws-lambda                                          
 aws-config                                          
 lambda                                              
 elastic-search                                      
 aws-secrets-manager                                 
 elasticsearch                                       
 hazelcast                                           
 arangodb                                            
 aws-kinesis                                         
 chatscript                                          
 aws-dynamo-db                                       
 aws-sns                                             
 azure                   storage-queue, storage-blob 
 hashicorp-vault         vault                       
 s3                                                  
 torch-serve                                         
 zookeeper                                           
 etcd3                                               
 ollama                                              
 google                  pub-sub                     
 dynamodb                                            
 aws-cloud-watch                                     
 aws-sqs                                             
 aws-iam                                             
 aws-ec2                                             
 cassandra                                           
 aws-kms                                             
 kafka                   redpanda, strimzi           
 aws-s3                                              
 sftp                                                
 couchdb                                             
 milvus                                              
 pulsar                                              
 hashicorp               vault                       
 cloud-watch                                         
 nats                                                
 qdrant                                              
 hive-mq                 sparkplug                   
 fhir                                                
 smb                                                 
 minio                                               
 dynamo-db                                           
 ec2                                                 
 aws-sts                                             
 couchbase                                           
 solr                                                
 mongodb                                             
 postgres                                            
 google-pub-sub          pub-sub                     
 ftps                                                
 artemis                 amqp, mqtt, persistent      
 hivemq                  sparkplug                   
 ftp                                                 
 microprofile            lra                         
 rocketmq                                            
 rabbitmq                                            
 aws-event-bridge                                    
 infinispan                                          
 mosquitto                                           
 azure-storage-blob      storage-blob                
 microprofile-lra        lra                         
 chat-script

list output json [{"aliasImplementation":"","alias":"ignite"},{"aliasImplementation":"storage-queue","alias":"azure-storage-queue"},{"aliasImplementation":"","alias":"openldap"},{"aliasImplementation":"","alias":"event-bridge"},{"aliasImplementation":"","alias":"kinesis"},{"aliasImplementation":"","alias":"xmpp"},{"aliasImplementation":"","alias":"redis"},{"aliasImplementation":"","alias":"aws-lambda"},{"aliasImplementation":"","alias":"aws-config"},{"aliasImplementation":"","alias":"lambda"},{"aliasImplementation":"","alias":"elastic-search"},{"aliasImplementation":"","alias":"aws-secrets-manager"},{"aliasImplementation":"","alias":"elasticsearch"},{"aliasImplementation":"","alias":"hazelcast"},{"aliasImplementation":"","alias":"arangodb"},{"aliasImplementation":"","alias":"aws-kinesis"},{"aliasImplementation":"","alias":"chatscript"},{"aliasImplementation":"","alias":"aws-dynamo-db"},{"aliasImplementation":"","alias":"aws-sns"},{"aliasImplementation":"storage-queue, storage-blob","alias":"azure"},{"aliasImplementation":"vault","alias":"hashicorp-vault"},{"aliasImplementation":"","alias":"s3"},{"aliasImplementation":"","alias":"torch-serve"},{"aliasImplementation":"","alias":"zookeeper"},{"aliasImplementation":"","alias":"etcd3"},{"aliasImplementation":"","alias":"ollama"},{"aliasImplementation":"pub-sub","alias":"google"},{"aliasImplementation":"","alias":"dynamodb"},{"aliasImplementation":"","alias":"aws-cloud-watch"},{"aliasImplementation":"","alias":"aws-sqs"},{"aliasImplementation":"","alias":"aws-iam"},{"aliasImplementation":"","alias":"aws-ec2"},{"aliasImplementation":"","alias":"cassandra"},{"aliasImplementation":"","alias":"aws-kms"},{"aliasImplementation":"redpanda, strimzi","alias":"kafka"},{"aliasImplementation":"","alias":"aws-s3"},{"aliasImplementation":"","alias":"sftp"},{"aliasImplementation":"","alias":"couchdb"},{"aliasImplementation":"","alias":"milvus"},{"aliasImplementation":"","alias":"pulsar"},{"aliasImplementation":"vault","alias":"hashicorp"},{"aliasImplementation":"","alias":"cloud-watch"},{"aliasImplementation":"","alias":"nats"},{"aliasImplementation":"","alias":"qdrant"},{"aliasImplementation":"sparkplug","alias":"hive-mq"},{"aliasImplementation":"","alias":"fhir"},{"aliasImplementation":"","alias":"smb"},{"aliasImplementation":"","alias":"minio"},{"aliasImplementation":"","alias":"dynamo-db"},{"aliasImplementation":"","alias":"ec2"},{"aliasImplementation":"","alias":"aws-sts"},{"aliasImplementation":"","alias":"couchbase"},{"aliasImplementation":"","alias":"solr"},{"aliasImplementation":"","alias":"mongodb"},{"aliasImplementation":"","alias":"postgres"},{"aliasImplementation":"pub-sub","alias":"google-pub-sub"},{"aliasImplementation":"","alias":"ftps"},{"aliasImplementation":"amqp, mqtt, persistent","alias":"artemis"},{"aliasImplementation":"sparkplug","alias":"hivemq"},{"aliasImplementation":"","alias":"ftp"},{"aliasImplementation":"lra","alias":"microprofile"},{"aliasImplementation":"","alias":"rocketmq"},{"aliasImplementation":"","alias":"rabbitmq"},{"aliasImplementation":"","alias":"aws-event-bridge"},{"aliasImplementation":"","alias":"infinispan"},{"aliasImplementation":"","alias":"mosquitto"},{"aliasImplementation":"storage-blob","alias":"azure-storage-blob"},{"aliasImplementation":"lra","alias":"microprofile-lra"},{"aliasImplementation":"","alias":"chat-script"}]

run command output camel infra run kafka redpanda

Starting service kafka with implementation redpanda
{
 "getBootstrapServers" : "localhost:32783"
}
To stop the execution press q

@davsclaus regarding the camel-catalog, I do not know how easy it is, right now I am using camel-test-infra-all module that contain all the test-infra dependencies, the jandex index and executes the Mojo to generate the json file, probably the json file can be simply copied under /camel/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/test-infra/metadata.json but I do not know if this is a good approach, given that the other generated files are generated by the camel-package-maven-plugin during runtime.

tooling/spi-annotations/src/main/java/org/apache/camel/spi/annotations/InfraService.java

Add non-test-jar dependency for component test-infra Signed-off-by: Tom Cunningham <[email protected]> Revert "Add non-test-jar dependency for component test-infra" This reverts commit 954da18. Add non-test-jar dependency for component test-infra CAMEL-21452 Changes to xmpp/zookeeper CAMEL-21452 Decouple the infrastructure from the testing API CAMEL-21452 Decouple the infrastructure from the testing API aws azure cassandra chatscript consul couchbase couchdb elasticsearch etcd3 fhir ftp jms/artemis Create a test-jar that contain main and test classes Align all components let camel compile Fix langchain4j compilation Revert Jetty test-infra Rebase Execute test-infra with camel jbang Add Camel JBang command Fix camel jbang infra run let mvnd compile Sort import Remove yaml output in favor of json/table Add javadoc Use Camel standard workflow for generated sources test-infra JUnit decouple documentation Add Camel JBang Infra Docs test infra metadata json Use LinkedHashSet for ordering

Croway requested review from davsclaus, orpiske and oscerd January 10, 2025 11:13

github-actions bot added components core dsl tooling tooling-maven tests test-infra camel-4 labels Jan 10, 2025

Croway force-pushed the execute-test-infra-camel-cli branch from 6a3840a to 27f532f Compare January 10, 2025 11:19

oscerd approved these changes Jan 10, 2025

View reviewed changes

components/camel-test/camel-test-junit5/src/main/java/org/apache/camel/test/AvailablePort.java Show resolved Hide resolved

Croway force-pushed the execute-test-infra-camel-cli branch 3 times, most recently from f00e136 to 1eb6619 Compare January 10, 2025 11:35

davsclaus approved these changes Jan 10, 2025

View reviewed changes

orpiske approved these changes Jan 10, 2025

View reviewed changes

orpiske reviewed Jan 10, 2025

View reviewed changes

Croway force-pushed the execute-test-infra-camel-cli branch 2 times, most recently from ca58f79 to 1bf98e1 Compare January 10, 2025 15:18

davsclaus reviewed Jan 11, 2025

View reviewed changes

tooling/spi-annotations/src/main/java/org/apache/camel/spi/annotations/InfraService.java Show resolved Hide resolved

Croway force-pushed the execute-test-infra-camel-cli branch from fc34baf to d586a51 Compare January 13, 2025 15:18

github-actions bot added the docs label Jan 13, 2025

Croway marked this pull request as ready for review January 13, 2025 16:03

Croway force-pushed the execute-test-infra-camel-cli branch from 12b9b7d to 44394be Compare January 13, 2025 16:16

davsclaus approved these changes Jan 13, 2025

View reviewed changes

Croway force-pushed the execute-test-infra-camel-cli branch from 44394be to 4da5ff5 Compare January 14, 2025 10:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expose test-infra services via Camel JBang #16773

Expose test-infra services via Camel JBang #16773

Croway commented Jan 10, 2025

github-actions bot commented Jan 10, 2025

Croway commented Jan 10, 2025

oscerd left a comment

davsclaus commented Jan 10, 2025

davsclaus commented Jan 10, 2025 •

edited

Loading

davsclaus commented Jan 10, 2025

davsclaus Jan 10, 2025

davsclaus Jan 10, 2025

davsclaus commented Jan 10, 2025

orpiske left a comment

orpiske Jan 10, 2025

Croway Jan 10, 2025

orpiske Jan 10, 2025

orpiske Jan 10, 2025

Croway Jan 10, 2025

Croway commented Jan 10, 2025

Expose test-infra services via Camel JBang #16773

Are you sure you want to change the base?

Expose test-infra services via Camel JBang #16773

Conversation

Croway commented Jan 10, 2025

github-actions bot commented Jan 10, 2025

Croway commented Jan 10, 2025

oscerd left a comment

Choose a reason for hiding this comment

davsclaus commented Jan 10, 2025

davsclaus commented Jan 10, 2025 • edited Loading

davsclaus commented Jan 10, 2025

davsclaus Jan 10, 2025

Choose a reason for hiding this comment

davsclaus Jan 10, 2025

Choose a reason for hiding this comment

davsclaus commented Jan 10, 2025

orpiske left a comment

Choose a reason for hiding this comment

orpiske Jan 10, 2025

Choose a reason for hiding this comment

Croway Jan 10, 2025

Choose a reason for hiding this comment

orpiske Jan 10, 2025

Choose a reason for hiding this comment

orpiske Jan 10, 2025

Choose a reason for hiding this comment

Croway Jan 10, 2025

Choose a reason for hiding this comment

Croway commented Jan 10, 2025

davsclaus commented Jan 10, 2025 •

edited

Loading