Argo workflow doesnot wait for sparkoperator to complete

[jijosg]

Hi,

I am trying to submit a spark job from argo workflow , the workflow succeeds in under 5s as soon as it submits the job to spark operator and doesnt wait for the job to complete.

I tried adding the wait and watch flag but didnot work.

Appreciate some pointers for understanding this behaviour.

[alexec]

We have other users using the Spark Operator successfully. I'm assuming you're using a resource template, so you probably need to set successCondition and failureCondition.

[alexec]

Could you please share your YAML? We should add something like this https://github.com/argoproj-labs/argo-workflows-catalog

[jijosg]

Hi @alexec , sharing the yaml for reference.
The spark job is trying to read two files from minio and display the dataframes.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: spark-argo 
  namespace: argo
spec:
  entrypoint: spark-job        
  templates:
  - name: spark-job
    resource:
      action: create
      manifest: |
        apiVersion: "sparkoperator.k8s.io/v1beta2"
        kind: SparkApplication
        metadata:
          generateName: spark-example-python
          namespace: spark-jobs
        spec:
          sparkConf:
            "spark.ui.port": "4041"
            "spark.hadoop.fs.s3a.endpoint": "http://minio.minio.svc.cluster.local:9000"
            "spark.hadoop.fs.s3a.access.key": "myaccesskey"
            "spark.hadoop.fs.s3a.secret.key": "mysecretkey"
            "spark.hadoop.fs.s3a.path.style.access": "true"
            "spark.hadoop.fs.s3a.connection.ssl.enabled": "false"
            "spark.hadoop.fs.s3a.impl": "org.apache.hadoop.fs.s3a.S3AFileSystem"
          deps:
            jars:
              - local:///opt/spark/jars/hadoop-aws-3.2.0.jar
              - local:///opt/spark/jars/hadoop-common-3.2.0.jar
              - local:///opt/spark/jars/aws-java-sdk-bundle-1.11.375.jar
          type: Python
          mode: cluster
          pythonVersion: "3"
          image: "spark-job-python:1.0"
          imagePullPolicy: Never
          mainApplicationFile: "local:///opt/spark/app/main.py"
          arguments:
          - "s3a://data/movies.csv"
          - "s3a://data/ratings.csv"
          sparkVersion: "3.0.1"
          restartPolicy:
            type: Never
          driver:
            cores: 1
            memory: "512m"
            labels:
              version: 3.0.1
            serviceAccount: my-spark-spark
          executor:
            cores: 1
            instances: 2
            memory: "512m"
            labels:
              version: 3.0.1

[jijosg]

I am yet to try the approach you shared with workflows catalog.

[alexec]

@jijosg good enough for public usage?

[jijosg]

If you are asking about the yaml, then yeah.

[alexec]

https://www.iteblog.com/ppt/dataai-summit-europe-2020/apache-spark-streaming-in-k8s-with-argocd-spark-operator-iteblog.com.pdf

[alexec]

#4174

[alexec]

We could do with some instructions for this use case at some point. I've asked in the community chat.

[hygor-knust]

Hi @jijosg, I'm trying to accomplish the same result.

I tried using successCondition but it did not work.

Did you manage to get it working?

Thanks!

[jijosg]

Hi @hygorkn , yes finally I got it working today with successCondition . I used the following with argo resource

successCondition : status.applicationState.state == COMPLETED

failureCondition : status.applicationState.state == FAILED

Thanks @alexec for your suggestions.

[hygor-knust]

Thanks, it worked!

[alexec]

You might find these examples helpful:

https://argoproj.github.io/argo/fields/#resourcetemplate