如何使用Rest API提交batch任务？ #3994

imperio-wxm · 2022-12-16T06:06:47Z

imperio-wxm
Dec 16, 2022

在文档中没有找到http rest请求响应的端口，看到默认配置kyuubi.frontend.rest.bind.port= 10099，但是telnet kyuubi server ip 10099 并不通。

看到文档中有frontend.protocols的配置，默认为THRIFT_BINARY：

是否同时只支持一种协议？如何jdbc（交互式）、rest（批处理）同时使用？

Dec 16, 2022

kyuubi.frontend.protocols=THRIFT_BINARY,REST

The docs say "A comma separated list ...", I think it's clear enough, right?

View full answer

pan3793 · 2022-12-16T06:13:35Z

pan3793
Dec 16, 2022
Collaborator

kyuubi.frontend.protocols=THRIFT_BINARY,REST

The docs say "A comma separated list ...", I think it's clear enough, right?

1 reply

imperio-wxm Dec 16, 2022
Author

Thanks.

jia-zhengwei · 2023-07-25T03:18:16Z

jia-zhengwei
Jul 25, 2023

能帮提供一个batch rest api demo?

我通过batch api 提交job, 发现kyuubi server 的日志中 spark-submit 带有 --proxy-user, yarn集群没有收到job， job报错日志如下

java.lang.NullPointerException
        at org.apache.spark.deploy.PythonRunner$.$btrace$SparkCollector$collectPySparkThrowEnd(PythonRunner.scala)
        at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:103)
        at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
        at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:165)
        at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:163)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
java.lang.NullPointerException
        at org.apache.spark.deploy.PythonRunner$.$btrace$SparkCollector$collectPySparkThrowEnd(PythonRunner.scala)
        at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:106)
        at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
        at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:165)
        at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:163)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1748)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:163)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.spark.SparkUserAppException: User application exited with 2
        at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:103)
        at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
        at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:165)
        at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        ... 6 more
23/07/25 10:53:44 INFO [shutdown-hook-0] ShutdownHookManager: Shutdown hook called
23/07/25 10:53:44 INFO [shutdown-hook-0] ShutdownHookManager: Deleting directory /tmp/spark-7fea0641-5136-4320-9da9-fb0d02cafbe6

5 replies

pan3793 Jul 25, 2023
Collaborator

Any more information? Like Kyuubi server version, batch request API/parameters, the full spark-submit command

jia-zhengwei Jul 25, 2023

2023-07-25T11:52:51,081 INFO org.apache.kyuubi.operation.BatchJobSubmission: Submitting SPARK batch[c2b7864c-b435-4aab-a256-10050eee568a] job:
/opt/apps/SPARK3/spark-3.3.1-hadoop3.2-1.0.3/bin/spark-submit \
        --class org.apache.spark.deploy.PythonRunner \
        --conf spark.kyuubi.batch.id=c2b7864c-b435-4aab-a256-10050eee568a \
        --conf spark.kyuubi.batch.resource.uploaded=false \
        --conf spark.kyuubi.client.ipAddress=10.109.16.42 \
        --conf spark.kyuubi.server.ipAddress=0.0.0.0 \
        --conf spark.kyuubi.session.connection.url=0.0.0.0:10099 \
        --conf spark.kyuubi.session.real.user=kyuubi \
        --conf spark.app.name=kyuubi_batch_demo \
        --conf spark.driver.memory=1g \
        --conf spark.executor.memory=2g \
        --conf spark.kyuubi.server.ipAddress=xxx \
        --conf spark.kyuubi.session.connection.url=xxxx:10009 \
        --conf spark.yarn.tags=KYUUBI,c2b7864c-b435-4aab-a256-10050eee568a \
        --proxy-user kyuubi hdfs://xxx:9000/spark_job_files/mysql.py \
        --argsxxxxxx

jia-zhengwei Jul 25, 2023

curl --location --request POST --X POST 'http://10.10.188.179:10099/api/v1/batches' \
--header 'User-Agent: Apipost client Runtime/+https://www.apipost.cn/' \
--header 'Authorization: Basic a3l1dWJpOmt5dXViaQ==' \
--header 'Content-Type: application/json' \
--data '{
    "batchType": "Spark",
    "resource": "hdfs://xxx:9000/spark_job_files/mysql.py",
    "name": "kyuubi_batch_demo",
    "conf": {
        "spark.executor.memory": "2g",
        "spark.driver.memory": "1g",
        "spark.kyuubi.session.connection.url": "xxx:10009",
        "spark.kyuubi.server.ipAddress": "10.10.188.179"
    },
    "className": "org.apache.spark.deploy.PythonRunner",
    "args": [
        "xxx",
    ]
}'

jia-zhengwei Jul 25, 2023

Kyuubi server version: 1.7.1

pan3793 Jul 25, 2023
Collaborator

java.lang.NullPointerException
        at org.apache.spark.deploy.PythonRunner$.$btrace$SparkCollector$collectPySparkThrowEnd(PythonRunner.scala)

This is likely caused by an extra Java Agent, not the open-sourced Spark part, ask your Spark cluster vendor for help.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

如何使用Rest API提交batch任务？ #3994

{{title}}

Replies: 2 comments 6 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

如何使用Rest API提交batch任务？ #3994

imperio-wxm Dec 16, 2022

Replies: 2 comments · 6 replies

pan3793 Dec 16, 2022 Collaborator

imperio-wxm Dec 16, 2022 Author

jia-zhengwei Jul 25, 2023

pan3793 Jul 25, 2023 Collaborator

jia-zhengwei Jul 25, 2023

jia-zhengwei Jul 25, 2023

jia-zhengwei Jul 25, 2023

pan3793 Jul 25, 2023 Collaborator

imperio-wxm
Dec 16, 2022

Replies: 2 comments 6 replies

pan3793
Dec 16, 2022
Collaborator

imperio-wxm Dec 16, 2022
Author

jia-zhengwei
Jul 25, 2023

pan3793 Jul 25, 2023
Collaborator

pan3793 Jul 25, 2023
Collaborator