We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Exception:
The error typically occurs when the default LogStore implementation, that is, HDFSLogStore, is used to write into a Delta table on a non-HDFS storage system. In order to get the transactional ACID guarantees on table updates, you have to use the correct implementation of LogStore that is appropriate for your storage system. See https://docs.delta.io/latest/delta-storage.html for details. java.io.IOException: The error typically occurs when the default LogStore implementation, that is, HDFSLogStore, is used to write into a Delta table on a non-HDFS storage system. In order to get the transactional ACID guarantees on table updates, you have to use the correct implementation of LogStore that is appropriate for your storage system. See https://docs.delta.io/latest/delta-storage.html for details. io.delta.storage.internal.LogStoreErrors.incorrectLogStoreImplementationException(LogStoreErrors.java:41) io.delta.storage.HDFSLogStore.writeInternal(HDFSLogStore.java:94) io.delta.storage.HDFSLogStore.write(HDFSLogStore.java:68) org.apache.spark.sql.delta.storage.LogStoreAdaptor.write(LogStore.scala:414) org.apache.spark.sql.delta.storage.DelegatingLogStore.write(DelegatingLogStore.scala:119) org.apache.spark.sql.delta.OptimisticTransactionImpl.doCommit(OptimisticTransaction.scala:1194) org.apache.spark.sql.delta.OptimisticTransactionImpl.doCommit$(OptimisticTransaction.scala:1165) org.apache.spark.sql.delta.OptimisticTransaction.doCommit(OptimisticTransaction.scala:101) org.apache.spark.sql.delta.OptimisticTransactionImpl.$anonfun$doCommitRetryIteratively$3(OptimisticTransaction.scala:1135) scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158) org.apache.spark.sql.delta.OptimisticTransactionImpl.$anonfun$doCommitRetryIteratively$2(OptimisticTransaction.scala:1132) scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) org.apache.spark.sql.delta.metering.DeltaLogging.recordFrameProfile(DeltaLogging.scala:139) org.apache.spark.sql.delta.metering.DeltaLogging.recordFrameProfile$(DeltaLogging.scala:137) org.apache.spark.sql.delta.OptimisticTransaction.recordFrameProfile(OptimisticTransaction.scala:101) org.apache.spark.sql.delta.metering.DeltaLogging.$anonfun$recordDeltaOperationInternal$1(DeltaLogging.scala:132) com.databricks.spark.util.DatabricksLogging.recordOperation(DatabricksLogging.scala:77) com.databricks.spark.util.DatabricksLogging.recordOperation$(DatabricksLogging.scala:67) org.apache.spark.sql.delta.OptimisticTransaction.recordOperation(OptimisticTransaction.scala:101) org.apache.spark.sql.delta.metering.DeltaLogging.recordDeltaOperationInternal(DeltaLogging.scala:131) org.apache.spark.sql.delta.metering.DeltaLogging.recordDeltaOperation(DeltaLogging.scala:121) org.apache.spark.sql.delta.metering.DeltaLogging.recordDeltaOperation$(DeltaLogging.scala:109) org.apache.spark.sql.delta.OptimisticTransaction.recordDeltaOperation(OptimisticTransaction.scala:101) org.apache.spark.sql.delta.OptimisticTransactionImpl.$anonfun$doCommitRetryIteratively$1(OptimisticTransaction.scala:1132) org.apache.spark.sql.delta.DeltaLog.lockInterruptibly(DeltaLog.scala:187) org.apache.spark.sql.delta.OptimisticTransactionImpl.lockCommitIfEnabled(OptimisticTransaction.scala:1108) org.apache.spark.sql.delta.OptimisticTransactionImpl.doCommitRetryIteratively(OptimisticTransaction.scala:1126) org.apache.spark.sql.delta.OptimisticTransactionImpl.doCommitRetryIteratively$(OptimisticTransaction.scala:1122) org.apache.spark.sql.delta.OptimisticTransaction.doCommitRetryIteratively(OptimisticTransaction.scala:101) org.apache.spark.sql.delta.OptimisticTransactionImpl.liftedTree1$1(OptimisticTransaction.scala:748) org.apache.spark.sql.delta.OptimisticTransactionImpl.$anonfun$commit$1(OptimisticTransaction.scala:691) scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) org.apache.spark.sql.delta.metering.DeltaLogging.recordFrameProfile(DeltaLogging.scala:139) org.apache.spark.sql.delta.metering.DeltaLogging.recordFrameProfile$(DeltaLogging.scala:137) org.apache.spark.sql.delta.OptimisticTransaction.recordFrameProfile(OptimisticTransaction.scala:101) org.apache.spark.sql.delta.metering.DeltaLogging.$anonfun$recordDeltaOperationInternal$1(DeltaLogging.scala:132) com.databricks.spark.util.DatabricksLogging.recordOperation(DatabricksLogging.scala:77) com.databricks.spark.util.DatabricksLogging.recordOperation$(DatabricksLogging.scala:67) org.apache.spark.sql.delta.OptimisticTransaction.recordOperation(OptimisticTransaction.scala:101) org.apache.spark.sql.delta.metering.DeltaLogging.recordDeltaOperationInternal(DeltaLogging.scala:131) org.apache.spark.sql.delta.metering.DeltaLogging.recordDeltaOperation(DeltaLogging.scala:121) org.apache.spark.sql.delta.metering.DeltaLogging.recordDeltaOperation$(DeltaLogging.scala:109) org.apache.spark.sql.delta.OptimisticTransaction.recordDeltaOperation(OptimisticTransaction.scala:101) org.apache.spark.sql.delta.OptimisticTransactionImpl.commit(OptimisticTransaction.scala:688) org.apache.spark.sql.delta.OptimisticTransactionImpl.commit$(OptimisticTransaction.scala:686) org.apache.spark.sql.delta.OptimisticTransaction.commit(OptimisticTransaction.scala:101) org.apache.spark.sql.delta.commands.WriteIntoDelta.$anonfun$run$1(WriteIntoDelta.scala:101) org.apache.spark.sql.delta.commands.WriteIntoDelta.$anonfun$run$1$adapted(WriteIntoDelta.scala:91) org.apache.spark.sql.delta.DeltaLog.withNewTransaction(DeltaLog.scala:221) org.apache.spark.sql.delta.commands.WriteIntoDelta.run(WriteIntoDelta.scala:91) org.apache.spark.sql.delta.sources.MLSQLDeltaDataSource.createRelation(MLSQLDeltaDataSource.scala:57) org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45) org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75) org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73) org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84) org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98) org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109) org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169) org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95) org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779) org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64) org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98) org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94) org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584) org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176) org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584) org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30) org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267) org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263) org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30) org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30) org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:560) org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94) org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81) org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79) org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:116) org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:860) org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:390) org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:303) org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239) streaming.core.datasource.impl.MLSQLDelta.save(MLSQLDelta.scala:104) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) tech.mlsql.dsl.adaptor.SaveAdaptor.$anonfun$parse$2(SaveAdaptor.scala:135) scala.Option.map(Option.scala:230) tech.mlsql.dsl.adaptor.SaveAdaptor.parse(SaveAdaptor.scala:126) streaming.dsl.ScriptSQLExecListener.execute$1(ScriptSQLExec.scala:406) streaming.dsl.ScriptSQLExecListener.exitSql(ScriptSQLExec.scala:424) streaming.dsl.parser.DSLSQLParser$SqlContext.exitRule(DSLSQLParser.java:296) org.antlr.v4.runtime.tree.ParseTreeWalker.exitRule(ParseTreeWalker.java:47) org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:30) org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) streaming.dsl.ScriptSQLExec$._parse(ScriptSQLExec.scala:159) streaming.dsl.ScriptSQLExec$.parse(ScriptSQLExec.scala:146) streaming.rest.RestController.$anonfun$script$1(RestController.scala:163) tech.mlsql.job.JobManager$.run(JobManager.scala:74) tech.mlsql.job.JobManager$$anon$1.run(JobManager.scala:91) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: fs.AbstractFileSystem.obs.impl=null: No AbstractFileSystem configured for scheme: obs org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSystem.java:177) org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:266) org.apache.hadoop.fs.FileContext$2.run(FileContext.java:342) org.apache.hadoop.fs.FileContext$2.run(FileContext.java:339) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:422) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:339) org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:465) io.delta.storage.HDFSLogStore.writeInternal(HDFSLogStore.java:90) io.delta.storage.HDFSLogStore.write(HDFSLogStore.java:68) org.apache.spark.sql.delta.storage.LogStoreAdaptor.write(LogStore.scala:414) org.apache.spark.sql.delta.storage.DelegatingLogStore.write(DelegatingLogStore.scala:119) org.apache.spark.sql.delta.OptimisticTransactionImpl.doCommit(OptimisticTransaction.scala:1194) org.apache.spark.sql.delta.OptimisticTransactionImpl.doCommit$(OptimisticTransaction.scala:1165) org.apache.spark.sql.delta.OptimisticTransaction.doCommit(OptimisticTransaction.scala:101) org.apache.spark.sql.delta.OptimisticTransactionImpl.$anonfun$doCommitRetryIteratively$3(OptimisticTransaction.scala:1135) scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158) org.apache.spark.sql.delta.OptimisticTransactionImpl.$anonfun$doCommitRetryIteratively$2(OptimisticTransaction.scala:1132) scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) org.apache.spark.sql.delta.metering.DeltaLogging.recordFrameProfile(DeltaLogging.scala:139) org.apache.spark.sql.delta.metering.DeltaLogging.recordFrameProfile$(DeltaLogging.scala:137) org.apache.spark.sql.delta.OptimisticTransaction.recordFrameProfile(OptimisticTransaction.scala:101) org.apache.spark.sql.delta.metering.DeltaLogging.$anonfun$recordDeltaOperationInternal$1(DeltaLogging.scala:132) com.databricks.spark.util.DatabricksLogging.recordOperation(DatabricksLogging.scala:77) com.databricks.spark.util.DatabricksLogging.recordOperation$(DatabricksLogging.scala:67) org.apache.spark.sql.delta.OptimisticTransaction.recordOperation(OptimisticTransaction.scala:101) org.apache.spark.sql.delta.metering.DeltaLogging.recordDeltaOperationInternal(DeltaLogging.scala:131) org.apache.spark.sql.delta.metering.DeltaLogging.recordDeltaOperation(DeltaLogging.scala:121) org.apache.spark.sql.delta.metering.DeltaLogging.recordDeltaOperation$(DeltaLogging.scala:109) org.apache.spark.sql.delta.OptimisticTransaction.recordDeltaOperation(OptimisticTransaction.scala:101) org.apache.spark.sql.delta.OptimisticTransactionImpl.$anonfun$doCommitRetryIteratively$1(OptimisticTransaction.scala:1132) org.apache.spark.sql.delta.DeltaLog.lockInterruptibly(DeltaLog.scala:187) org.apache.spark.sql.delta.OptimisticTransactionImpl.lockCommitIfEnabled(OptimisticTransaction.scala:1108) org.apache.spark.sql.delta.OptimisticTransactionImpl.doCommitRetryIteratively(OptimisticTransaction.scala:1126) org.apache.spark.sql.delta.OptimisticTransactionImpl.doCommitRetryIteratively$(OptimisticTransaction.scala:1122) org.apache.spark.sql.delta.OptimisticTransaction.doCommitRetryIteratively(OptimisticTransaction.scala:101) org.apache.spark.sql.delta.OptimisticTransactionImpl.liftedTree1$1(OptimisticTransaction.scala:748) org.apache.spark.sql.delta.OptimisticTransactionImpl.$anonfun$commit$1(OptimisticTransaction.scala:691) scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) org.apache.spark.sql.delta.metering.DeltaLogging.recordFrameProfile(DeltaLogging.scala:139) org.apache.spark.sql.delta.metering.DeltaLogging.recordFrameProfile$(DeltaLogging.scala:137) org.apache.spark.sql.delta.OptimisticTransaction.recordFrameProfile(OptimisticTransaction.scala:101) org.apache.spark.sql.delta.metering.DeltaLogging.$anonfun$recordDeltaOperationInternal$1(DeltaLogging.scala:132) com.databricks.spark.util.DatabricksLogging.recordOperation(DatabricksLogging.scala:77) com.databricks.spark.util.DatabricksLogging.recordOperation$(DatabricksLogging.scala:67) org.apache.spark.sql.delta.OptimisticTransaction.recordOperation(OptimisticTransaction.scala:101) org.apache.spark.sql.delta.metering.DeltaLogging.recordDeltaOperationInternal(DeltaLogging.scala:131) org.apache.spark.sql.delta.metering.DeltaLogging.recordDeltaOperation(DeltaLogging.scala:121) org.apache.spark.sql.delta.metering.DeltaLogging.recordDeltaOperation$(DeltaLogging.scala:109) org.apache.spark.sql.delta.OptimisticTransaction.recordDeltaOperation(OptimisticTransaction.scala:101) org.apache.spark.sql.delta.OptimisticTransactionImpl.commit(OptimisticTransaction.scala:688) org.apache.spark.sql.delta.OptimisticTransactionImpl.commit$(OptimisticTransaction.scala:686) org.apache.spark.sql.delta.OptimisticTransaction.commit(OptimisticTransaction.scala:101) org.apache.spark.sql.delta.commands.WriteIntoDelta.$anonfun$run$1(WriteIntoDelta.scala:101) org.apache.spark.sql.delta.commands.WriteIntoDelta.$anonfun$run$1$adapted(WriteIntoDelta.scala:91) org.apache.spark.sql.delta.DeltaLog.withNewTransaction(DeltaLog.scala:221) org.apache.spark.sql.delta.commands.WriteIntoDelta.run(WriteIntoDelta.scala:91) org.apache.spark.sql.delta.sources.MLSQLDeltaDataSource.createRelation(MLSQLDeltaDataSource.scala:57) org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45) org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75) org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73) org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84) org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98) org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109) org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169) org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95) org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779) org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64) org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98) org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94) org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584) org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176) org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584) org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30) org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267) org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263) org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30) org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30) org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:560) org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94) org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81) org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79) org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:116) org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:860) org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:390) org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:303) org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239) streaming.core.datasource.impl.MLSQLDelta.save(MLSQLDelta.scala:104) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) tech.mlsql.dsl.adaptor.SaveAdaptor.$anonfun$parse$2(SaveAdaptor.scala:135) scala.Option.map(Option.scala:230) tech.mlsql.dsl.adaptor.SaveAdaptor.parse(SaveAdaptor.scala:126) streaming.dsl.ScriptSQLExecListener.execute$1(ScriptSQLExec.scala:406) streaming.dsl.ScriptSQLExecListener.exitSql(ScriptSQLExec.scala:424) streaming.dsl.parser.DSLSQLParser$SqlContext.exitRule(DSLSQLParser.java:296) org.antlr.v4.runtime.tree.ParseTreeWalker.exitRule(ParseTreeWalker.java:47) org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:30) org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) streaming.dsl.ScriptSQLExec$._parse(ScriptSQLExec.scala:159) streaming.dsl.ScriptSQLExec$.parse(ScriptSQLExec.scala:146) streaming.rest.RestController.$anonfun$script$1(RestController.scala:163) tech.mlsql.job.JobManager$.run(JobManager.scala:74) tech.mlsql.job.JobManager$$anon$1.run(JobManager.scala:91) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750)
The text was updated successfully, but these errors were encountered:
No branches or pull requests
Exception:
The text was updated successfully, but these errors were encountered: