List table issue in presto hbase

[varun-tangoit]

Hi,

Im trying to query from presto hbase table, im facing below issue. i have configured all the schemas and put it compiled jar to presto/plugin folder. but still im facing issue while show tables internal server error issue. Kindly help me out from this issue,

2020-12-07T11:52:30.364Z	ERROR	remote-task-callback-5	io.prestosql.execution.StageStateMachine	Stage 20201207_115229_00003_tdxze.2 failed
java.lang.NullPointerException
	at com.analysys.presto.connector.hbase.meta.HBaseMetadata.listTables(HBaseMetadata.java:111)
	at io.prestosql.metadata.MetadataManager.listTables(MetadataManager.java:573)
	at io.prestosql.metadata.MetadataListing.listTables(MetadataListing.java:93)
	at io.prestosql.connector.informationschema.InformationSchemaPageSource.addTablesRecords(InformationSchemaPageSource.java:281)
	at io.prestosql.connector.informationschema.InformationSchemaPageSource.buildPages(InformationSchemaPageSource.java:219)
	at io.prestosql.connector.informationschema.InformationSchemaPageSource.getNextPage(InformationSchemaPageSource.java:183)
	at io.prestosql.operator.ScanFilterAndProjectOperator$ConnectorPageSourceToPages.process(ScanFilterAndProjectOperator.java:376)
	at io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:372)
	at io.prestosql.operator.WorkProcessorUtils.getNextState(WorkProcessorUtils.java:221)
	at io.prestosql.operator.WorkProcessorUtils$YieldingProcess.process(WorkProcessorUtils.java:181)
	at io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:372)
	at io.prestosql.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:306)
	at io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:372)
	at io.prestosql.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:306)
	at io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:372)
	at io.prestosql.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:306)
	at io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:372)
	at io.prestosql.operator.WorkProcessorUtils.getNextState(WorkProcessorUtils.java:221)
	at io.prestosql.operator.WorkProcessorUtils.lambda$processStateMonitor$2(WorkProcessorUtils.java:200)
	at io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:372)
	at io.prestosql.operator.WorkProcessorUtils.lambda$flatten$6(WorkProcessorUtils.java:277)
	at io.prestosql.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:319)
	at io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:372)
	at io.prestosql.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:306)
	at io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:372)
	at io.prestosql.operator.WorkProcessorUtils.getNextState(WorkProcessorUtils.java:221)
	at io.prestosql.operator.WorkProcessorUtils.lambda$processStateMonitor$2(WorkProcessorUtils.java:200)
	at io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:372)
	at io.prestosql.operator.WorkProcessorUtils.getNextState(WorkProcessorUtils.java:221)
	at io.prestosql.operator.WorkProcessorUtils.lambda$finishWhen$3(WorkProcessorUtils.java:215)
	at io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:372)
	at io.prestosql.operator.WorkProcessorSourceOperatorAdapter.getOutput(WorkProcessorSourceOperatorAdapter.java:149)
	at io.prestosql.operator.Driver.processInternal(Driver.java:379)
	at io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283)
	at io.prestosql.operator.Driver.tryWithLock(Driver.java:675)
	at io.prestosql.operator.Driver.processFor(Driver.java:276)
	at io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1076)
	at io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
	at io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)
	at io.prestosql.$gen.Presto_347____20201207_115115_2.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

[Crossoverrr]

Sorry for the late.
Checkout the code, I think the problem is the result of method HBaseTables#getTables is null.

Thats to say, the result of method Admin#listTableDescriptorsByNamespace has no descripters returned:

Please checkout if your namespace has tables, thank you for using the connector! :)

[varun-tangoit]

Thanks for the reply @Crossoverrr. Yeah I have resolved this issue. But what I'm currently facing issue bigint, array of dictionary, complex datatypes not render properly and I'm currently using cdh hbase 2.1.0 and presto 338 version.can you help how can define metainfo for those types, I'm stuck on this more than 3 days. Please help us.
For example:
101117733
[{"Id":44, "name": [22,33], "aa": "xx"}]

[Crossoverrr]

We support bigint, decimal and array[string] type already, you can checkout the doc here.

[varun-tangoit]

Yeah @Crossoverrr Yeah i can able to add those datatypes but still im facing this issue;Please find the below screen shots and column_name last_processed as bigint in presto, it render other datatypes, is there anything i missed and is it supports with cloudera hbase?

[Crossoverrr]

Well, I cannot see your pics :(

[varun-tangoit]

Okay @Crossoverrr . Here i have copy pasted information from console. Kindly help us;

Version 2.1.0-cdh6.3.3, rUnknown, Mon Jan 20 04:40:30 PST 2020
Took 0.0026 seconds
hbase(main):001:0> scan 'edge_data1'
ROW COLUMN+CELL
1 column=processeddata:inserted_at, timestamp=1607858232464, value=1605891634
1 column=processeddata:last_processed, timestamp=1607858232530, value=1605891634
1 row(s)
Took 1.7596 seconds
hbase(main):002:0>

presto:default> describe edge_data1;
Column | Type | Extra | Comment
----------------+---------+-------+---------
rowkey | varchar | |
inserted_at | integer | |
last_processed | bigint | |

Query 20201214_062525_00013_cvhch, FINISHED, 1 node
Splits: 19 total, 19 done (100.00%)
0.45 [114 rows, 8.06KB] [251 rows/s, 17.8KB/s]

presto:default> select last_processed from edge_data1;
last_processed

3546074770257753143
(10 rows)

[Crossoverrr]

You mean the true value for column last_processed is 1605891634, but when query from hbase-connector its value turn to be 3546074770257753143?

[varun-tangoit]

Yeah exactly :) I have tried all the datatypes bigint, integer, but its not working same way for other datatypes array of dictionary.

[Crossoverrr]

I think this may be due to a data type mismatch, you can checkout method HBaseRecordCursor#matchValue, in this method I change bytes from hbase to specific presto value.
and also method HBasePageSink#appendColumnValue I change value in presto data type to bytes and save them to hbase.

[varun-tangoit]

Yeah sure let me try @Crossoverrr . Can i ping any queries in presto-sql slack window?