Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Druid -s3 Migration #615

Open
BigDataArtist opened this issue Apr 21, 2020 · 0 comments
Open

Druid -s3 Migration #615

BigDataArtist opened this issue Apr 21, 2020 · 0 comments

Comments

@BigDataArtist
Copy link

I am trying to migrate the data from druid to s3 bucket. Below are my configurations

Error

020-04-21T12:57:26,235 INFO [main] org.apache.druid.cli.CliPeon - * java.vm.specification.name: Java Virtual Machine Specification
2020-04-21T12:57:26,235 INFO [main] org.apache.druid.cli.CliPeon - * java.vm.specification.vendor: Oracle Corporation
2020-04-21T12:57:26,235 INFO [main] org.apache.druid.cli.CliPeon - * java.vm.specification.version: 1.8
2020-04-21T12:57:26,235 INFO [main] org.apache.druid.cli.CliPeon - * java.vm.vendor: Oracle Corporation
2020-04-21T12:57:26,236 INFO [main] org.apache.druid.cli.CliPeon - * java.vm.version: 25.241-b07
2020-04-21T12:57:26,236 INFO [main] org.apache.druid.cli.CliPeon - * line.separator:

2020-04-21T12:57:26,236 INFO [main] org.apache.druid.cli.CliPeon - * log4j.shutdownCallbackRegistry: org.apache.druid.common.config.Log4jShutdown
2020-04-21T12:57:26,236 INFO [main] org.apache.druid.cli.CliPeon - * log4j.shutdownHookEnabled: true
2020-04-21T12:57:26,236 INFO [main] org.apache.druid.cli.CliPeon - * os.arch: x86_64
2020-04-21T12:57:26,236 INFO [main] org.apache.druid.cli.CliPeon - * os.name: Mac OS X
2020-04-21T12:57:26,236 INFO [main] org.apache.druid.cli.CliPeon - * os.version: 10.15.1
2020-04-21T12:57:26,237 INFO [main] org.apache.druid.cli.CliPeon - * path.separator: :
2020-04-21T12:57:26,237 INFO [main] org.apache.druid.cli.CliPeon - * socksNonProxyHosts: local|.local|169.254/16|.169.254/16
2020-04-21T12:57:26,237 INFO [main] org.apache.druid.cli.CliPeon - * sun.arch.data.model: 64
2020-04-21T12:57:26,237 INFO [main] org.apache.druid.cli.CliPeon - * sun.boot.class.path: /Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/jre/lib/sunrsasign.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/jre/classes
2020-04-21T12:57:26,238 INFO [main] org.apache.druid.cli.CliPeon - * sun.boot.library.path: /Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/jre/lib
2020-04-21T12:57:26,238 INFO [main] org.apache.druid.cli.CliPeon - * sun.cpu.endian: little
2020-04-21T12:57:26,238 INFO [main] org.apache.druid.cli.CliPeon - * sun.cpu.isalist:
2020-04-21T12:57:26,238 INFO [main] org.apache.druid.cli.CliPeon - * sun.io.unicode.encoding: UnicodeBig
2020-04-21T12:57:26,238 INFO [main] org.apache.druid.cli.CliPeon - * sun.java.command: org.apache.druid.cli.Main internal peon var/druid/task/index_parallel_test_ijbajejj_2020-04-21T12:57:10.447Z/task.json var/druid/task/index_parallel_test_ijbajejj_2020-04-21T12:57:10.447Z/32a4549d-e110-49a5-9e56-69e994674447/status.json var/druid/task/index_parallel_test_ijbajejj_2020-04-21T12:57:10.447Z/32a4549d-e110-49a5-9e56-69e994674447/report.json
2020-04-21T12:57:26,238 INFO [main] org.apache.druid.cli.CliPeon - * sun.java.launcher: SUN_STANDARD
2020-04-21T12:57:26,238 INFO [main] org.apache.druid.cli.CliPeon - * sun.jnu.encoding: UTF-8
2020-04-21T12:57:26,238 INFO [main] org.apache.druid.cli.CliPeon - * sun.management.compiler: HotSpot 64-Bit Tiered Compilers
2020-04-21T12:57:26,238 INFO [main] org.apache.druid.cli.CliPeon - * sun.os.patch.level: unknown
2020-04-21T12:57:26,239 INFO [main] org.apache.druid.cli.CliPeon - * user.country: US
2020-04-21T12:57:26,239 INFO [main] org.apache.druid.cli.CliPeon - * user.dir: /Users/adityaverma/Downloads/apache-druid-0.17.1
2020-04-21T12:57:26,239 INFO [main] org.apache.druid.cli.CliPeon - * user.home: /Users/adityaverma
2020-04-21T12:57:26,239 INFO [main] org.apache.druid.cli.CliPeon - * user.language: en
2020-04-21T12:57:26,239 INFO [main] org.apache.druid.cli.CliPeon - * user.name: adityaverma
2020-04-21T12:57:26,239 INFO [main] org.apache.druid.cli.CliPeon - * user.timezone: UTC
2020-04-21T12:57:29,816 ERROR [main] org.apache.druid.cli.CliPeon - Error when starting up. Failing.
com.google.inject.ProvisionException: Unable to provision, see the following errors:

  1. Error injecting constructor, java.lang.IllegalArgumentException: Can not create a Path from an empty string
    at org.apache.druid.storage.hdfs.HdfsDataSegmentKiller.(HdfsDataSegmentKiller.java:48)
    while locating org.apache.druid.storage.hdfs.HdfsDataSegmentKiller
    at org.apache.druid.storage.hdfs.HdfsStorageDruidModule.configure(HdfsStorageDruidModule.java:83) (via modules: com.google.inject.util.Modules$OverrideModule -> org.apache.druid.storage.hdfs.HdfsStorageDruidModule)
    while locating org.apache.druid.segment.loading.DataSegmentKiller annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=155, type=MAPBINDER, keyType=java.lang.String)
    at org.apache.druid.guice.Binders.dataSegmentKillerBinder(Binders.java:40) (via modules: com.google.inject.util.Modules$OverrideModule -> org.apache.druid.storage.s3.S3StorageDruidModule -> com.google.inject.multibindings.MapBinder$RealMapBinder)
    while locating java.util.Map<java.lang.String, org.apache.druid.segment.loading.DataSegmentKiller>
    for the 1st parameter of org.apache.druid.segment.loading.OmniDataSegmentKiller.(OmniDataSegmentKiller.java:38)
    while locating org.apache.druid.segment.loading.OmniDataSegmentKiller
    at org.apache.druid.cli.CliPeon.bindPeonDataSegmentHandlers(CliPeon.java:362) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.cli.CliPeon$1)
    while locating org.apache.druid.segment.loading.DataSegmentKiller
    for the 6th parameter of org.apache.druid.indexing.common.TaskToolboxFactory.(TaskToolboxFactory.java:120)
    at org.apache.druid.cli.CliPeon.bindTaskConfigAndClients(CliPeon.java:400) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.cli.CliPeon$1)
    while locating org.apache.druid.indexing.common.TaskToolboxFactory
    for the 1st parameter of org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner.(SingleTaskBackgroundRunner.java:95)
    at org.apache.druid.cli.CliPeon$1.configure(CliPeon.java:210) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.cli.CliPeon$1)
    while locating org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner
    while locating org.apache.druid.indexing.overlord.TaskRunner
    for the 4th parameter of org.apache.druid.indexing.worker.executor.ExecutorLifecycle.(ExecutorLifecycle.java:80)
    at org.apache.druid.cli.CliPeon$1.configure(CliPeon.java:194) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.cli.CliPeon$1)
    while locating org.apache.druid.indexing.worker.executor.ExecutorLifecycle

1 error
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1028) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1050) ~[guice-4.1.0.jar:?]
at org.apache.druid.guice.LifecycleModule$2.start(LifecycleModule.java:141) ~[druid-core-0.17.1.jar:0.17.1]
at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:115) [druid-services-0.17.1.jar:0.17.1]
at org.apache.druid.cli.CliPeon.run(CliPeon.java:280) [druid-services-0.17.1.jar:0.17.1]
at org.apache.druid.cli.Main.main(Main.java:113) [druid-services-0.17.1.jar:0.17.1]
Caused by: java.lang.IllegalArgumentException: Can not create a Path from an empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:163) ~[?:?]
at org.apache.hadoop.fs.Path.(Path.java:175) ~[?:?]
at org.apache.druid.storage.hdfs.HdfsDataSegmentKiller.(HdfsDataSegmentKiller.java:50) ~[?:?]
at org.apache.druid.storage.hdfs.HdfsDataSegmentKiller$$FastClassByGuice$$b7e9ca1b.newInstance() ~[?:?]
at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:111) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) ~[guice-4.1.0.jar:?]
at com.google.inject.spi.ProviderLookup$1.get(ProviderLookup.java:104) ~[guice-4.1.0.jar:?]
at com.google.inject.spi.ProviderLookup$1.get(ProviderLookup.java:104) ~[guice-4.1.0.jar:?]
at com.google.inject.multibindings.MapBinder$RealMapBinder$ValueProvider.get(MapBinder.java:821) ~[guice-multibindings-4.1.0.jar:?]
at com.google.inject.multibindings.MapBinder$RealMapBinder$RealMapProvider.get(MapBinder.java:605) ~[guice-multibindings-4.1.0.jar:?]
at com.google.inject.multibindings.MapBinder$RealMapBinder$RealMapProvider.get(MapBinder.java:586) ~[guice-multibindings-4.1.0.jar:?]
at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.1.0.jar:?]
at org.apache.druid.guice.LifecycleScope$1.get(LifecycleScope.java:68) ~[druid-core-0.17.1.jar:0.17.1]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.1.0.jar:?]
at org.apache.druid.guice.LifecycleScope$1.get(LifecycleScope.java:68) ~[druid-core-0.17.1.jar:0.17.1]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) ~[guice-4.1.0.jar:?]

DEEP Storage properties

common.runtime.properties

//I commented out for the localdisk

For local disk (only viable in a cluster if this is a network mount):

#druid.storage.type=local
#druid.storage.storageDirectory=var/druid/segments

For S3:

druid.storage.type=s3
druid.storage.bucket=druids3migration
druid.storage.baseKey=druid/segments
druid.s3.accessKeyi=AZKB47
druid.s3.secretKey=d1b*ABExuCxvxv3Use+Uzs

-Daws.region=ap-south-1 --> middleManager/jvm.properties

// middleManager/runtime.properties
druid.indexer.runner.javaOpts=-server -Xms1g -Xmx1g -XX:MaxDirectMemorySize=1g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+ExitOnOutOfMemoryError -Daws.region=ap-south-1 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@BigDataArtist