Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Permission denied error #9

Open
panda87 opened this issue Oct 10, 2015 · 11 comments
Open

Permission denied error #9

panda87 opened this issue Oct 10, 2015 · 11 comments

Comments

@panda87
Copy link

panda87 commented Oct 10, 2015

Hi

When I'm trying to run the "More complex local development setup" I'm getting the error below,
Someone knows how can I save the data to another location in the file system?
i.e the logs folder will be transparent in my /kafka/logs ?

[2015-10-10 19:30:42,236] FATAL [Kafka Server 2], Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.io.IOException: Permission denied
    at java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.io.File.createNewFile(File.java:1006)
    at kafka.utils.FileLock.<init>(FileLock.scala:28)
    at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:96)
    at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:95)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
    at scala.collection.AbstractTraversable.map(Traversable.scala:105)
    at kafka.log.LogManager.lockLogDirs(LogManager.scala:95)
    at kafka.log.LogManager.<init>(LogManager.scala:57)
    at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:335)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:85)
    at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:29)
    at kafka.Kafka$.main(Kafka.scala:46)
    at kafka.Kafka.main(Kafka.scala)
[2015-10-10 19:30:42,248] INFO [Kafka Server 2], shutting down (kafka.server.KafkaServer)
...

Thanks
D.
@ches
Copy link
Owner

ches commented Oct 11, 2015

Are you running Docker on OS X or Windows via docker-machine or boot2docker? In that case you're probably running into this, probably the single worst remaining Docker usability issue for development that I know of:

boot2docker/boot2docker#581

@panda87
Copy link
Author

panda87 commented Oct 12, 2015

Hi @ches thanks for the quick response.
Im running this container on Ubuntu14.04.
What do you think can be the problem there?

Thanks

@ches
Copy link
Owner

ches commented Oct 13, 2015

What's the command you're using to run the container, and what are the permissions on the directory that you're mounting for the volume?

@panda87
Copy link
Author

panda87 commented Oct 14, 2015 

$ mkdir -p kafka-ex/{data,logs} && cd kafka-ex
$ ls -l
total 16
drwxrwxr-x 4  david 4096 Oct 14 10:42 ./
drwxr-xr-x 5 david david 4096 Oct 14 10:42 ../
drwxrwxr-x 2 david david 4096 Oct 14 10:42 data/
drwxrwxr-x 2 david david 4096 Oct 14 10:42 logs/
$ docker run -d \
    --hostname 127.0.0.1 \
    --name kafka \
    --volume ./data:/data --volume ./logs:/logs \
    --publish 9092:9092 --publish 7203:7203 \
    --env KAFKA_ADVERTISED_HOST_NAME=127.0.0.1 --env ZOOKEEPER_IP=EXT_IP \
    ches/kafka

$ docker logs -f kafka
Starting kafka
[2015-10-14 11:10:22,238] INFO Verifying properties (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,274] INFO Property advertised.host.name is overridden to 127.0.0.1 (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,274] INFO Property advertised.port is overridden to 9092 (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,274] INFO Property auto.create.topics.enable is overridden to true (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,275] INFO Property auto.leader.rebalance.enable is overridden to true (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,275] INFO Property broker.id is overridden to 0 (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,275] INFO Property controlled.shutdown.enable is overridden to true (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,275] INFO Property default.replication.factor is overridden to 1 (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,275] INFO Property log.dir is overridden to /data (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,276] INFO Property log.dirs is overridden to /data (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,276] INFO Property log.retention.hours is overridden to 168 (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,276] INFO Property num.partitions is overridden to 1 (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,276] INFO Property port is overridden to 9092 (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,276] INFO Property zookeeper.connect is overridden to EXT_IP:2181 (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,276] INFO Property zookeeper.connection.timeout.ms is overridden to 10000 (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,277] INFO Property zookeeper.session.timeout.ms is overridden to 10000 (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,316] INFO [Kafka Server 0], starting (kafka.server.KafkaServer)
[2015-10-14 11:10:22,318] INFO [Kafka Server 0], Connecting to zookeeper on EXT_IP:2181 (kafka.server.KafkaServer)
[2015-10-14 11:10:22,329] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2015-10-14 11:10:22,336] INFO Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,336] INFO Client environment:host.name=localhost (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,336] INFO Client environment:java.version=1.7.0_72 (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,336] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,336] INFO Client environment:java.home=/usr/lib/jvm/java-7-oracle/jre (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,336] INFO Client environment:java.class.path=:/kafka/bin/../core/build/dependant-libs-2.10.4*/*.jar:/kafka/bin/../examples/build/libs//kafka-examples*.jar:/kafka/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/kafka/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/kafka/bin/../clients/build/libs/kafka-clients*.jar:/kafka/bin/../libs/jopt-simple-3.2.jar:/kafka/bin/../libs/kafka-clients-0.8.2.1.jar:/kafka/bin/../libs/kafka_2.10-0.8.2.1-javadoc.jar:/kafka/bin/../libs/kafka_2.10-0.8.2.1-scaladoc.jar:/kafka/bin/../libs/kafka_2.10-0.8.2.1-sources.jar:/kafka/bin/../libs/kafka_2.10-0.8.2.1-test.jar:/kafka/bin/../libs/kafka_2.10-0.8.2.1.jar:/kafka/bin/../libs/log4j-1.2.16.jar:/kafka/bin/../libs/lz4-1.2.0.jar:/kafka/bin/../libs/metrics-core-2.2.0.jar:/kafka/bin/../libs/scala-library-2.10.4.jar:/kafka/bin/../libs/slf4j-api-1.7.6.jar:/kafka/bin/../libs/slf4j-log4j12-1.6.1.jar:/kafka/bin/../libs/snappy-java-1.1.1.6.jar:/kafka/bin/../libs/zkclient-0.3.jar:/kafka/bin/../libs/zookeeper-3.4.6.jar:/kafka/bin/../core/build/libs/kafka_2.10*.jar (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,336] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,337] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,337] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,337] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,337] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,337] INFO Client environment:os.version=3.13.0-48-generic (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,337] INFO Client environment:user.name=kafka (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,337] INFO Client environment:user.home=/kafka (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,337] INFO Client environment:user.dir=/kafka (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,338] INFO Initiating client connection, connectString=EXT_IP:2181 sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient@546ce79d (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,359] INFO Opening socket connection to server EXT_IP/EXT_IP:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2015-10-14 11:10:22,365] INFO Socket connection established to EXT_IP/EXT_IP:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2015-10-14 11:10:22,376] INFO Session establishment complete on server EXT_IP/EXT_IP:2181, sessionid = 0x1506602d5790003, negotiated timeout = 10000 (org.apache.zookeeper.ClientCnxn)
[2015-10-14 11:10:22,378] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2015-10-14 11:10:22,438] FATAL [Kafka Server 0], Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.io.IOException: Permission denied
    at java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.io.File.createNewFile(File.java:1006)
    at kafka.utils.FileLock.<init>(FileLock.scala:28)
    at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:96)
    at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:95)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
    at scala.collection.AbstractTraversable.map(Traversable.scala:105)
    at kafka.log.LogManager.lockLogDirs(LogManager.scala:95)
    at kafka.log.LogManager.<init>(LogManager.scala:57)
    at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:335)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:85)
    at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:29)
    at kafka.Kafka$.main(Kafka.scala:46)
    at kafka.Kafka.main(Kafka.scala)
[2015-10-14 11:10:22,441] INFO [Kafka Server 0], shutting down (kafka.server.KafkaServer)
[2015-10-14 11:10:22,445] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2015-10-14 11:10:22,451] INFO Session: 0x1506602d5790003 closed (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,451] INFO EventThread shut down (org.apache.zookeeper.ClientCnxn)
[2015-10-14 11:10:22,452] INFO [Kafka Server 0], shut down completed (kafka.server.KafkaServer)
[2015-10-14 11:10:22,453] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
java.io.IOException: Permission denied
    at java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.io.File.createNewFile(File.java:1006)
    at kafka.utils.FileLock.<init>(FileLock.scala:28)
    at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:96)
    at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:95)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
    at scala.collection.AbstractTraversable.map(Traversable.scala:105)
    at kafka.log.LogManager.lockLogDirs(LogManager.scala:95)
    at kafka.log.LogManager.<init>(LogManager.scala:57)
    at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:335)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:85)
    at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:29)
    at kafka.Kafka$.main(Kafka.scala:46)
    at kafka.Kafka.main(Kafka.scala)
[2015-10-14 11:10:22,455] INFO [Kafka Server 0], shutting down (kafka.server.KafkaServer)

What do you think?

@ches
Copy link
Owner

ches commented Oct 14, 2015

I haven't tested this approach for awhile, it might take some time before I have a chance to, but I would imagine that a kludgey solution to give you a workaround for now is: chmod 777 data logs.

The longer answer is probably that Docker data volume containers are "the Docker way" to deal with this, but that's ultimately only an indirection. This still boils down to the same root usability needs on the issue I mentioned at first. See also:

http://stackoverflow.com/questions/23544282/what-is-the-best-way-to-manage-permissions-for-docker-shared-volumes
moby/moby#7198

Hope that helps for now.

@panda87
Copy link
Author

panda87 commented Oct 14, 2015

Thanks for the answer,
I already tried the 'chmod' command without success.
I'll read the links above and I hope this will solve my issue.

@SimonMiaou
Copy link

@panda87 Did you solve this issue?

@panda87
Copy link
Author

panda87 commented Jan 5, 2016

Yes, I set the volume permissions to be ubuntu:ubuntu
This is the only way it works

@SimonMiaou SimonMiaou mentioned this issue Jan 6, 2016
Closed
@MazizEsa
Copy link

MazizEsa commented Dec 2, 2016 •
edited
Loading

@ches is this suppose to happen when you run on aws ebs?

*edited: I am running it via kubernetes. Tried running as root and also fsGroup 1000 (assuming the newly added "kafka" group is 1000) it returns "kafka.common.KafkaException: Found directory /data/lost+found, 'lost+found' is not in the form of topic-partition"

Any pointers?

@ches
Copy link
Owner

ches commented Dec 5, 2016

Sorry for the delayed response @MazizEsa, for anyone else that different lost+found issue has a response on #28.

@imjuoy
Copy link

imjuoy commented Nov 21, 2019

I was getting this error because /kafka/data was not assigned to the right group/owner. Had to change the group/owner to Kafka and bam! Kafka was up and running.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@ches @MazizEsa @SimonMiaou @panda87 @imjuoy