Skip to content
This repository has been archived by the owner on Nov 23, 2017. It is now read-only.

add java 8 to image creation script; remove duplication #12

Open
wants to merge 1 commit into
base: branch-1.5
Choose a base branch
from

Conversation

denbkh
Copy link

@denbkh denbkh commented Sep 24, 2015

No description provided.

@nchammas
Copy link
Contributor

This is a neat PR.

@shivaram - What is the likelihood that a new set of spark-ec2 AMIs will ever be generated? Do we want to mark files like create_image.sh as "for the record" only, or do we want to actively maintain them?

@shivaram
Copy link
Contributor

I think we will do a round of AMI generation for Spark 2.0 -- this will give us a chance to upgrade to Java 8, Scala 2.11 etc. Does this sound good ?

@douglaz
Copy link
Contributor

douglaz commented Aug 18, 2016

Any plans on this?

@shivaram
Copy link
Contributor

Unfortunately I haven't had the time to get around to this yet. I will try to get to it soon or check if there is somebody else interested in helping out with this.

@pcandido
Copy link

Hi, I've tested it, but didn't work for me. It seems the 'create_image.sh' file wasn't run on any node. Maybe it needs to be called.
I made config manually. I had to:

  • install j8
  • config alternatives
  • set JAVA_HOME in .bash_profile

@douglaz
Copy link
Contributor

douglaz commented Jan 12, 2017

@pcandido, this won't run automatically. You manually need to:

  1. Launch a machine in Amazon
  2. Login, then run the script
  3. Create an AMI (Amazon Machine Image) from this instance
  4. Then, when you want to launch your cluster, specify this new AMI

Of course, someone can do this process, create a public AMI and make it the default in the spark-ec2 scripts. This is what is really missing here.

@nchammas
Copy link
Contributor

The way we solved this in Flintrock is to dynamically install Java 8 at launch time if it isn't detected on the instance. That avoids the lengthy process of needing to create new AMIs and, at least for Flintrock, it didn't add much time to the launch.

@shivaram
Copy link
Contributor

Thats very useful to know @nchammas - In that case I'd be fine with a change to setup-slave.sh to do this in spark-ec2

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants