Find out the Spark 2.x build options to create a ruable distribution.
Background
Trying to create a Spark cluster from scratch with compiling the source code . It looks to build a cluster, it requires to build a distribution first. Looking at Building a Ruable Distribution of the Spark 2.0.0 document and run the command described as the example, but got the error.
I suppose -Psparkr is the option to include R support and the build is checking if R is installed as in Github check-cran.sh shell script.
Question
- Where/how can I find out the build options and their explanations?
- Do I need to install R and Python to create a distribution?
(dropping -Psparkr eliminated the error, but not sure it is the correct way)
Command
./dev/make-distribution.sh --name custom-spark --tgz -Psparkr -Phadoop-2.7 -Phive -Phive-thriftserver -Pyarn
Error
...
Caot find 'R_HOME'. Please specify 'R_HOME' or make sure R is properly installed.
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 55.318 s (Wall Clock)
[INFO] Finished at: 2016-09-16T17:32:34+10:00
[INFO] Final Memory: 52M/866M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:exec (sparkr-pkg) on project spark-core_2.11: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :spark-core_2.11
Environment
Ubuntu 14.04
Scala 2.11.8
Oracle JDK 8
Build
Cloned the git repository and run the command.
[Q] Is there a way to specify a specific version of Spark to build with the cloned repository?
git clone git://github.com/apache/spark.git