Ruing spark word Count program in eclipse using sbt build tool. Unable to coect to remote cloudera spark master ruing virtual box. I have used following code.
The project structure is as follows.
WordCount.scala is as follows
package greeter
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
object HelloSpark {
def main(args: Array[String]) {
val logFile = "file:C:\Users\Sat\Desktop\regex_sum_230158.txt"
val conf = new SparkConf().setAppName("Simple Application").setMaster("spark://127.0.0.1:7077")
val sc = new SparkContext(conf)
val logData = sc.textFile(logFile)
val numAs = logData.filter(line => line.contains("a")).count()
val numBs = logData.filter(line => line.contains("b")).count()
println("Lines with a: %s, Lines with b: %s".format(numAs, numBs))
}
}
build.sbt is as follows
name := "Hello SparkApp"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "1.5.2"
Please let me know how to coect spark master runing in virtual box.