back soft

امروزدخترم رابردم دندانپزشکی .یکی ازهمکاران قدیمی به اسم خانم شریفی رادیدم خوشحال شدم ،راستش برای من فرقی نمی کنه که خاطره خوب ازهمکارام داشته باشم یانه وقتی بعدازمدتهااونهارامی بینم خوشحال میشم،بعدرف, ...ادامه مطلب

امروزدخترم رابردم دندانپزشکی .یکی ازهمکاران قدیمی به اسم خانم شریفی رادیدم خوشحال شدم ،راستش برای من فرقی نمی کنه که خاطره خوب ازهمکارام داشته باشم یانه وقتی بعدازمدتهااونهارامی بینم خوشحال میشم،بعدرف, ...ادامه مطلب

امروزدخترم رابردم دندانپزشکی .یکی ازهمکاران قدیمی به اسم خانم شریفی رادیدم خوشحال شدم ،راستش برای من فرقی نمی کنه که خاطره خوب ازهمکارام داشته باشم یانه وقتی بعدازمدتهااونهارامی بینم خوشحال میشم،بعدرف, ...ادامه مطلب

امروزدخترم رابردم دندانپزشکی .یکی ازهمکاران قدیمی به اسم خانم شریفی رادیدم خوشحال شدم ،راستش برای من فرقی نمی کنه که خاطره خوب ازهمکارام داشته باشم یانه وقتی بعدازمدتهااونهارامی بینم خوشحال میشم،بعدرف, ...ادامه مطلب

امروزدخترم رابردم دندانپزشکی .یکی ازهمکاران قدیمی به اسم خانم شریفی رادیدم خوشحال شدم ،راستش برای من فرقی نمی کنه که خاطره خوب ازهمکارام داشته باشم یانه وقتی بعدازمدتهااونهارامی بینم خوشحال میشم،بعدرف, ...ادامه مطلب

Vote count: 0I was getting below error for one of my spark job -Task not serializable: java.io.NotSerializableException: java.lang.reflect.FieldI realised that I had a class in one of the closures which was using keeping a java.lang.re, ...ادامه مطلب

Vote count: 0In a Spark standalone cluster, what is exactly the role of the master (a node started with start_master.sh script)?I understand that is the node that receives the jobs from the submit-job.sh script, but what is its role when, ...ادامه مطلب

Vote count: 0I am trying to obtain a cartesian product of 1.5 million records(7 columns) with itself,I am currently using this configuration "./pyspark --master ya-client --num-executors 50 --name "LND_TEST" --conf "spark.executor.cores=2" --conf "spark.executor.memory=14g" --conf "s, ...ادامه مطلب

Vote count: 0 We have the use case in Spark where we want to load historical data from our database to Spark and keep on adding new streaming data to Spark, then we can do analysis on the whole up-to-date dataset. As far as I know, neither Spark SQL nor Spark Streaming can combine the historical data with the streaming data. Then I found the Structured Streaming in Spark 2.0 which seems to be built for this problem. But after some experiment, I still cannot figure it out. Here are my codes: SparkSession spark = SparkSession .builder() .config(conf) .getOrCreate(); JavaSparkContext jsc = new JavaSparkContext(spark.sparkContext()); // Load historical data from MongoDB JavaMongoRDD<Document> mongordd = MongoSpark.load(jsc); // Create typed dataset with customized schema JavaRDD<JavaRecordForSingleTick> rdd = mongordd.flatMap(new FlatMapFunction<Document, JavaRecordForSingleTick>() {...}); Dataset&, ...ادامه مطلب

Vote count: 0 Find out the Spark 2.x build options to create a runnable distribution. Background Trying to create a Spark cluster from scratch with compiling the source code . It looks to build a cluster, it requires to build a distribution first. Looking at Building a Runnable Distribution of the Spark 2.0.0 document and run the command described as the example, but got the error. I suppose -Psparkr is the option to include R support and the build is checking if R is installed as in Github check-cran.sh shell script. Question Where/how can I find out the build options and their explanations? Do I need to install R and Python to create a distribution?(dropping -Psparkr eliminated the error, but not sure it is the correct way) Command ./dev/make-distribution.sh --name custom-spark --tgz -Psparkr -Phadoop-2.7 -Phive -Phive-thriftserver -Pya Error ... Cannot find 'R_HOME'. Please specify 'R_HOME' or make sure R is properly installed. ... [INFO] -------------------------------, ...ادامه مطلب

Vote count: 0 The default metadata store can cause all kinds of interesting problems for concurrent users running jobs against the server; it's an interesting choice for amazon to use it at all (especially since they use a local mysql as the metadata store for hive on other deployments) So I'm wondering: Is it feasible (and recommended?) to get a 5.0 emr deployment of spark up to par (that is, install mysql locally on the master node and use it as the metadata store), without killing the cluster in the process? (This is a relevant question, since it appears that many services and libraries are installed at very specific versions to result in a stable cluster) If it is feasible/recommended, has any one publicly posted a script or tested instructions that will achieve this? asked 1 min agoblueberryfields Let's block ads! بخوانید,is it feasible,is it possible to fly,is it possible to be allergic to weed,is it possible to overdose on weed,is it possible to get pregnant on birth control,is it possible to get taller,is it possible to ovulate early,is it possible to shrink,is it possible to have powers,is it possible to grow taller ...ادامه مطلب

Vote count: 0 Can I have a shared object between executors on the same worker? For example, I have some data in files. This data should be compiled before I can use it. After compilation it is not-serializable Java object. But unfortunately compilation takes a lot of time. I want to compile it once on each worker and use this object in all tasks on the worker. Could you give me some advice how I can achieve this? Thanks. asked 1 min agoMarat Kamalov Let's block ads! بخوانید,spark huge,huge plumper spark ...ادامه مطلب

Vote count: 0 val manager=sc.textFile ("hdfs://localhost:54310/user/training/employee_dir/employeeManager") manager.first ERROR: java.io.EOFException: End of File Exception between local host is: "localhost.localdomain/127.0.0.1"; destination host is: "localhost":54310; :java.io.EOFException; For more details see:http://wiki.apache.org/hadoop/EOFException asked 32 secs agoSridhar Let's block ads! بخوانید,error while creating pse,error while creating database accessor,error while creating directories invalid argument,error while creating ks from kcmo,error while creating table 'edisegment' entry,error while creating the asio driver,error while creating message adobe reader,error while creating report jacoco,error while creating maven project in eclipse,error while creating module org apache subversion javahl clientexception ...ادامه مطلب

Vote count: 0 I have to compare coordinates in order to get the distance. Therefor i load the data with sc.textFile() and make a cartesian product. There are about 2.000.000 lines in the textfile thus 2.000.000 x 2.000.000 to be compared coordinates. I tested the code with about 2.000 coordinates and it worked fine within seconds. But using the big file it seems to stop at a certain point and i don't know why. The code looks as follows: def concat(x,y): if(isinstance(y, list)&(isinstance(x,list))): retu x + y if(isinstance(x,list)&isinstance(y,tuple)): retu x + [y] if(isinstance(x,tuple)&isinstance(y,list)): retu [x] + y else: retu [x,y] def haversian_dist(tuple): lat1 = float(tuple[0][0]) lat2 = float(tuple[1][0]) lon1 = float(tuple[0][2]) lon2 = float(tuple[1][2]) p = 0.017453292519943295 a = 0.5 - cos((lat2 - lat1) * p)/2 + cos(lat1 * p) * cos(lat2 * p) * (1 - cos((lon2 - lon1) * p)) / 2 print(tuple[0][1]) retu (int(float(tuple[0][1])), (int(float(tuple[1][1])),12742 * asin(sqrt(a)))) def sort_val(tuple): dtype = [("globalid", int),("distance",float)] a = np.array(tuple[1], dtype=dtype) sorted_mins = np.sort(a, order="distance",kind="mergesort") retu (tuple[0], sorted_mins) def calc_matrix(sc, path, rangeval, savepath, name): data = sc.textFile(path) data = data.map(lambda x: x.split(";")) matrix = data.cartesian(data) values = matrix.map(haversian_dist) values = values.reduceByKey(concat) values = values.map(sort_val) values = values.map(lambda x: (x[0], x[1][1:int(rangeval)].tolist())) values = values.map(lambda x: (x[0], [y[0] for y in x[1]])) dicti = values.collectAsMap() hp.save_pickle(dicti, savepath, name) Even a file with about 15.000 entries doesn't work. I know the cartesian causes O(n^2) runtime. But shouldn't spark handle this? Or is something wrong? The only starting point is a error message, but i d,spark cartesian product,spark rdd cartesian product ...ادامه مطلب

Vote count: 0 I recently did a laravel spark update (I hadn't updated for some time). Now, I have a registration problem. The first time I hit register, I get the error: Something went wrong. Please try again or contact customer support. However, the next time I hit register (without changing anything), it registers without a hitch...weird. Here is my registration form code in the /resources/views/vendor/spark/auth/register-common-form.blade.php: <form class="form-horizontal" role="form"> <!-- Team Name --> @if (Spark::usesTeams()) <div class="form-group" :class="{'has-error': registerForm.errors.has('team')}" v-if=" ! invitation"> <label class="col-md-4 control-label">Team Name</label> <div class="col-md-6"> <input type="name" class="form-control" name="team" v-model="registerForm.team" autofocus> <span class="help-block" v-show="registerForm.errors.has('team')"> @{{ registerForm.errors.get('team') }} </span> </div> </div> @endif <!-- Name --> <div class="form-group" :class="{'has-error': registerForm.errors.has('name')}"> <label class="col-md-4 control-label">Name</label> <div class="col-md-6"> <input type="name" class="form-control" name="name" v-model="registerForm.name" autofocus> <span class="help-block" v-show="registerForm.errors.has('name')"> @{{ registerForm.errors.get('name') }} </span> </div> </div> <!-- E-Mail Address --> <div class="form-group" :class="{'has-error': registerForm.errors.has('email')}"> <label class="col-md-4 control-label">E-Mail Address</label> <div class="col-md-6"> <input type="email" class="form-control" name=", ...ادامه مطلب