خرید بک لینک

Vote count: 0

I have a K,V DataFrame 'animals_fruits' which looks like this:

DataFrame[animal_id: bigint, fruit_id: bigint]. 

The purpose is matching animals with foods that they eat.

I want to build a binary matrix using the derived vectors from this matrix. E.G

[animal_id:1, fruit_id:3] 

0 0 0
0 0 0
1 0 0 

I've defined the following arrays to iterate through:

a = animal_fruits.select(animal_fruits.animal_id).map(lambda v: (v.animal_id))
b = animal_fruits.select(animal_fruits.fruit_id).map(lambda v: (v.fruit_id)

Here's some sample output: a.take(5) [36566, 41366, 30620, 64820, 30755]

Now, I'd like iterate through animals_fruits and take each value of a,b as coordinates for the matrix:

def fruit_mapper(a,b):
    mat = CoordinateMatrix(sc.parallelize([MatrixEntry(a,b, 1)]))
animals_fruits.foreach(fruit_mapper)

However this gives me the following exception:

Exception: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers.

For testing purposes, I've tried ruing just the mat = Coordinatematrix... line inside the REPL and came with the following error:

long() argument must be a string or a number, not 'PipelinedRDD'

So it's pretty clear that the coordinate matrix doesn't know how to handle being passed an array of values (at least not in this way). Does anyone have any suggestions of how to iterate through that DF and creating the matrix I'm after?

Much appreciated.

asked 7 mins ago

برچسب: نویسنده: استخدام کار تاريخ: پنجشنبه 27 خرداد 1395 ساعت: 10:30

صفحه بندی