خرید بک لینک

Vote count: 0

I am trying to use cascading to generate a giant table that matches url's to other attributes (i.e, traffic on that page, size of that page etc).

I have a list of tsv files that look like this:

file1: url -> traffic
file2: url -> size of page, bad words on website
file3: url

I have a list of buckets, where each bucket has these format.

MyBucket{
    String inputPath = http://... (to get the file1, file 2 etc)
    int keyColumn = 1
    int valueColumn = 2
    boolean TypeColumn = false;
    boolean multivalue = false;
}

Here, I have their path (for download).

Their key Column and value Column (the key column is where the urls would be in the file, and the value column is where the item I am interested in is). (for file 1, key would equal 1 for url column and value would equal 2 for traffic column)

multivalue defines if there are multiple values for the column i.e. file2: bad words on website have multiple words associated with them (3 multivalue items) TypeColumn essentially is to check if we are appending a boolean symbol on.

If the TypeColumn is a boolean (such as would be for file3), we want to check our merged table, and any url that appears in the totalFile and in file3 would be marked as true, the rest as false.

So far, my logic is as follows:

1. Create tap for each source
2. Generate a Pipe 

    int size = bucketList.size();
    Pipe[] pipeArr = new pipeArr[size];
    for (Bucket b : bucketList) {
    for (int i = 0; i < size; i++) {
        Bucket b = bucketList.get();
        String inputPath = b.getInputPath();
        int keyColumn = b.getKeyColumn();
        int valueColumn = b.getValueColumn();
        Boolean typeColumn = b.isTypeColumn();
        Boolean multivalue = b.isMultivalue();
        Fields selector = Fields.resolve(Fields.ALL, new Fields("url"), new Fields("value"));
        Tap currentTap = new Hfs(new TextDelimited(selector), outputFile);
        Pipe myPipe = new Pipe(b.getName());
        pipeArr[size] = myPipe;
        ...
        ...
    }

3. For each of the myPipes, I would append a each pipe to do some logic (select the key and value, check if it is a multivalue, if it is a typeColumn)...
4. GroupBy all of the pipes into one single pipe that would pump out all the data
5. Have an every pipe, such that I can aggregate the lines such that url -> [list of attributes] and publish a bunch of jsons. 

I am having serious issues with the pipe logic of step 3 and 4. I know that it should hopefully be a very simple pipe, but I'm not quite sure how the function I push inside the each pipe would be able to deal with these parameters. Also, I don't quite know how cascading would deal with the TypeColumn issue, as if it is a boolean, how do I go through the other urls and mark them as false? Is that even possible?

Thank you for any help!

asked 57 secs ago

برچسب: نویسنده: استخدام کار تاريخ: پنجشنبه 20 خرداد 1395 ساعت: 1:35

صفحه بندی