خرید بک لینک

Vote count: 0

It doesn't look like I have wired my web crawler (built using Python's Scrapy) properly with my Node application. I have installed python-shell module and followed the basic instructions, but the response I get back on the client side is null.

The web crawler will work if I run the script in Python's shell, but when I use python-shell to run the script in my Node application, it retus null.

It passes the right http link all the way through the express route and into the crawler.

I'm unclear what I'm missing/not syncing properly. The URL I'm trying to scrape is http://www.c.com/2016/07/17/politics/john-kerry-interview-turkey-nice/index.html.

router.post('/article', function(req, res){
  var articleUrl = '';
  articleUrl = req.body.content
  // script options
  var options = {
    mode: 'json',
    pythonPath: '/usr/local/bin/python3',
    pythonOptions: ['-u'],
    scriptPath: 'path/to/script',
    args: [articleUrl]
  };


  PythonShell.run('/crawler/spiders/exteal_spider.py', options, (err, results)=>{
    if(err) throw err;

    console.log("============")
    console.log("================")
    console.log("these are the results from the crawler")
    console.log('results: %j', results); // retus `data: null`
    console.log("================")

    res.json(results)
  })

})

#!/usr/local/bin/python3

import scrapy
from scrapy.crawler import CrawlerProcess

class SnapshotItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    title = scrapy.Field()
    snippet = scrapy.Field()
    image = scrapy.Field()
    link = scrapy.Field()


# should take extra **kwargs
# to pass in the `url` posted
# by the User
class ExtealSpider(scrapy.Spider):

    name = "exteal"

    def __init__(self, *args, **kwargs):
        super(ExtealSpider, self).__init__(*args, **kwargs)
        print("==========")
        print(args[0])
        print("==========")
        self.start_urls=['%s' % args[0]] 

    allowed_domains = ["c.com"]  # don't restrict the domains


    def parse(self, response):

        item = SnapshotItem()


        item['title'] = response.selector.xpath("//meta[@property='og:title']/attribute::content").extract()[0]
        item['snippet'] = response.selector.xpath("//meta[@property='og:description']/attribute::content").extract()[0]
        item['image'] = response.selector.xpath("//meta[@property='og:image']/attribute::content").extract()[0]
        item['link'] = response.selector.xpath("//meta[@property='og:url']/attribute::content").extract()[0]

        # print(item)
        retu item

asked 38 secs ago

برچسب: نویسنده: استخدام کار تاريخ: شنبه 16 مرداد 1395 ساعت: 13:15

صفحه بندی