Блог

Я пытаюсь написать паук для usnews.com , код и ошибка приведены ниже.

 class USMethodologySpider(scrapy.Spider):
    """
    Парсинг методологии  US
    """
    name = 'US_Methodology_Spider'
    def start_requests(self):
        headers = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.76 Safari/537.36',
            "Upgrade-Insecure-Requests": "1", "DNT": "1",
            "Accept": "text/html,application/xhtml xml,application/xml;q=0.9,*/*;q=0.8",
            "Accept-Language": "en-US,en;q=0.5",
            "Accept-Encoding": "gzip, deflate"}
        yield scrapy.Request(url='http://www.usnews.com/education/best-global-universities/rankings', callback=self.parse, headers=headers)

    def parse(self):
        logging.info(response.body)
 

журнал консоли:

 2020-11-23 13:46:18 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2020-11-23 13:46:18 [scrapy.core.engine] INFO: Spider opened
2020-11-23 13:46:18 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-11-23 13:46:18 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-11-23 13:47:18 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-11-23 13:48:18 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-11-23 13:49:18 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-11-23 13:49:18 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.usnews.com/education/best-global-universities/rankings> (failed 1 times): User timeout caused connection failure: Getting http://www.usnews.com/ed
ucation/best-global-universities/rankings took longer than 180.0 seconds..
2020-11-23 13:50:15 [scrapy.crawler] INFO: Received SIGINT, shutting down gracefully. Send again to force
2020-11-23 13:50:15 [scrapy.core.engine] INFO: Closing spider (shutdown)
2020-11-23 13:50:18 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-11-23 13:51:18 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-11-23 13:52:18 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-11-23 13:52:18 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.usnews.com/education/best-global-universities/rankings> (failed 2 times): User timeout caused connection failure: Getting http://www.usnews.com/ed
ucation/best-global-universities/rankings took longer than 180.0 seconds..
 

У меня есть доступ к этому сайту через браузер, и я также могу подключиться к нему через запросы python с теми же заголовками.

Возможно, новостной сайт заблокировал любое сканирование веб-страниц. Это то же самое, что и Google News, его нелегко сканировать.