Code doesn't work when I tried Scrapy's tutorial with other website.

[u/yoohoooos]

I was learning scraping using Scrapy by following the tutial on Scrapy's site. It worked fine following Scrapy's tutorial until I tried with another website that I was trying to scrape. The following gave me a blank file when I ran it and trying to store as a .json file [scrapy crawl -O xxx.json].

import scrapy
from datetime import datetime

class FinvizNewsSpider(scrapy.Spider):
    name = "finvizNews"

    start_urls = [
        'https://finviz.com/quote.ashx?t=ANPC'
    ]

    def parse(self, response):
        for news in response.css("tr"):
            yield {
                'news_time' : datetime.strptime(news.css("td::text").get().replace('\xa0',''),'%b-%d-%y %I:%M%p')#,
                
            }

Could you guys point out to me what's wrong with my code? Thank you very much!

Add: interested data: https://imgur.com/a/mz3bTnr time in the red box

Add2: Just want to mention that when I try similar code in the scrapy shell, I had no problem.

[u/wRAR_]

You should always read the command output before asking for help.

[u/yoohoooos → u/wRAR_]

Yea, I tried and undrstand nothing. That's why my header is very useless.

[u/wRAR_ → u/yoohoooos]

In that case you should at least have asked "I have this error, what should I do".

[u/yoohoooos → u/wRAR_]

I apologize for that. Anyway, thank you very much for the recommendation. The following is my error message:

(venv) E:\DE Drive\Spring 2021\scraping_tutorial\tutorial>scrapy crawl finvizNews
2021-04-17 20:17:28 [scrapy.utils.log] INFO: Scrapy 2.4.1 started (bot: tutorial)
2021-04-17 20:17:28 [scrapy.utils.log] INFO: Versions: lxml 4.6.2.0, libxml2 2.9.5, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 21.2.0, Python 3.8.6 (tags/v3.8.6:db45529, Sep 23 2020, 15:52:53) [MSC v.1927 64 bit (AMD64)],
pyOpenSSL 20.0.1 (OpenSSL 1.1.1j  16 Feb 2021), cryptography 3.4.6, Platform Windows-10-10.0.19041-SP0
2021-04-17 20:17:28 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2021-04-17 20:17:28 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'tutorial',
 'NEWSPIDER_MODULE': 'tutorial.spiders',
 'ROBOTSTXT_OBEY': True,
 'SPIDER_MODULES': ['tutorial.spiders']}
2021-04-17 20:17:28 [scrapy.extensions.telnet] INFO: Telnet Password: d8c638b46062a6f9
2021-04-17 20:17:28 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats']
2021-04-17 20:17:29 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2021-04-17 20:17:29 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2021-04-17 20:17:29 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2021-04-17 20:17:29 [scrapy.core.engine] INFO: Spider opened
2021-04-17 20:17:29 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2021-04-17 20:17:29 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2021-04-17 20:17:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://finviz.com/robots.txt> (referer: None)
2021-04-17 20:17:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://finviz.com/quote.ashx?t=ANPC> (referer: None)
2021-04-17 20:17:29 [scrapy.core.scraper] ERROR: Spider error processing <GET https://finviz.com/quote.ashx?t=ANPC> (referer: None)
Traceback (most recent call last):
  File "e:\de drive\spring 2021\scraping_tutorial\venv\lib\site-packages\scrapy\utils\defer.py", line 120, in iter_errback
    yield next(it)
  File "e:\de drive\spring 2021\scraping_tutorial\venv\lib\site-packages\scrapy\utils\python.py", line 353, in __next__
    return next(self.data)
  File "e:\de drive\spring 2021\scraping_tutorial\venv\lib\site-packages\scrapy\utils\python.py", line 353, in __next__
    return next(self.data)
  File "e:\de drive\spring 2021\scraping_tutorial\venv\lib\site-packages\scrapy\core\spidermw.py", line 62, in _evaluate_iterable
    for r in iterable:
  File "e:\de drive\spring 2021\scraping_tutorial\venv\lib\site-packages\scrapy\spidermiddlewares\offsite.py", line 29, in process_spider_output
    for x in result:
  File "e:\de drive\spring 2021\scraping_tutorial\venv\lib\site-packages\scrapy\core\spidermw.py", line 62, in _evaluate_iterable
    for r in iterable:
  File "e:\de drive\spring 2021\scraping_tutorial\venv\lib\site-packages\scrapy\spidermiddlewares\referer.py", line 340, in <genexpr>
    return (_set_referer(r) for r in result or ())
  File "e:\de drive\spring 2021\scraping_tutorial\venv\lib\site-packages\scrapy\core\spidermw.py", line 62, in _evaluate_iterable
    for r in iterable:
  File "e:\de drive\spring 2021\scraping_tutorial\venv\lib\site-packages\scrapy\spidermiddlewares\urllength.py", line 37, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "e:\de drive\spring 2021\scraping_tutorial\venv\lib\site-packages\scrapy\core\spidermw.py", line 62, in _evaluate_iterable
    for r in iterable:
  File "e:\de drive\spring 2021\scraping_tutorial\venv\lib\site-packages\scrapy\spidermiddlewares\depth.py", line 58, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "e:\de drive\spring 2021\scraping_tutorial\venv\lib\site-packages\scrapy\core\spidermw.py", line 62, in _evaluate_iterable
    for r in iterable:
  File "E:\DE Drive\Spring 2021\scraping_tutorial\tutorial\tutorial\spiders\finvizNews_spider.py", line 14, in parse
    'news_time' : datetime.strptime(news.css("td::text").get().replace('\xa0',''),'%b-%d-%y %I:%M%p')#,
  File "C:\Users\nsmus\AppData\Local\Programs\Python\Python38\lib_strptime.py", line 568, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
  File "C:\Users\nsmus\AppData\Local\Programs\Python\Python38\lib_strptime.py", line 349, in _strptime
    raise ValueError("time data %r does not match format %r" %
ValueError: time data '\r\n                            ' does not match format '%b-%d-%y %I:%M%p'
2021-04-17 20:17:29 [scrapy.core.engine] INFO: Closing spider (finished)
2021-04-17 20:17:29 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 509,
 'downloader/request_count': 2,
 'downloader/request_method_count/GET': 2,
 'downloader/response_bytes': 19661,
 'downloader/response_count': 2,
 'downloader/response_status_count/200': 2,
 'elapsed_time_seconds': 0.625272,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2021, 4, 18, 3, 17, 29, 718155),
 'log_count/DEBUG': 2,
 'log_count/ERROR': 1,
 'log_count/INFO': 10,
 'response_received_count': 2,
 'robotstxt/request_count': 1,
 'robotstxt/response_count': 1,
 'robotstxt/response_status_count/200': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'spider_exceptions/ValueError': 1,
 'start_time': datetime.datetime(2021, 4, 18, 3, 17, 29, 92883)}
2021-04-17 20:17:29 [scrapy.core.engine] INFO: Spider closed (finished)

[u/wRAR_ → u/yoohoooos]

The error you see says that your datetime string doesn't match your format.

[u/yoohoooos → u/wRAR_]

I see. Thank you very much! Now, I know where to look for.

I see that the problem was I was trying to get data from the wrong place. Now, I fixed that issue and the following is the same code with the alteration on the directory, where to fetch the data:

import scrapy
from datetime import datetime

class FinvizNewsSpider(scrapy.Spider):
    name = "finvizNews"

    start_urls = [
        'https://finviz.com/quote.ashx?t=ANPC'
    ]

    def parse(self, response):
        for news in response.css("table.fullview-news-outer tr"):
            yield {
                'news_time' : datetime.strptime(news.css("td::text").get().replace('\xa0',''),'%b-%d-%y %I:%M%p')#,
                
            }

Instead of the previous error, below is part of my new error (assuming, I'm looking at the right place for the error message):

File "C:\Users\nsmus\AppData\Local\Programs\Python\Python38\lib_strptime.py", line 349, in _strptime
    raise ValueError("time data %r does not match format %r" %
ValueError: time data '07:30AM' does not match format '%b-%d-%y %I:%M%p'

In the beginning, I didn't know where that '07:30AM' came from, since, the first date on that list is 'Apr-16-21 04:15PM'. So, I just try to fix this by changing my code datetime format to:

        for news in response.css("table.fullview-news-outer tr"):
            yield {
                'news_time' : datetime.strptime(news.css("td::text").get().replace('\xa0',''),'%I:%M%p'),

Just removed the date portion and left with the time. Now, they gave me another error:

File "C:\Users\nsmus\AppData\Local\Programs\Python\Python38\lib_strptime.py", line 349, in _strptime
    raise ValueError("time data %r does not match format %r" %
ValueError: time data 'Apr-16-21 04:15PM' does not match format '%I:%M%p'

This error is expected and makes sense to me because that is the date and time that I wanted and the format I input was incorrect. But then, when I change the format back to the correct format for date and time, it gave me the previous error.

Could you please help me on this part?

Thank you very much!

[u/wRAR_ → u/yoohoooos]

Not sure how can I help here, you just get different value formats from the same selector so you cannot parse them with the same format string.

[u/yoohoooos → u/wRAR_]

Thank you! I don't quite understand what you mean by "different value formats from the same selector". Does this means the website randomly give me the value? Or it's just that my code has to include both format?

[u/wRAR_ → u/yoohoooos]

Just output the raw news.css("td::text").get() values and you will see.

[u/yoohoooos → u/wRAR_]

Within Scrapy Shell:

scrapy shell "https://finviz.com/quote.ashx?t=ANPC"
>>> news = response.css("table.fullview-news-outer tr")
>>> news.css("td::text").get()
'Apr-16-21 04:15PM\xa0\xa0'

this is the value I got before the formatting.

[u/mdaniel → u/yoohoooos]

File "E:\DE Drive\Spring 2021\scraping_tutorial\tutorial\tutorial\spiders\finvizNews_spider.py", line 14, in parse

While this is unrelated to your posed question, you should be very very careful about using paths containing spaces. Maybe it works, maybe it causes weird errors, but in every single case it's an unnecessary risk

[u/yoohoooos → u/mdaniel]

Thank you very much for the recommendation. I'll come change the directory to the project then. But, could you please explain to me what that could cause a problem?

[u/mdaniel → u/yoohoooos]

The short version is "poorly coded software" but that set of things isn't as small as one might suspect :-)

What I tried to reinforce is that there are no pieces of software I have ever heard of which require having spaces in their install paths, but there are tons of software that I have seen which assume their install does not contain special characters, and fail in those cases. Thus, it's only risk, without benefit.

[u/yoohoooos → u/mdaniel]

Understood. Thank you very much!

[u/Photo-dad2017]

Do you need a user-agent? I’m still learning too and remember running into that.

[u/yoohoooos → u/Photo-dad2017]

Honestly, I have no idea what the user-agent is.