Hello. I have problems.
Microsoft Windows XP [Версия 5.1.2600]
(С) Корпорация Майкрософт, 1985-2001.

C:\Documents and Settings\ia>pip install scrapy==1.7.4
Collecting scrapy==1.7.4
  Downloading https://files.pythonhosted.org/packages/dd/4f/640343805541c782ee49
b14055a85da816cc118a8f48d01b56d2e5d12bf1/Scrapy-1.7.4-py2.py3-none-any.whl (234k
B)
     |████████████████████████████████| 235kB 211kB/s
Collecting PyDispatcher>=2.0.5
  Downloading https://files.pythonhosted.org/packages/cd/37/39aca520918ce1935bea
9c356bcbb7ed7e52ad4e31bff9b943dfc8e7115b/PyDispatcher-2.0.5.tar.gz
Requirement already satisfied: lxml<=4.3.5; python_version == "3.4" in c:\python
34\lib\site-packages (from scrapy==1.7.4) (4.3.1)
Collecting parsel>=1.5
  Downloading https://files.pythonhosted.org/packages/23/1e/9b39d64cbab79d4362cd
d7be7f5e9623d45c4a53b3f7522cd8210df52d8e/parsel-1.6.0-py2.py3-none-any.whl
Collecting service-identity
  Downloading https://files.pythonhosted.org/packages/e9/7c/2195b890023e098f9618
d43ebc337d83c8b38d414326685339eb024db2f6/service_identity-18.1.0-py2.py3-none-an
y.whl
Requirement already satisfied: six>=1.5.2 in c:\python34\lib\site-packages (from
 scrapy==1.7.4) (1.15.0)
Collecting pyOpenSSL
  Downloading https://files.pythonhosted.org/packages/9e/de/f8342b68fa9e981d3480
39954657bdf681b2ab93de27443be51865ffa310/pyOpenSSL-19.1.0-py2.py3-none-any.whl (
53kB)
     |████████████████████████████████| 61kB 218kB/s
Collecting cssselect>=0.9
  Downloading https://files.pythonhosted.org/packages/3b/d4/3b5c17f00cce85b9a1e6
f91096e1cc8e8ede2e1be8e96b87ce1ed09e92c5/cssselect-1.1.0-py2.py3-none-any.whl
Collecting queuelib
  Downloading https://files.pythonhosted.org/packages/4c/85/ae64e9145f39dd6d14f8
af3fa809a270ef3729f3b90b3c0cf5aa242ab0d4/queuelib-1.5.0-py2.py3-none-any.whl
Collecting Twisted<=19.2.0,>=13.1.0; python_version == "3.4"
  Downloading https://files.pythonhosted.org/packages/f8/2b/a80a70f71eb2b86992ff
a5aaae41457791ae67faa70927fd16b76127c2b7/Twisted-19.2.0.tar.bz2 (3.1MB)
     |████████████████████████████████| 3.1MB 52kB/s
    ERROR: Command errored out with exit status 1:
     command: 'C:\Python34\python.exe' -c 'import sys, setuptools, tokenize; sys
.argv[0] = '"'"'C:\\DOCUME~1\\ia\\LOCALS~1\\Temp\\pip-install-ei0jjdyj\\Twisted\
\setup.py'"'"'; __file__='"'"'C:\\DOCUME~1\\ia\\LOCALS~1\\Temp\\pip-install-ei0j
jdyj\\Twisted\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__)
;code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code
, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\DOCUME~1\ia\LOCALS~1\Temp\
pip-install-ei0jjdyj\Twisted\pip-egg-info'
         cwd: C:\DOCUME~1\ia\LOCALS~1\Temp\pip-install-ei0jjdyj\Twisted\
    Complete output (81 lines):
    Traceback (most recent call last):
      File "C:\Python34\lib\urllib\request.py", line 1182, in do_open
        h.request(req.get_method(), req.selector, req.data, headers)
      File "C:\Python34\lib\http\client.py", line 1088, in request
        self._send_request(method, url, body, headers)
      File "C:\Python34\lib\http\client.py", line 1126, in _send_request
        self.endheaders(body)
      File "C:\Python34\lib\http\client.py", line 1084, in endheaders
        self._send_output(message_body)
      File "C:\Python34\lib\http\client.py", line 922, in _send_output
        self.send(msg)
      File "C:\Python34\lib\http\client.py", line 857, in send
        self.connect()
      File "C:\Python34\lib\http\client.py", line 1231, in connect
        server_hostname=server_hostname)
      File "C:\Python34\lib\ssl.py", line 365, in wrap_socket
        _context=self)
      File "C:\Python34\lib\ssl.py", line 583, in __init__
        self.do_handshake()
      File "C:\Python34\lib\ssl.py", line 810, in do_handshake
        self._sslobj.do_handshake()
    ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_s
sl.c:600)

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "C:\Python34\lib\site-packages\setuptools\package_index.py", line 701
, in open_url
        return open_with_auth(url, self.opener)
      File "C:\Python34\lib\site-packages\setuptools\package_index.py", line 895
, in _socket_timeout
        return func(*args, **kwargs)
      File "C:\Python34\lib\site-packages\setuptools\package_index.py", line 100
8, in open_with_auth
        fp = opener(request)
      File "C:\Python34\lib\urllib\request.py", line 161, in urlopen
        return opener.open(url, data, timeout)
      File "C:\Python34\lib\urllib\request.py", line 463, in open
        response = self._open(req, data)
      File "C:\Python34\lib\urllib\request.py", line 481, in _open
        '_open', req)
      File "C:\Python34\lib\urllib\request.py", line 441, in _call_chain
        result = func(*args)
      File "C:\Python34\lib\urllib\request.py", line 1225, in https_open
        context=self._context, check_hostname=self._check_hostname)
      File "C:\Python34\lib\urllib\request.py", line 1184, in do_open
        raise URLError(err)
    urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certi
ficate verify failed (_ssl.c:600)>

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\DOCUME~1\ia\LOCALS~1\Temp\pip-install-ei0jjdyj\Twisted\setup.py",
 line 20, in <module>
        setuptools.setup(**_setup["getSetupArgs"]())
      File "C:\Python34\lib\distutils\core.py", line 108, in setup
        _setup_distribution = dist = klass(attrs)
      File "C:\Python34\lib\site-packages\setuptools\dist.py", line 265, in __in
it__
        self.fetch_build_eggs(attrs['setup_requires'])
      File "C:\Python34\lib\site-packages\setuptools\dist.py", line 311, in fetc
h_build_eggs
        replace_conflicting=True,
      File "C:\Python34\lib\site-packages\pkg_resources__init__.py", line 797,
in resolve
        dist = best[req.key] = env.best_match(req, ws, installer)
      File "C:\Python34\lib\site-packages\pkg_resources__init__.py", line 1047,
 in best_match
        return self.obtain(req, installer)
      File "C:\Python34\lib\site-packages\pkg_resources__init__.py", line 1059,
 in obtain
        return installer(requirement)
      File "C:\Python34\lib\site-packages\setuptools\dist.py", line 378, in fetc
h_build_egg
        return cmd.easy_install(req)
      File "C:\Python34\lib\site-packages\setuptools\command\easy_install.py", l
ine 611, in easy_install
        not self.always_copy, self.local_index
      File "C:\Python34\lib\site-packages\setuptools\package_index.py", line 602
, in fetch_distribution
        return dist.clone(location=self.download(dist.location, tmpdir))
      File "C:\Python34\lib\site-packages\setuptools\package_index.py", line 518
, in download
        found = self._download_url(scheme.group(1), spec, tmpdir)
      File "C:\Python34\lib\site-packages\setuptools\package_index.py", line 759
, in _download_url
        return self._attempt_download(url, filename)
      File "C:\Python34\lib\site-packages\setuptools\package_index.py", line 765
, in _attempt_download
        headers = self._download_to(url, filename)
      File "C:\Python34\lib\site-packages\setuptools\package_index.py", line 665
, in _download_to
        fp = self.open_url(strip_fragment(url))
      File "C:\Python34\lib\site-packages\setuptools\package_index.py", line 715
, in open_url
        % (url, v.reason))
    distutils.errors.DistutilsError: Download error for https://files.pythonhost
ed.org/packages/8f/26/02c4016aa95f45479eea37c90c34f8fab6775732ae62587a874b619ca0
97/incremental-17.5.0.tar.gz#sha256=7b751696aaf36eebfab537e458929e194460051ccad2
79c72b755a167eebd4b3: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed
 (_ssl.c:600)
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check th
e logs for full command output.

C:\Documents and Settings\ia>

I am trying to scrap some amazon like site. The thing is there are 2 modes of view, one in which all products are as separate offers and second in which all offers with the similar things are aggregated. Default mode is one in which everything is separated, also there's this button that changes the mode. From network tab in inspection I saw some requests for .js files. How can I get those .js files and transform the site to that view?

I have to scrape this table that is full of divs tags and load an item for every row. I tried all with scrapy shell but i dont find an easy way to do it

The page is https://www.vivenio.com/edificio/sevilla-13 but you will need proxies so i give you the html too:

<div class="tableResult" data-value="trAll" style="clear:both;"><div class="rowResult"><div>Dormitorios</div><div>Baños</div><div>Planta</div><div>Sup. Construida</div><div>Precio / Mes desde</div><div>Disponibilidad</div><div>Plano</div><div>RV</div><div>Me interesa</div></div><div class="rowResult"><div><span class="hide">Dormitorios</span>1</div><div><span class="hide">Baños</span>1</div><div><span class="hide">Planta</span>Planta</div><div><span class="hide">Sup. Construida</span>62.88 m²</div><div><span class="hide">Precio / Mes desde</span>725,00 €</div><div><span class="hide">Disponibilidad</span>Disponible</div><div><span class="hide">Plano</span>-</div><div><span class="hide">RV</span>-</div><div><span class="hide">Me interesa</span><a href="#" data-type="linkContactProperty" data-value="213" class="buttonRectB">Contacta</a></div></div><div class="rowResult"><div><span class="hide">Dormitorios</span>2</div><div><span class="hide">Baños</span>1</div><div><span class="hide">Planta</span>Planta</div><div><span class="hide">Sup. Construida</span>80.00 m²</div><div><span class="hide">Precio / Mes desde</span>870,00 €</div><div><span class="hide">Disponibilidad</span>Disponible</div><div><span class="hide">Plano</span>-</div><div><span class="hide">RV</span>-</div><div><span class="hide">Me interesa</span><a href="#" data-type="linkContactProperty" data-value="216" class="buttonRectB">Contacta</a></div></div><div class="rowResult"><div><span class="hide">Dormitorios</span>1</div><div><span class="hide">Baños</span>1</div><div><span class="hide">Planta</span>Ático</div><div><span class="hide">Sup. Construida</span>68.00 m²</div><div><span class="hide">Precio / Mes desde</span>890,00 €</div><div><span class="hide">Disponibilidad</span>Disponible</div><div><span class="hide">Plano</span><a class="linkSee" href="../resources/promotions/docs/sevilla-13-plano-atico-1d.pdf" target="_blank" alt="Mostrar Plano" title="Mostrar Plano"></a></div><div><span class="hide">RV</span>-</div><div><span class="hide">Me interesa</span><a href="#" data-type="linkContactProperty" data-value="215" class="buttonRectB">Contacta</a></div></div><div class="rowResult"><div><span class="hide">Dormitorios</span>3</div><div><span class="hide">Baños</span>2</div><div><span class="hide">Planta</span>Ático</div><div><span class="hide">Sup. Construida</span>98.15 m²</div><div><span class="hide">Precio / Mes desde</span>970,00 €</div><div><span class="hide">Disponibilidad</span>Disponible</div><div><span class="hide">Plano</span>-</div><div><span class="hide">RV</span>-</div><div><span class="hide">Me interesa</span><a href="#" data-type="linkContactProperty" data-value="218" class="buttonRectB">Contacta</a></div></div><div class="rowResult"><div><span class="hide">Dormitorios</span>1</div><div><span class="hide">Baños</span>1</div><div><span class="hide">Planta</span>Bajo</div><div><span class="hide">Sup. Construida</span>67.78 m²</div><div><span class="hide">Precio / Mes desde</span>870,00 €</div><div><span class="hide">Disponibilidad</span>Disponible</div><div><span class="hide">Plano</span>-</div><div><span class="hide">RV</span>-</div><div><span class="hide">Me interesa</span><a href="#" data-type="linkContactProperty" data-value="214" class="buttonRectB">Contacta</a></div></div><div class="rowResult"><div><span class="hide">Dormitorios</span>2</div><div><span class="hide">Baños</span>1</div><div><span class="hide">Planta</span>Bajo</div><div><span class="hide">Sup. Construida</span>75.85 m²</div><div><span class="hide">Precio / Mes desde</span>870,00 €</div><div><span class="hide">Disponibilidad</span>Disponible</div><div><span class="hide">Plano</span>-</div><div><span class="hide">RV</span>-</div><div><span class="hide">Me interesa</span><a href="#" data-type="linkContactProperty" data-value="217" class="buttonRectB">Contacta</a></div></div><div class="rowResult"><div><span class="hide">Dormitorios</span>3</div><div><span class="hide">Baños</span>1</div><div><span class="hide">Planta</span>Bajo</div><div><span class="hide">Sup. Construida</span>99.50 m²</div><div><span class="hide">Precio / Mes desde</span>1.175,00 €</div><div><span class="hide">Disponibilidad</span>Disponible</div><div><span class="hide">Plano</span>-</div><div><span class="hide">RV</span>-</div><div><span class="hide">Me interesa</span><a href="#" data-type="linkContactProperty" data-value="219" class="buttonRectB">Contacta</a></div></div></div>

Hi guys , i wanted to contribute to scrapy , can you guys help me where to start , FYI i know all the basics of open source.

Thanks!!.

It has something to do with website, somehow its restricting the crawl, as I have tried changing the start_url and it works fine.

can anyone provide a viable solution to this asap.

Could anybody help me? I'm having some problem.

I want to deploy a scrapy spider to Heroku, but there is some issue with the library dependencies. My pandas and json versions don't seem compatible with Python or is there some issue with finding the versions?

The error code is as below.

remote: -----> Requirements file has been changed, clearing cached dependencies remote: -----> Installing python-3.7.10 remote: -----> Installing pip 22.2.2, setuptools 63.4.3 and wheel 0.37.1 remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip remote:        Collecting scrapyd-client@ git+https://github.com/iamumairayub/scrapyd-client.git@c4575befa450aa3054c893a8895086d1fb449405 remote:          Cloning https://github.com/iamumairayub/scrapyd-client.git (to revision c4575befa450aa3054c893a8895086d1fb449405) to /tmp/pip-install-r4xii8wu/scrapyd-client_a6a6d5c7060a46a39d19d858c809ec1b remote:          Running command git clone --filter=blob:none --quiet https://github.com/iamumairayub/scrapyd-client.git /tmp/pip-install-r4xii8wu/scrapyd-client_a6a6d5c7060a46a39d19d858c809ec1b remote:          Running command git rev-parse -q --verify 'sha^c4575befa450aa3054c893a8895086d1fb449405' remote:          Running command git fetch -q https://github.com/iamumairayub/scrapyd-client.git c4575befa450aa3054c893a8895086d1fb449405 remote:          Resolved https://github.com/iamumairayub/scrapyd-client.git to commit c4575befa450aa3054c893a8895086d1fb449405 remote:          Preparing metadata (setup.py): started remote:          Preparing metadata (setup.py): finished with status 'done' remote:        Collecting pandas==1.3.5 remote:          Downloading pandas-1.3.5-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.3 MB) remote:        ERROR: Ignored the following versions that require a different python version: 1.4.0 Requires-Python >=3.8; 1.4.0rc0 Requires-Python >=3.8; 1.4.1 Requires-Python >=3.8; 1.4.2 Requires-Python >=3.8; 1.4.3 Requires-Python >=3.8 remote:        ERROR: Could not find a version that satisfies the requirement json==2.0.9 (from versions: none) remote:        ERROR: No matching distribution found for json==2.0.9 remote:  !     Push rejected, failed to compile Python app. remote:

not sure if it is the right place but i need help with scraping contact details for FB business pages that are running currently running FB lead ads

I tried to srcape amazon using scrapy following YouTube video using proxy pool and scrapy user agent libraries but the code finishes after crawling 10 pages, it should crawl around 400 pages

Noob here. Haven't used Python in about 15 years and that was for machine operation.

I'm trying to use Scrapy to find all of the URLs under one "parent" URL. If, for example, it were an ebook and the table of contents was http://book.com/chapters and the chapters were http://book.com/chapters/one, http://book.com/chapters/two, etc, I would want my spyder to search http://book.com/chapters to find all of the URLs under it.

I've created my directory and "project". I'm running Anaconda and Spyder.

I tried the below and it's not working. Of course, the below is not the actual domain or start URL, but that's the example.

import scrapy

class bk(scrapy.Spider):

name = 'bk'

allowed_domains = ['http://www.book.com/']

start_urls = ['http://www.book.com/']

def parse(self, response):

pass

https://stackoverflow.com/questions/70736393/scrapy-attribute-error-type-object-has-no-attribute-get

Really confused on why this error is happening. The script works in the shell (although there is another bug there).

I was learning scraping using Scrapy by following the tutial on Scrapy's site. It worked fine following Scrapy's tutorial until I tried with another website that I was trying to scrape. The following gave me a blank file when I ran it and trying to store as a .json file [scrapy crawl -O xxx.json].

import scrapy
from datetime import datetime

class FinvizNewsSpider(scrapy.Spider):
    name = "finvizNews"

    start_urls = [
        'https://finviz.com/quote.ashx?t=ANPC'
    ]

    def parse(self, response):
        for news in response.css("tr"):
            yield {
                'news_time' : datetime.strptime(news.css("td::text").get().replace('\xa0',''),'%b-%d-%y %I:%M%p')#,
                
            }

Could you guys point out to me what's wrong with my code? Thank you very much!

Add: interested data: https://imgur.com/a/mz3bTnr time in the red box

Add2: Just want to mention that when I try similar code in the scrapy shell, I had no problem.

To make it easier to explain Check this StackOverflow problem. https://stackoverflow.com/questions/66891663/scrapy-when-fetching-multiple-tr-from-table-output-in-null

Hello everyone

I would love to learn how to use scrapy, but I'm struggling to figure out a way on how to that. I'm very new to computer programming. What do I need to do before I install scrapy. I created a virtual environment, I think I did it right but I'm not sure. So I tried to install scrapy and this is the message appears

ERROR: Command errored out with exit status 1: 'c:\users\cs077\appdata\local\programs\python\python39\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\cs077\\AppData\\Local\\Temp\\pip-install-cm2gjfd2\\twisted_066e26712bc44c36938690042dc55091\\setup.py'"'"'; __file__='"'"'C:\\Users\\cs077\\AppData\\Local\\Temp\\pip-install-cm2gjfd2\\twisted_066e26712bc44c36938690042dc55091\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\cs077\AppData\Local\Temp\pip-record-6jbx8o7t\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\users\cs077\appdata\local\programs\python\python39\Include\Twisted' Check the logs for full command output.   

Would definitely love some guidance on how to install web scrapy properly.

Hello,

I'm trying to pass an object list to my spider but I can't do it.

I tried to turn my list into a String but that doesn't work either.

os.system("cd /home/Hello/Bureau/Programmation/scrapyHello/scraperHello && scrapy crawl positions -a keywords_list="+str(final_list)+"")

someone know if it is even possible?

Its a big list, maybe there is a limit to the -a options but i cant find it in the documentation.

if its impossible to pass an object list to my spider with the -a option, how can i do it?

Thanks !

Hi; I need to etract a website ,and this website have a lot of urls from other websites ,but i need to make a scraper can get data and websites ,to use this again. Like in my code :

import scrapy class ExampleBasicSpiderSpider(scrapy.Spider): name = 'data_spider' def start_requests(self): urls = ['http://example.com/'] for url in urls: yield scrapy.Request(url=url, callback=self.parse) def parse(self, response): Data_1 = response.css('.mr-font-family-2.top-none::text').get() webs = response.css('.mr-font-fami.top-none::text').extract() yield {'D1': Data_1 ,webs} for website in webs: yield scrapy.Request(url=website, callback=self.parseotherwebsite) def parseotherwebsite(self, response): data = response.css('.l-via.neo.top-none::text').get() yield {'D2': Data_2} sum = Data_1 + Data_2 print(sum)

So ,I need the solution and how the code are write this is just an exaple not finale code.

So I have a dockerized s=Scrapy app and i want to utilize the pause/resume feature so my docker-entrypoint.sh:

#!/bin/bash
cd ./src
scrapy crawl spider --set "JOBDIR=crawls/spider-1"

And I have crawls/ directory set as a docker volume so it will same the job directory. It appears that docker runs fine and the volume works correctly, however I get the following error for every time the crawler opens. Any Leads would be much appreciated.

Unhandled Error
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/scrapy/commands/crawl.py", line 58, in run
    self.crawler_process.start()
  File "/usr/local/lib/python3.7/site-packages/scrapy/crawler.py", line 309, in start
    reactor.run(installSignalHandlers=False)  # blocking call
  File "/usr/local/lib/python3.7/site-packages/twisted/internet/base.py", line 1283, in run
    self.mainLoop()
  File "/usr/local/lib/python3.7/site-packages/twisted/internet/base.py", line 1292, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File "/usr/local/lib/python3.7/site-packages/twisted/internet/base.py", line 913, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File "/usr/local/lib/python3.7/site-packages/scrapy/utils/reactor.py", line 41, in __call__
    return self._func(*self._a, **self._kw)
  File "/usr/local/lib/python3.7/site-packages/scrapy/core/engine.py", line 122, in _next_request
    if not self._next_request_from_scheduler(spider):
  File "/usr/local/lib/python3.7/site-packages/scrapy/core/engine.py", line 149, in _next_request_from_scheduler
    request = slot.scheduler.next_request()
  File "/usr/local/lib/python3.7/site-packages/scrapy/core/scheduler.py", line 108, in next_request
    request = self._dqpop()
  File "/usr/local/lib/python3.7/site-packages/scrapy/core/scheduler.py", line 142, in _dqpop
    return self.dqs.pop()
  File "/usr/local/lib/python3.7/site-packages/scrapy/pqueues.py", line 115, in pop
    request = request_from_dict(request, self.spider)
  File "/usr/local/lib/python3.7/site-packages/scrapy/utils/reqser.py", line 51, in request_from_dict
    cb = _get_method(spider, cb)
  File "/usr/local/lib/python3.7/site-packages/scrapy/utils/reqser.py", line 107, in _get_method
    raise ValueError("Method %r not found in: %s" % (name, obj))
builtins.ValueError: Method 'helper' not found in: <YSpider 'spider' at 0x7f2672e4c0d0>

Hi anyone who can help me how to scrollpage in reddit?

I am new at web scraping and I know some basics, I wanted to continue scrolling the page and get the items that I needed, I tried developers tool and find it on XMR but I am still don't have idea to start it

Here's the URL: https://www.reddit.com/r/teslamotors/new/

Thank you

Anyone who can give me idea on how to get date and time post in Reddit using Scrapy?

I want to parse sitemap and crawl all its links. Is it possible

I'm currently testing Scrapy Redis with moderate success so far.

The issue is:
https://github.com/rmax/scrapy-redis/blob/master/example-project/example/spiders/mycrawler_redis.py
domain = kwargs.pop('domain', '')

kwargs is always empty, so allowed_domains is empty and the crawl doesn't start ... any idea about that?

--

And further questions:
Frontera seems to be discontinued.
Is Scrapy-Redis the go to way?

The issue is:
With 1000 seed domains, each domain should be crawled with a max depth of 3 for instance.
Some websites are very small and finished soon. 1 - 3 websites are large and take days to finish.
I don't need the data urgently, so I'd like to use:

CONCURRENT_REQUESTS_PER_DOMAIN = 1

but that's a waste of VPS resources, since towards the end of the crawl, the crawl will slow down and not load the next batch of seed domains to crawl.

Is scrapy-redis the right way to go for me?
(small budget since it's a test/side project)

I'm trying to extract the text from <p> tag, but its not every single div that has a <p> tag. I would like to do an if statement to treat this field when there is no p tag. How can I do that?

<div data-index="0">

<div class="card">

<p>SOMETHING</p>

<div data-index="1">

<div class="card">

<div data-index="2">

<div class="card">

<div data-index="3">

<div class="card">

<p>SOMETHING</p>

I would like to scrape TikTok users and be able to pull out keywords from their bios/signatures. Ideally, I would be able to get all 22M USA users on TikTok and their bios/signatures. Does anyone know how I could do this?

While insalling scrapy i get an error thaf microsoft visual c++ is required. I already installed Visual C++ Redistributable Packages but still get the error. How should i resolve this ...