Developer Snippet Diary

Simple Grabing tutorial | Create Spider, Extract data, Save data in json, with pagination scraping | scrapy cheatsheet Python

We will grabe https://quotes.toscrape.com/

1.Run command below to start a scrapy project 

scrapy startproject tutorial

It will create a new folder tutorial and scrapy.cfg file

scrapy.cfg            # deploy configuration file

    tutorial/             # project's Python module, you'll import your code from here

        items.py          # project items definition file

        middlewares.py    # project middlewares file

        pipelines.py      # project pipelines file

        settings.py       # project settings file

        spiders/          # a directory where you'll later put your spiders

Spiders are classes that you define and that Scrapy uses to

  • scrape information from a website .
  • how to follow links in the pages,
  • how to parse the downloaded page content to extract data.

2. change directory to tutorial folder

cd tutorial

3. Create a new file quotes_spider.py inside spiders directory.

from pathlib import Path

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"

    def start_requests(self):
        urls = [
            'https://quotes.toscrape.com/page/1/',
            'https://quotes.toscrape.com/page/2/',
        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        page = response.url.split("/")[-2]
        filename = f'quotes-{page}.html'
        Path(filename).write_bytes(response.body)
        self.log(f'Saved file {filename}')

name: identifies the Spider. It must be unique within a project, that is, you can’t set the same name for different Spiders.

start_requests(): must return an iterable of Requests which the Spider will begin to crawl from.

parse(): handle the response downloaded for each of the requests made. The response parameter is an instance of TextResponse that holds the page content

4.Run the spider

scrapy crawl quotes

It will save two new files have been created: quotes-1.html and quotes-2.html in root directory.

background Working:

  • Scrapy schedules the scrapy.Request objects returned by the start_requests method of the Spider.
  • instantiates the Response objects and calls the callback method associated with the request (in this case, the parse method)

parse() is Scrapy’s default callback method so we can use short code also

from pathlib import Path
import scrapy
class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'https://quotes.toscrape.com/page/1/',
        'https://quotes.toscrape.com/page/2/',
    ]
    def parse(self, response):
        page = response.url.split("/")[-2]
        filename = f'quotes-{page}.html'
        Path(filename).write_bytes(response.body)

Extract Data

Run the command 

scrapy shell "https://quotes.toscrape.com/page/1/"

Now play with below commands

Using CSS Selector

response.css('title') #return selectorList object
response.css('title').getall() # return list of elements with content
response.css('title::text').getall() #return list of all text from element
response.css('title::text').get() # first result only
response.css('title::text')[0].get() # first result only
response.css('title::text').re(r'Quotes.*') # return all elements that start with Qoutes
response.css('title::text').re(r'(\w+) to (\w+)')
response.css("div.quote")
SUB Queries using css selector
quote = response.css("div.quote")[0]
text = quote.css("span.text::text").get() #return first found element
tags = quote.css("a.tag::text").getall() #return list

Using Xpath

Xpaths : more powerfull then css selectors
response.xpath('//title') #return selectorList object
response.xpath('//title/text()').get() # return text 

Details: https://docs.scrapy.org/en/latest/topics/selectors.html#topics-selectors

________________________

Extract Data from our project 

A Scrapy spider typically generates many dictionaries containing the data extracted from the page.

"yield" is a keyword used in generator functions to return a sequence of values one at a time.

from pathlib import Path
import scrapy
class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'https://quotes.toscrape.com/page/1/',
        'https://quotes.toscrape.com/page/2/',
    ]
    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').get(),
                'author': quote.css('small.author::text').get(),
                'tags': quote.css('div.tags a.tag::text').getall(),
            }

Below command will generate a quotes.json file containing all scraped items, serialized in JSON.

scrapy crawl quotes -O quotes.json

-O: overwrite the data with same file

save data using json lines, its append the data in previous file

scrapy crawl quotes -o quotes.jsonl

With Pagination:

Get Attribute of elements

response.css('li.next a::attr(href)').get()
OR
response.css('li.next a').attrib['href']

Scrape all https://quotes.toscrape.com/

import scrapy
class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'https://quotes.toscrape.com/page/1/',
    ]
    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').get(),
                'author': quote.css('small.author::text').get(),
                'tags': quote.css('div.tags a.tag::text').getall(),
            }
        next_page = response.css('li.next a::attr(href)').get()
        if next_page is not None:
            next_page = response.urljoin(next_page)
            yield scrapy.Request(next_page, callback=self.parse)
Posted by: R GONDAL
Email: [email protected]