Simple Grabing tutorial | Create Spider, Extract data, Save data in json, with pagination scraping | scrapy cheatsheet Python
We will grabe https://quotes.toscrape.com/
1.Run command below to start a scrapy project
scrapy startproject tutorial
It will create a new folder tutorial and scrapy.cfg file
scrapy.cfg # deploy configuration file
tutorial/ # project's Python module, you'll import your code from here
items.py # project items definition file
middlewares.py # project middlewares file
pipelines.py # project pipelines file
settings.py # project settings file
spiders/ # a directory where you'll later put your spiders
Spiders are classes that you define and that Scrapy uses to
- scrape information from a website .
- how to follow links in the pages,
- how to parse the downloaded page content to extract data.
2. change directory to tutorial folder
cd tutorial
3. Create a new file quotes_spider.py inside spiders directory.
from pathlib import Path
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
def start_requests(self):
urls = [
'https://quotes.toscrape.com/page/1/',
'https://quotes.toscrape.com/page/2/',
]
for url in urls:
yield scrapy.Request(url=url, callback=self.parse)
def parse(self, response):
page = response.url.split("/")[-2]
filename = f'quotes-{page}.html'
Path(filename).write_bytes(response.body)
self.log(f'Saved file {filename}')
name: identifies the Spider. It must be unique within a project, that is, you can’t set the same name for different Spiders.
start_requests(): must return an iterable of Requests which the Spider will begin to crawl from.
parse(): handle the response downloaded for each of the requests made. The response parameter is an instance of TextResponse that holds the page content
4.Run the spider
scrapy crawl quotes
It will save two new files have been created: quotes-1.html and quotes-2.html in root directory.
background Working:
- Scrapy schedules the scrapy.Request objects returned by the start_requests method of the Spider.
- instantiates the Response objects and calls the callback method associated with the request (in this case, the parse method)
parse() is Scrapy’s default callback method so we can use short code also
from pathlib import Path
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = [
'https://quotes.toscrape.com/page/1/',
'https://quotes.toscrape.com/page/2/',
]
def parse(self, response):
page = response.url.split("/")[-2]
filename = f'quotes-{page}.html'
Path(filename).write_bytes(response.body)
Extract Data
Run the command
scrapy shell "https://quotes.toscrape.com/page/1/"
Now play with below commands
Using CSS Selector
response.css('title') #return selectorList object
response.css('title').getall() # return list of elements with content
response.css('title::text').getall() #return list of all text from element
response.css('title::text').get() # first result only
response.css('title::text')[0].get() # first result only
response.css('title::text').re(r'Quotes.*') # return all elements that start with Qoutes
response.css('title::text').re(r'(\w+) to (\w+)')
response.css("div.quote")
SUB Queries using css selector
quote = response.css("div.quote")[0]
text = quote.css("span.text::text").get() #return first found element
tags = quote.css("a.tag::text").getall() #return list
Using Xpath
Xpaths : more powerfull then css selectors
response.xpath('//title') #return selectorList object
response.xpath('//title/text()').get() # return text
Details: https://docs.scrapy.org/en/latest/topics/selectors.html#topics-selectors
________________________
Extract Data from our project
A Scrapy spider typically generates many dictionaries containing the data extracted from the page.
"yield" is a keyword used in generator functions to return a sequence of values one at a time.
from pathlib import Path
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = [
'https://quotes.toscrape.com/page/1/',
'https://quotes.toscrape.com/page/2/',
]
def parse(self, response):
for quote in response.css('div.quote'):
yield {
'text': quote.css('span.text::text').get(),
'author': quote.css('small.author::text').get(),
'tags': quote.css('div.tags a.tag::text').getall(),
}
Below command will generate a quotes.json file containing all scraped items, serialized in JSON.
scrapy crawl quotes -O quotes.json
-O: overwrite the data with same file
save data using json lines, its append the data in previous file
scrapy crawl quotes -o quotes.jsonl
With Pagination:
Get Attribute of elements
response.css('li.next a::attr(href)').get()
OR
response.css('li.next a').attrib['href']
Scrape all https://quotes.toscrape.com/
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = [
'https://quotes.toscrape.com/page/1/',
]
def parse(self, response):
for quote in response.css('div.quote'):
yield {
'text': quote.css('span.text::text').get(),
'author': quote.css('small.author::text').get(),
'tags': quote.css('div.tags a.tag::text').getall(),
}
next_page = response.css('li.next a::attr(href)').get()
if next_page is not None:
next_page = response.urljoin(next_page)
yield scrapy.Request(next_page, callback=self.parse)