notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scrapping quotes\n",
    "<b>Author</b>: Rohit Patil\n",
    "\n",
    "## Introduction\n",
    "In this notebook we will be extracting quotes and their authors from the website (http://quotes.toscrape.com)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing Libraries\n",
    "\n",
    "We will import the following libraries for scraping the webpage\n",
    "1. requests: Used for basic get, post operations to the webpage. Here, to get the data from quotes.toscrape.com Servers\n",
    "2. bs4 (BeautifulSoup): To extract the content based on html tags and their attributes\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Defining two variables for inputs:\n",
    "tag (tag to search for a quote)\n",
    "index (the number of quote which we want to extract)\n",
    "\n",
    "We define modindex = index -1 as indexing starts from 0 in python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"love\"\n",
      "5\n"
     ]
    }
   ],
   "source": [
    "# tag=input(\"Enter the tag for which you want to find quotes\")\n",
    "# index=int(input(\"Enter which quote to show\"))\n",
    "tag=input()\n",
    "index=int(input())\n",
    "modindex=index-1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Getting the required pages\n",
    "Let's have a look at webpage now. The webpage for every tag is of form <b>http://quotes.toscrape.com/tag/'tag_name'/page/1or2/ </b> and by observation we see that page one contains maximum 10 quotes and remaining quotes are on page 2 we get the html content of webpage by `requests.get()`.\n",
    "\n",
    "The `request.get()` method gets a response object returned by the server based on the given url. Based on the response, we extract its content using bs4 and create what is generally called a `soup`\n",
    "The `soup.prettify()` method prettifies the extracted html content in the soup so that it is clearly legible\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!DOCTYPE html>\n",
      "\n",
      "<html lang=\"en\">\n",
      "<head>\n",
      "<meta charset=\"utf-8\">\n",
      "<title>Quotes to Scrape</title>\n",
      "<link href=\"/static/bootstrap.min.css\" rel=\"stylesheet\">\n",
      "<link href=\"/static/main.css\" rel=\"stylesheet\">\n",
      "</link></link></meta></head>\n",
      "<body>\n",
      "<div class=\"container\">\n",
      "<div class=\"row header-box\">\n",
      "<div class=\"col-md-8\">\n",
      "<h1>\n",
      "<a href=\"/\" style=\"text-decoration: none\">Quotes to Scrape</a>\n",
      "</h1>\n",
      "</div>\n",
      "<div class=\"col-md-4\">\n",
      "<p>\n",
      "<a href=\"/login\">Login</a>\n",
      "</p>\n",
      "</div>\n",
      "</div>\n",
      "<h3>Viewing tag: <a href=\"/tag/love/page/1/\">love</a></h3>\n",
      "<div class=\"row\">\n",
      "<div class=\"col-md-8\">\n",
      "<div class=\"quote\" itemscope=\"\" itemtype=\"http://schema.org/CreativeWork\">\n",
      "<span class=\"text\" itemprop=\"text\">“It is better to be hated for what you are than to be loved for what you are not.”</span>\n",
      "<span>by <small class=\"author\" itemprop=\"author\">André Gide</small>\n",
      "<a href=\"/author/Andre-Gide\">(about)</a>\n",
      "</span>\n",
      "<div class=\"tags\">\n",
      "            Tags:\n",
      "            <meta class=\"keywords\" content=\"life,love\" itemprop=\"keywords\">\n",
      "<a class=\"tag\" href=\"/tag/life/page/1/\">life</a>\n",
      "<a class=\"tag\" href=\"/tag/love/page/1/\">love</a>\n",
      "</meta></div>\n",
      "</div>\n",
      "<div class=\"quote\" itemscope=\"\" itemtype=\"http://schema.org/CreativeWork\">\n",
      "<span class=\"text\" itemprop=\"text\">“This life is what you make it. No matter what, you're going to mess up sometimes, it's a universal truth. But the good part is you get to decide how you're going to mess it up. Girls will be your friends - they'll act like it anyway. But just remember, some come, some go. The ones that stay with you through everything - they're your true best friends. Don't let go of them. Also remember, sisters make the best friends in the world. As for lovers, well, they'll come and go too. And baby, I hate to say it, most of them - actually pretty much all of them are going to break your heart, but you can't give up because if you give up, you'll never find your soulmate. You'll never find that half who makes you whole and that goes for everything. Just because you fail once, doesn't mean you're gonna fail at everything. Keep trying, hold on, and always, always, always believe in yourself, because if you don't, then who will, sweetie? So keep your head high, keep your chin up, and most importantly, keep smiling, because life's a beautiful thing and there's so much to smile about.”</span>\n",
      "<span>by <small class=\"author\" itemprop=\"author\">Marilyn Monroe</small>\n",
      "<a href=\"/author/Marilyn-Monroe\">(about)</a>\n",
      "</span>\n",
      "<div class=\"tags\">\n",
      "            Tags:\n",
      "            <meta class=\"keywords\" content=\"friends,heartbreak,inspirational,life,love,sisters\" itemprop=\"keywords\">\n",
      "<a class=\"tag\" href=\"/tag/friends/page/1/\">friends</a>\n",
      "<a class=\"tag\" href=\"/tag/heartbreak/page/1/\">heartbreak</a>\n",
      "<a class=\"tag\" href=\"/tag/inspirational/page/1/\">inspirational</a>\n",
      "<a class=\"tag\" href=\"/tag/life/page/1/\">life</a>\n",
      "<a class=\"tag\" href=\"/tag/love/page/1/\">love</a>\n",
      "<a class=\"tag\" href=\"/tag/sisters/page/1/\">sisters</a>\n",
      "</meta></div>\n",
      "</div>\n",
      "<div class=\"quote\" itemscope=\"\" itemtype=\"http://schema.org/CreativeWork\">\n",
      "<span class=\"text\" itemprop=\"text\">“You may not be her first, her last, or her only. She loved before she may love again. But if she loves you now, what else matters? She's not perfect—you aren't either, and the two of you may never be perfect together but if she can make you laugh, cause you to think twice, and admit to being human and making mistakes, hold onto her and give her the most you can. She may not be thinking about you every second of the day, but she will give you a part of her that she knows you can break—her heart. So don't hurt her, don't change her, don't analyze and don't expect more than she can give. Smile when she makes you happy, let her know when she makes you mad, and miss her when she's not there.”</span>\n",
      "<span>by <small class=\"author\" itemprop=\"author\">Bob Marley</small>\n",
      "<a href=\"/author/Bob-Marley\">(about)</a>\n",
      "</span>\n",
      "<div class=\"tags\">\n",
      "            Tags:\n",
      "            <meta class=\"keywords\" content=\"love\" itemprop=\"keywords\">\n",
      "<a class=\"tag\" href=\"/tag/love/page/1/\">love</a>\n",
      "</meta></div>\n",
      "</div>\n",
      "<div class=\"quote\" itemscope=\"\" itemtype=\"http://schema.org/CreativeWork\">\n",
      "<span class=\"text\" itemprop=\"text\">“The opposite of love is not hate, it's indifference. The opposite of art is not ugliness, it's indifference. The opposite of faith is not heresy, it's indifference. And the opposite of life is not death, it's indifference.”</span>\n",
      "<span>by <small class=\"author\" itemprop=\"author\">Elie Wiesel</small>\n",
      "<a href=\"/author/Elie-Wiesel\">(about)</a>\n",
      "</span>\n",
      "<div class=\"tags\">\n",
      "            Tags:\n",
      "            <meta class=\"keywords\" content=\"activism,apathy,hate,indifference,inspirational,love,opposite,philosophy\" itemprop=\"keywords\">\n",
      "<a class=\"tag\" href=\"/tag/activism/page/1/\">activism</a>\n",
      "<a class=\"tag\" href=\"/tag/apathy/page/1/\">apathy</a>\n",
      "<a class=\"tag\" href=\"/tag/hate/page/1/\">hate</a>\n",
      "<a class=\"tag\" href=\"/tag/indifference/page/1/\">indifference</a>\n",
      "<a class=\"tag\" href=\"/tag/inspirational/page/1/\">inspirational</a>\n",
      "<a class=\"tag\" href=\"/tag/love/page/1/\">love</a>\n",
      "<a class=\"tag\" href=\"/tag/opposite/page/1/\">opposite</a>\n",
      "<a class=\"tag\" href=\"/tag/philosophy/page/1/\">philosophy</a>\n",
      "</meta></div>\n",
      "</div>\n",
      "<div class=\"quote\" itemscope=\"\" itemtype=\"http://schema.org/CreativeWork\">\n",
      "<span class=\"text\" itemprop=\"text\">“It is not a lack of love, but a lack of friendship that makes unhappy marriages.”</span>\n",
      "<span>by <small class=\"author\" itemprop=\"author\">Friedrich Nietzsche</small>\n",
      "<a href=\"/author/Friedrich-Nietzsche\">(about)</a>\n",
      "</span>\n",
      "<div class=\"tags\">\n",
      "            Tags:\n",
      "            <meta class=\"keywords\" content=\"friendship,lack-of-friendship,lack-of-love,love,marriage,unhappy-marriage\" itemprop=\"keywords\">\n",
      "<a class=\"tag\" href=\"/tag/friendship/page/1/\">friendship</a>\n",
      "<a class=\"tag\" href=\"/tag/lack-of-friendship/page/1/\">lack-of-friendship</a>\n",
      "<a class=\"tag\" href=\"/tag/lack-of-love/page/1/\">lack-of-love</a>\n",
      "<a class=\"tag\" href=\"/tag/love/page/1/\">love</a>\n",
      "<a class=\"tag\" href=\"/tag/marriage/page/1/\">marriage</a>\n",
      "<a class=\"tag\" href=\"/tag/unhappy-marriage/page/1/\">unhappy-marriage</a>\n",
      "</meta></div>\n",
      "</div>\n",
      "<div class=\"quote\" itemscope=\"\" itemtype=\"http://schema.org/CreativeWork\">\n",
      "<span class=\"text\" itemprop=\"text\">“I love you without knowing how, or when, or from where. I love you simply, without problems or pride: I love you in this way because I do not know any other way of loving but this, in which there is no I or you, so intimate that your hand upon my chest is my hand, so intimate that when I fall asleep your eyes close.”</span>\n",
      "<span>by <small class=\"author\" itemprop=\"author\">Pablo Neruda</small>\n",
      "<a href=\"/author/Pablo-Neruda\">(about)</a>\n",
      "</span>\n",
      "<div class=\"tags\">\n",
      "            Tags:\n",
      "            <meta class=\"keywords\" content=\"love,poetry\" itemprop=\"keywords\">\n",
      "<a class=\"tag\" href=\"/tag/love/page/1/\">love</a>\n",
      "<a class=\"tag\" href=\"/tag/poetry/page/1/\">poetry</a>\n",
      "</meta></div>\n",
      "</div>\n",
      "<div class=\"quote\" itemscope=\"\" itemtype=\"http://schema.org/CreativeWork\">\n",
      "<span class=\"text\" itemprop=\"text\">“If you can make a woman laugh, you can make her do anything.”</span>\n",
      "<span>by <small class=\"author\" itemprop=\"author\">Marilyn Monroe</small>\n",
      "<a href=\"/author/Marilyn-Monroe\">(about)</a>\n",
      "</span>\n",
      "<div class=\"tags\">\n",
      "            Tags:\n",
      "            <meta class=\"keywords\" content=\"girls,love\" itemprop=\"keywords\">\n",
      "<a class=\"tag\" href=\"/tag/girls/page/1/\">girls</a>\n",
      "<a class=\"tag\" href=\"/tag/love/page/1/\">love</a>\n",
      "</meta></div>\n",
      "</div>\n",
      "<div class=\"quote\" itemscope=\"\" itemtype=\"http://schema.org/CreativeWork\">\n",
      "<span class=\"text\" itemprop=\"text\">“The real lover is the man who can thrill you by kissing your forehead or smiling into your eyes or just staring into space.”</span>\n",
      "<span>by <small class=\"author\" itemprop=\"author\">Marilyn Monroe</small>\n",
      "<a href=\"/author/Marilyn-Monroe\">(about)</a>\n",
      "</span>\n",
      "<div class=\"tags\">\n",
      "            Tags:\n",
      "            <meta class=\"keywords\" content=\"love\" itemprop=\"keywords\">\n",
      "<a class=\"tag\" href=\"/tag/love/page/1/\">love</a>\n",
      "</meta></div>\n",
      "</div>\n",
      "<div class=\"quote\" itemscope=\"\" itemtype=\"http://schema.org/CreativeWork\">\n",
      "<span class=\"text\" itemprop=\"text\">“Love does not begin and end the way we seem to think it does. Love is a battle, love is a war; love is a growing up.”</span>\n",
      "<span>by <small class=\"author\" itemprop=\"author\">James Baldwin</small>\n",
      "<a href=\"/author/James-Baldwin\">(about)</a>\n",
      "</span>\n",
      "<div class=\"tags\">\n",
      "            Tags:\n",
      "            <meta class=\"keywords\" content=\"love\" itemprop=\"keywords\">\n",
      "<a class=\"tag\" href=\"/tag/love/page/1/\">love</a>\n",
      "</meta></div>\n",
      "</div>\n",
      "<div class=\"quote\" itemscope=\"\" itemtype=\"http://schema.org/CreativeWork\">\n",
      "<span class=\"text\" itemprop=\"text\">“There is nothing I would not do for those who are really my friends. I have no notion of loving people by halves, it is not my nature.”</span>\n",
      "<span>by <small class=\"author\" itemprop=\"author\">Jane Austen</small>\n",
      "<a href=\"/author/Jane-Austen\">(about)</a>\n",
      "</span>\n",
      "<div class=\"tags\">\n",
      "            Tags:\n",
      "            <meta class=\"keywords\" content=\"friendship,love\" itemprop=\"keywords\">\n",
      "<a class=\"tag\" href=\"/tag/friendship/page/1/\">friendship</a>\n",
      "<a class=\"tag\" href=\"/tag/love/page/1/\">love</a>\n",
      "</meta></div>\n",
      "</div>\n",
      "<nav>\n",
      "<ul class=\"pager\">\n",
      "<li class=\"next\">\n",
      "<a href=\"/tag/love/page/2/\">Next <span aria-hidden=\"true\">?</span></a>\n",
      "</li>\n",
      "</ul>\n",
      "</nav>\n",
      "</div>\n",
      "<div class=\"col-md-4 tags-box\">\n",
      "<h2>Top Ten tags</h2>\n",
      "<span class=\"tag-item\">\n",
      "<a class=\"tag\" href=\"/tag/love/\" style=\"font-size: 28px\">love</a>\n",
      "</span>\n",
      "<span class=\"tag-item\">\n",
      "<a class=\"tag\" href=\"/tag/inspirational/\" style=\"font-size: 26px\">inspirational</a>\n",
      "</span>\n",
      "<span class=\"tag-item\">\n",
      "<a class=\"tag\" href=\"/tag/life/\" style=\"font-size: 26px\">life</a>\n",
      "</span>\n",
      "<span class=\"tag-item\">\n",
      "<a class=\"tag\" href=\"/tag/humor/\" style=\"font-size: 24px\">humor</a>\n",
      "</span>\n",
      "<span class=\"tag-item\">\n",
      "<a class=\"tag\" href=\"/tag/books/\" style=\"font-size: 22px\">books</a>\n",
      "</span>\n",
      "<span class=\"tag-item\">\n",
      "<a class=\"tag\" href=\"/tag/reading/\" style=\"font-size: 14px\">reading</a>\n",
      "</span>\n",
      "<span class=\"tag-item\">\n",
      "<a class=\"tag\" href=\"/tag/friendship/\" style=\"font-size: 10px\">friendship</a>\n",
      "</span>\n",
      "<span class=\"tag-item\">\n",
      "<a class=\"tag\" href=\"/tag/friends/\" style=\"font-size: 8px\">friends</a>\n",
      "</span>\n",
      "<span class=\"tag-item\">\n",
      "<a class=\"tag\" href=\"/tag/truth/\" style=\"font-size: 8px\">truth</a>\n",
      "</span>\n",
      "<span class=\"tag-item\">\n",
      "<a class=\"tag\" href=\"/tag/simile/\" style=\"font-size: 6px\">simile</a>\n",
      "</span>\n",
      "</div>\n",
      "</div>\n",
      "</div>\n",
      "<footer class=\"footer\">\n",
      "<div class=\"container\">\n",
      "<p class=\"text-muted\">\n",
      "                Quotes by: <a href=\"https://www.goodreads.com/quotes\">GoodReads.com</a>\n",
      "</p>\n",
      "<p class=\"copyright\">\n",
      "                Made with <span class=\"sh-red\">?</span> by <a href=\"https://scrapinghub.com\">Scrapinghub</a>\n",
      "</p>\n",
      "</div>\n",
      "</footer>\n",
      "</body>\n",
      "</html>\n"
     ]
    }
   ],
   "source": [
    "if modindex<10:\n",
    "        r = requests.get(\"http://quotes.toscrape.com/tag/\"+tag+\"/\")\n",
    "        c = r.content\n",
    "        soup=BeautifulSoup(c,\"html.parser\")\n",
    "        print(soup)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the html content, observe that all the data relevant to quotes is in a `span` tag and class `text`\n",
    "and data relevent to the author is in a `class` tag and class `author`\n",
    "The `find_all()` method for the soup object is used here to get all such occurences. It gives us the list of all occurences.\n",
    "we store the list in variable `t`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<span class=\"text\" itemprop=\"text\">\\u201cIt is better to be hated for what you are than to be loved for what you are not.\\u201d</span>, <span class=\"text\" itemprop=\"text\">\\u201cThis life is what you make it. No matter what, you're going to mess up sometimes, it's a universal truth. But the good part is you get to decide how you're going to mess it up. Girls will be your friends - they'll act like it anyway. But just remember, some come, some go. The ones that stay with you through everything - they're your true best friends. Don't let go of them. Also remember, sisters make the best friends in the world. As for lovers, well, they'll come and go too. And baby, I hate to say it, most of them - actually pretty much all of them are going to break your heart, but you can't give up because if you give up, you'll never find your soulmate. You'll never find that half who makes you whole and that goes for everything. Just because you fail once, doesn't mean you're gonna fail at everything. Keep trying, hold on, and always, always, always believe in yourself, because if you don't, then who will, sweetie? So keep your head high, keep your chin up, and most importantly, keep smiling, because life's a beautiful thing and there's so much to smile about.\\u201d</span>, <span class=\"text\" itemprop=\"text\">\\u201cYou may not be her first, her last, or her only. She loved before she may love again. But if she loves you now, what else matters? She's not perfect\\u2014you aren't either, and the two of you may never be perfect together but if she can make you laugh, cause you to think twice, and admit to being human and making mistakes, hold onto her and give her the most you can. She may not be thinking about you every second of the day, but she will give you a part of her that she knows you can break\\u2014her heart. So don't hurt her, don't change her, don't analyze and don't expect more than she can give. Smile when she makes you happy, let her know when she makes you mad, and miss her when she's not there.\\u201d</span>, <span class=\"text\" itemprop=\"text\">\\u201cThe opposite of love is not hate, it's indifference. The opposite of art is not ugliness, it's indifference. The opposite of faith is not heresy, it's indifference. And the opposite of life is not death, it's indifference.\\u201d</span>, <span class=\"text\" itemprop=\"text\">\\u201cIt is not a lack of love, but a lack of friendship that makes unhappy marriages.\\u201d</span>, <span class=\"text\" itemprop=\"text\">\\u201cI love you without knowing how, or when, or from where. I love you simply, without problems or pride: I love you in this way because I do not know any other way of loving but this, in which there is no I or you, so intimate that your hand upon my chest is my hand, so intimate that when I fall asleep your eyes close.\\u201d</span>, <span class=\"text\" itemprop=\"text\">\\u201cIf you can make a woman laugh, you can make her do anything.\\u201d</span>, <span class=\"text\" itemprop=\"text\">\\u201cThe real lover is the man who can thrill you by kissing your forehead or smiling into your eyes or just staring into space.\\u201d</span>, <span class=\"text\" itemprop=\"text\">\\u201cLove does not begin and end the way we seem to think it does. Love is a battle, love is a war; love is a growing up.\\u201d</span>, <span class=\"text\" itemprop=\"text\">\\u201cThere is nothing I would not do for those who are really my friends. I have no notion of loving people by halves, it is not my nature.\\u201d</span>]\n"
     ]
    }
   ],
   "source": [
    "        text = soup.find_all('span',class_=\"text\")\n",
    "        print(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hence here we can see all the quotes relevent to input tags are stored in the list <b>t</b>\n",
    "to access the required quote(nth quote):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "“It is not a lack of love, but a lack of friendship that makes unhappy marriages.”\n"
     ]
    }
   ],
   "source": [
    "        print(text[modindex].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly we find the author's name we find the tag `small` and class `author`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "by Friedrich Nietzsche\n"
     ]
    }
   ],
   "source": [
    "        au = soup.find_all('small',class_=\"author\")\n",
    "        print \"by\", au[modindex].text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we write the similar code for quotes on `page 2`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "if modindex>=10:\n",
    "        modindex=modindex-10\n",
    "        r =requests.get(\"http://quotes.toscrape.com/tag/\"+tag+\"/page/2\")\n",
    "        c = r.content\n",
    "        soup = BeautifulSoup(c,\"html.parser\")\n",
    "        t = soup.find_all('span',class_=\"text\")\n",
    "        print(t[modindex].text[1:-1])\n",
    "        au=soup.find_all('small',class_=\"author\")\n",
    "        print(\"by\", au[modindex].text)\t"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}