-
-Welcome, fellow developers and enthusiasts, to a workshop that will illuminate the path to mastering one of the most powerful tools in the web automation arsenal – Puppeteer. Developed by Google, Puppeteer is more than just a library; it's a gateway to automating tasks, testing websites, and breathing life into your web projects.
-Just as Git revolutionizes collaboration in the coding world, Puppeteer empowers developers to navigate the digital landscape with precision and finesse. With Puppeteer as your companion, you'll embark on a journey where virtual pages become your canvas, and scripts wield the power to interact with web elements, simulate user actions, and extract valuable insights.
-Throughout this workshop, we'll dive into the intricacies of Puppeteer, unraveling its capabilities, exploring practical use cases, and equipping you with the skills to wield this tool effectively. Whether you're a seasoned developer or a curious novice, Puppeteer offers a gateway to a world where automation is not just a convenience but a game-changer.
-As promised, we'll look into ways of finding rent in Cluj Napoca in a more efficient way. So, get ready to dive into the world of Puppeteer, where lines of code become puppet strings, and the web transforms into your stage. Let's embark on this journey together, where every click, every scroll, and every interaction is powered by the magic of Puppeteer. Your apartment in Cluj Napoca awaits!
-
-Puppeteer is a Node.js library which provides a high-level API to control Chrome/Chromium over the DevTools Protocol. While it's primarily used for automating web tasks, Puppeteer can also be used for web scraping, testing, and creating snapshots of web pages.
-
-
-
-
-
-Most things that you can do manually in the browser can be done using Puppeteer! Here are a few examples to get you started:
-
-- Automate form submission, UI testing, keyboard input, etc.
-- Create an automated testing environment using the latest JavaScript and browser features.
-- Capture a timeline trace of your site to help diagnose performance issues.
-- Generate screenshots and PDFs of pages.
-- Crawl a SPA (Single-Page Application) and generate pre-rendered content (i.e. "SSR" (Server-Side Rendering)).
-
-
-
-To use Puppeteer in your project, run:
-npm install puppeteer
-# or
-yarn add puppeteer
-# or
-pnpm add puppeteer
-
-When you install Puppeteer, it automatically downloads a recent version of Chrome for Testing (~170MB macOS, ~282MB Linux, ~280MB Windows) and a chrome-headless-shell binary that is guaranteed to work with Puppeteer. The browser is downloaded to the $HOME/.cache/puppeteer folder by default
-
-
-At the heart of Puppeteer lies the concept of a browser and a page. The browser represents the Chrome or Chromium instance controlled by Puppeteer, while the page represents a single tab within that browser instance. Think of the browser as your window to the internet, and each page as a separate document or website within that window.
-
-Selectors are a fundamental aspect of web automation, allowing you to pinpoint and interact with specific elements on a web page. Puppeteer leverages various types of selectors, including:
-
-- CSS Selectors: Utilize CSS syntax to target elements based on their attributes, classes, or relationships with other elements.
-- XPath Selectors: Employ XPath expressions to locate elements by their position in the HTML document tree.
-- DOM Selectors: Interact with elements using their unique IDs, names, or other properties within the Document Object Model (DOM).
-
-Understanding selectors is crucial for navigating and manipulating web pages effectively with Puppeteer. They serve as the building blocks for automating interactions such as clicking buttons, inputting text, or extracting data from specific elements.
-
-Usually, you start working with Puppeteer by launching a browser instance. This browser instance is controlled by Puppeteer and can be used to open new tabs, navigate to different URLs, interact with web pages, and perform various tasks.
-
-import puppeteer from "puppeteer";
-
-const browser = await puppeteer.launch();
-
-const page = await browser.newPage();
-
-When launching a browser, you can specify various options to customize its behavior. These options include:
-
-headless: A boolean value that determines whether the browser should run in headless mode (without a visible UI). By default, this option is set to true.slowMo: A number that introduces a delay (in milliseconds) between Puppeteer actions. This can be useful for debugging or observing the automation process.defaultViewport: An object that defines the initial browser window size and scale factor. This can be useful for ensuring consistent rendering across different devices.args: An array of strings that specify additional command-line arguments to pass to the browser instance. These arguments can modify the browser's behavior or enable specific features.- and more...
-
-const browser = await puppeteer.launch({
- headless: false,
- slowMo: 250,
- defaultViewport: null,
- args: ["--start-maximized"],
-});
-
-
-Once you're done working with the browser, you should close it to free up system resources. You can close the browser instance using the browser.close() method.
-import puppeteer from "puppeteer";
-
-const browser = await puppeteer.launch();
-
-const page = await browser.newPage();
-
-await browser.close();
-
-
-In this section, we will learn how to interact with a page using Puppeteer. Puppeteer provides a high-level API to interact with a page, which includes navigating to a page, clicking on elements, typing text, and more. All the functions are asynchronous and return promises, which makes it easy to work with Puppeteer in an asynchronous manner. There are many methods available to interact with a page, and we will cover some of the most common ones, but you can find the full list in the official documentation.
-
-To navigate to a page, you can use the page.goto() method. This method takes a URL as an argument and loads the page at the specified URL.
-import puppeteer from "puppeteer";
-
-(async () => {
- const browser = await puppeteer.launch();
- const page = await browser.newPage();
-
- await page.goto("https://www.example.com");
-
- await browser.close();
-})();
-
-In the example above, we launch a browser instance, create a new page, and navigate to https://www.example.com. Once the page is loaded, we close the browser.
-
-You can take a screenshot of a page using the page.screenshot() method. This method captures a screenshot of the current page and saves it to a file, to the specified path.
-import puppeteer from "puppeteer";
-
-(async () => {
- const browser = await puppeteer.launch();
- const page = await browser.newPage();
-
- await page.goto("https://www.example.com");
- await page.screenshot({ path: "example.png", fullPage: true });
-
- await browser.close();
-})();
-
-
-You can generate a PDF of a page using the page.pdf() method. This method generates a PDF of the current page and saves it to a file, to the specified path.
-import puppeteer from "puppeteer";
-
-(async () => {
- const browser = await puppeteer.launch();
- const page = await browser.newPage();
-
- await page.goto("https://www.example.com");
- await page.pdf({ path: "example.pdf", format: "A4" });
-
- await browser.close();
-})();
-
-
-
-
-page.$(selector): This method returns the first element that matches the specified selector. If no element matches the selector, the method returns null.page.$$(selector): This method returns an array of elements that match the specified selector. If no elements match the selector, the method returns an empty array.page.$eval(selector, pageFunction, ...args): This method evaluates the specified function in the context of the first element that matches the selector. The function can access the DOM of the element and return a value. The arguments passed to the function are serialized and can be accessed as function parameters.page.evaluate(pageFunction, ...args): This method evaluates the specified function in the context of the page. The function can access the DOM of the page and return a value. The arguments passed to the function are serialized and can be accessed as function parameters.
-When evaluating JavaScript on a page, you can interact with elements, get their attributes, and perform various actions. This allows you to automate tasks, scrape data, and test web applications.
-import puppeteer from "puppeteer";
-
-const selector = ".selector";
-const textSelector = ".text-selector";
-
-(async () => {
- const browser = await puppeteer.launch();
- const page = await browser.newPage();
- await page.goto("https://www.example.com");
-
- const element = await page.$(selector); // Get the first element that matches the selector
- const elements = await page.$$(selector); // Get all elements that match the selector
-
- let value = await page.$eval(selector, (element) => {
- return element..innerText;
- });
-
- value = await page.evaluate((selector) => {
- return document.querySelector(selector).textContent;
- }, selector);
-
- const text = await element.evaluate((el) => el.textContent); // Get the text content of the element
-})();
-
-
-When interacting with a page, it's important to wait for elements to be available before performing actions on them. Puppeteer provides the page.waitForSelector() method to wait for an element to be present in the DOM. By default, this method waits for 30 seconds before timing out, but you can specify a custom timeout using the timeout option.
-import puppeteer from "puppeteer";
-
-const selector = ".selector";
-
-(async () => {
- const browser = await puppeteer.launch();
- const page = await browser.newPage();
- await page.goto("https://www.example.com");
-
- await page.waitForSelector(selector, { timeout: 5000 });
-
- await browser.close();
-})();
-
-Also, it's advisable to insert some delay before performing actions on the page. This can be done creating a delay function and using it before performing actions.
-function delay(time) {
- return new Promise(function (resolve) {
- setTimeout(resolve, time);
- });
-}
-
-(async () => {
- const browser = await puppeteer.launch();
- const page = await browser.newPage();
- await page.goto("https://www.example.com");
-
- await delay(2000); // Wait for 2 seconds
-
- await browser.close();
-})();
-
-
-To click on an element on a page, you can use the element.click() method. This method simulates a mouse click on the element. While this method seems pretty straightforward, in some cases, the element might not be completely visible or clickable, that's why it's more recommended to use the evaluation methods mentioned above.
-import puppeteer from "puppeteer";
-
-const selector = ".selector";
-
-(async () => {
- const browser = await puppeteer.launch();
- const page = await browser.newPage();
- await page.goto("https://www.example.com");
-
- await page.waitForSelector(selector);
-
- const element = await page.$(selector);
-
- await element.click();
-
- // safer approach
- await element.evaluate((el) => el.click());
-
- await browser.close();
-})();
-
-
-For input fields, you can use the element.type() method to type text into the field. This method simulates typing text on the keyboard and is useful for automating form submissions, login processes, and more.
-import puppeteer from "puppeteer";
-
-const selector = ".selector";
-
-(async () => {
- const browser = await puppeteer.launch();
- const page = await browser.newPage();
- await page.goto("https://www.example.com");
-
- await page.waitForSelector(selector);
-
- const element = await page.$(selector);
-
- await element.type("Hello, World!");
-
- await browser.close();
-})();
-
-
-Puppeteer is a Node.js library which provides a high-level API to control Chrome/Chromium over the DevTools Protocol. While it's primarily used for automating web tasks, Puppeteer can also be used for web scraping, testing, and creating snapshots of web pages.
-
-
-
-
-
-Most things that you can do manually in the browser can be done using Puppeteer! Here are a few examples to get you started:
-
-- Automate form submission, UI testing, keyboard input, etc.
-- Create an automated testing environment using the latest JavaScript and browser features.
-- Capture a timeline trace of your site to help diagnose performance issues.
-- Generate screenshots and PDFs of pages.
-- Crawl a SPA (Single-Page Application) and generate pre-rendered content (i.e. "SSR" (Server-Side Rendering)).
-
-
-
-To use Puppeteer in your project, run:
-npm install puppeteer
-# or
-yarn add puppeteer
-# or
-pnpm add puppeteer
-
-When you install Puppeteer, it automatically downloads a recent version of Chrome for Testing (~170MB macOS, ~282MB Linux, ~280MB Windows) and a chrome-headless-shell binary that is guaranteed to work with Puppeteer. The browser is downloaded to the $HOME/.cache/puppeteer folder by default
-
-
-At the heart of Puppeteer lies the concept of a browser and a page. The browser represents the Chrome or Chromium instance controlled by Puppeteer, while the page represents a single tab within that browser instance. Think of the browser as your window to the internet, and each page as a separate document or website within that window.
-
-Selectors are a fundamental aspect of web automation, allowing you to pinpoint and interact with specific elements on a web page. Puppeteer leverages various types of selectors, including:
-
-- CSS Selectors: Utilize CSS syntax to target elements based on their attributes, classes, or relationships with other elements.
-- XPath Selectors: Employ XPath expressions to locate elements by their position in the HTML document tree.
-- DOM Selectors: Interact with elements using their unique IDs, names, or other properties within the Document Object Model (DOM).
-
-Understanding selectors is crucial for navigating and manipulating web pages effectively with Puppeteer. They serve as the building blocks for automating interactions such as clicking buttons, inputting text, or extracting data from specific elements.
-
-Usually, you start working with Puppeteer by launching a browser instance. This browser instance is controlled by Puppeteer and can be used to open new tabs, navigate to different URLs, interact with web pages, and perform various tasks.
-
-import puppeteer from "puppeteer";
-
-const browser = await puppeteer.launch();
-
-const page = await browser.newPage();
-
-When launching a browser, you can specify various options to customize its behavior. These options include:
-
-headless: A boolean value that determines whether the browser should run in headless mode (without a visible UI). By default, this option is set to true.slowMo: A number that introduces a delay (in milliseconds) between Puppeteer actions. This can be useful for debugging or observing the automation process.defaultViewport: An object that defines the initial browser window size and scale factor. This can be useful for ensuring consistent rendering across different devices.args: An array of strings that specify additional command-line arguments to pass to the browser instance. These arguments can modify the browser's behavior or enable specific features.- and more...
-
-const browser = await puppeteer.launch({
- headless: false,
- slowMo: 250,
- defaultViewport: null,
- args: ["--start-maximized"],
-});
-
-
-Once you're done working with the browser, you should close it to free up system resources. You can close the browser instance using the browser.close() method.
-import puppeteer from "puppeteer";
-
-const browser = await puppeteer.launch();
-
-const page = await browser.newPage();
-
-await browser.close();
-
-
-In this section, we will learn how to interact with a page using Puppeteer. Puppeteer provides a high-level API to interact with a page, which includes navigating to a page, clicking on elements, typing text, and more. All the functions are asynchronous and return promises, which makes it easy to work with Puppeteer in an asynchronous manner. There are many methods available to interact with a page, and we will cover some of the most common ones, but you can find the full list in the official documentation.
-
-To navigate to a page, you can use the page.goto() method. This method takes a URL as an argument and loads the page at the specified URL.
-import puppeteer from "puppeteer";
-
-(async () => {
- const browser = await puppeteer.launch();
- const page = await browser.newPage();
-
- await page.goto("https://www.example.com");
-
- await browser.close();
-})();
-
-In the example above, we launch a browser instance, create a new page, and navigate to https://www.example.com. Once the page is loaded, we close the browser.
-
-You can take a screenshot of a page using the page.screenshot() method. This method captures a screenshot of the current page and saves it to a file, to the specified path.
-import puppeteer from "puppeteer";
-
-(async () => {
- const browser = await puppeteer.launch();
- const page = await browser.newPage();
-
- await page.goto("https://www.example.com");
- await page.screenshot({ path: "example.png", fullPage: true });
-
- await browser.close();
-})();
-
-
-You can generate a PDF of a page using the page.pdf() method. This method generates a PDF of the current page and saves it to a file, to the specified path.
-import puppeteer from "puppeteer";
-
-(async () => {
- const browser = await puppeteer.launch();
- const page = await browser.newPage();
-
- await page.goto("https://www.example.com");
- await page.pdf({ path: "example.pdf", format: "A4" });
-
- await browser.close();
-})();
-
-
-
-
-page.$(selector): This method returns the first element that matches the specified selector. If no element matches the selector, the method returns null.page.$$(selector): This method returns an array of elements that match the specified selector. If no elements match the selector, the method returns an empty array.page.$eval(selector, pageFunction, ...args): This method evaluates the specified function in the context of the first element that matches the selector. The function can access the DOM of the element and return a value. The arguments passed to the function are serialized and can be accessed as function parameters.page.evaluate(pageFunction, ...args): This method evaluates the specified function in the context of the page. The function can access the DOM of the page and return a value. The arguments passed to the function are serialized and can be accessed as function parameters.
-When evaluating JavaScript on a page, you can interact with elements, get their attributes, and perform various actions. This allows you to automate tasks, scrape data, and test web applications.
-import puppeteer from "puppeteer";
-
-const selector = ".selector";
-const textSelector = ".text-selector";
-
-(async () => {
- const browser = await puppeteer.launch();
- const page = await browser.newPage();
- await page.goto("https://www.example.com");
-
- const element = await page.$(selector); // Get the first element that matches the selector
- const elements = await page.$$(selector); // Get all elements that match the selector
-
- let value = await page.$eval(selector, (element) => {
- return element..innerText;
- });
-
- value = await page.evaluate((selector) => {
- return document.querySelector(selector).textContent;
- }, selector);
-
- const text = await element.evaluate((el) => el.textContent); // Get the text content of the element
-})();
-
-
-When interacting with a page, it's important to wait for elements to be available before performing actions on them. Puppeteer provides the page.waitForSelector() method to wait for an element to be present in the DOM. By default, this method waits for 30 seconds before timing out, but you can specify a custom timeout using the timeout option.
-import puppeteer from "puppeteer";
-
-const selector = ".selector";
-
-(async () => {
- const browser = await puppeteer.launch();
- const page = await browser.newPage();
- await page.goto("https://www.example.com");
-
- await page.waitForSelector(selector, { timeout: 5000 });
-
- await browser.close();
-})();
-
-Also, it's advisable to insert some delay before performing actions on the page. This can be done creating a delay function and using it before performing actions.
-function delay(time) {
- return new Promise(function (resolve) {
- setTimeout(resolve, time);
- });
-}
-
-(async () => {
- const browser = await puppeteer.launch();
- const page = await browser.newPage();
- await page.goto("https://www.example.com");
-
- await delay(2000); // Wait for 2 seconds
-
- await browser.close();
-})();
-
-
-To click on an element on a page, you can use the element.click() method. This method simulates a mouse click on the element. While this method seems pretty straightforward, in some cases, the element might not be completely visible or clickable, that's why it's more recommended to use the evaluation methods mentioned above.
-import puppeteer from "puppeteer";
-
-const selector = ".selector";
-
-(async () => {
- const browser = await puppeteer.launch();
- const page = await browser.newPage();
- await page.goto("https://www.example.com");
-
- await page.waitForSelector(selector);
-
- const element = await page.$(selector);
-
- await element.click();
-
- // safer approach
- await element.evaluate((el) => el.click());
-
- await browser.close();
-})();
-
-
-For input fields, you can use the element.type() method to type text into the field. This method simulates typing text on the keyboard and is useful for automating form submissions, login processes, and more.
-import puppeteer from "puppeteer";
-
-const selector = ".selector";
-
-(async () => {
- const browser = await puppeteer.launch();
- const page = await browser.newPage();
- await page.goto("https://www.example.com");
-
- await page.waitForSelector(selector);
-
- const element = await page.$(selector);
-
- await element.type("Hello, World!");
-
- await browser.close();
-})();
-
-
-
-
-- Cloning: Creating a local copy of a repository. This enables work on the project locally and facilitates pushing changes to the remote repository.
-- Push/Pull: The primary methods of syncing changes between a local and remote repository. Pushing sends committed changes to a remote repository, while pulling retrieves changes from a remote repository.
-- Merge: The process of combining changes from different branches. Typically done when a feature branch is complete and ready to integrate into the main branch.
-- Forking: Creating a copy of a repository under your account. This allows experimentation with changes without affecting the original project.
-
-
-Pull requests simplify collaboration using GitHub. They notify project maintainers about changes in a forked repository, making it easier to review and discuss proposed changes before integration into the official project. Pull requests are commonly used to propose changes, notify maintainers about changes in a forked repository, discuss proposed changes, and collaborate with other developers.
-When submitted, pull requests are reviewed by project maintainers. They can approve, request changes, or close the request. They're an excellent way to contribute to open source projects and gain experience with Git and GitHub. Merging pull requests offers two options:
-
-- Merge Commit: Creates a new commit to record changes, the default option.
-- Squash and Merge: Combines all commits into a single commit before merging, keeping the commit history concise.
-
-
-
-
-
-git init: Initialize a new Git repository.git add <file>: Add changes to the staging area.git commit -m "message": Commit changes with a descriptive message.git status: Check the status of your working directory.git log: View the commit history.git diff: View changes made to a file.git checkout <branch>: Switch to a different branch. Use git checkout -b <branch> to create a new branch.git reset <file>: Remove a file from the staging area.git rm <file>: Remove a file from the working directory and staging area.git mv <file> <new file>: Rename a file and stage the change.git remote add origin <url>: Add a remote repository.git push origin <branch>: Push changes to a remote repository.git pull origin <branch>: Pull changes from a remote repository.git clone <url>: Clone a remote repository to your local machine.git merge <branch>: Merge a branch into the active branch.git fetch: Download changes from a remote repository.git stash: Temporarily store changes that you don't want to commit.git push --force: Force push changes to a remote repository (Use with caution).
-
-
-
-- Descriptive Commit Messages: Write messages explaining changes made, avoiding vague statements like "fixed bug". This aids in tracking changes and understanding history.
-- Regular Pulling Changes: Pull changes regularly from the remote repository to keep the local repository updated and reduce the risk of conflicts.
-- Branch Naming Conventions: Use descriptive branch names, making it easier to understand their purpose and track changes.
-- Code Reviews and Collaboration: Utilize pull requests for reviewing proposed changes before integrating them into the official project. It's a great way to collaborate and learn about Git and GitHub.
-
-
-
-Having multiple branches is extremely convenient to keep new changes separated and avoid pushing unapproved or broken changes to production. When the changes are approved, integrating them into the production branch is essential.
-
-A fast-forward merge occurs when the current branch has no additional commits compared to the branch being merged. Git prefers this as the easiest option: it merges the commit(s) from the branch being merged into the current branch without creating a new commit.
-
-
-In case the current branch has commits not present in the branch to be merged, Git performs a no-fast-forward merge. This creates a new merging commit on the active branch, integrating changes from both branches.
-
-
-Merging branches with conflicting changes in the same lines or with deleted files that were modified elsewhere can cause merge conflicts. Git highlights these conflicts, enabling manual resolution by removing unwanted changes, saving, re-adding the changed file, and committing the changes.
-
-
-Git rebase copies commits from the current branch and places them on top of the specified branch. Unlike merging, rebasing doesn't attempt to decide which files to keep, maintaining the branch's latest changes and avoiding merging conflicts.
-
-Rebasing is ideal when integrating updates from the master branch into a feature branch, preventing future merging conflicts.
-
-Interactive rebase allows modification of commits before rebasing, offering six actions:
-
-- reword: Edit the commit message.
-- edit: Amend the commit.
-- squash: Combine the commit into the previous one.
-- fixup: Merge the commit into the previous one without retaining the commit's log message.
-- exec: Execute a command on each commit being rebased.
-- drop: Remove the commit.
-
-
-Feel free to use these actions for full control over your commits, whether removing or merging them. Interactive rebase facilitates the modification of commits.
-
-A soft reset shifts HEAD to the specified commit without discarding changes introduced by the following commits. It preserves access to the changes made in the previous commits, enabling fixing and recommitting them later.
-
-
-A hard reset returns Git's state to the specified commit, including changes in the working directory and staged files, discarding all changes introduced after the commit.
-
-
-Git revert creates a new commit that undoes the changes introduced by the specified commit, ensuring the commit history remains unaltered.
-
-Git revert is useful for undoing a specific commit without altering the branch's history.
-
-Welcome to the Git Essentials second exercise! In this exercise, you'll dive into resolving merge conflicts in a collaborative Git environment. The focus will be on updating personal information in a shared file, generating merge conflicts, and exploring different conflict resolution strategies. Here's the tasks you'll be performing:
-
-
-
--
-
Updating Your Profile:
-
-- Your task is to update the
gdsc.txt file in repositories belonging to your fellow participants. Add your name, occupation, favorite hobby, and something you dislike to their gdsc.txt.
-
-
--
-
Creating Pull Requests:
-
-- Initiate a Pull Request in the repository you're modifying, aiming to update the
gdsc.txt file.
-- Ensure your PR clearly states the purpose: "Adding Personal Information."
-
-
--
-
Simulating Merge Conflicts:
-
-- Your goal is to make changes to the same line or section of the
gdsc.txt file in repositories of other participants.
-- When your PRs are submitted, try merging them into your repository. You may face merge conflicts due to changes in the same sections of the file.
-
-
--
-
Resolving Conflicts:
-
-- Use Git commands (
git status, git diff, git checkout --ours/--theirs) to identify conflicting sections.
-- Discuss and communicate effectively with the repository owners to decide on retaining specific changes.
-- Resolve conflicts either by manual editing or using Git tools, ensuring file consistency and preventing data loss during conflict resolution.
-
-
--
-
Exploring Git Strategies:
-
-- Experiment with different merge strategies (merge, rebase, squash) while integrating changes from the main repository.
-- Consider the advantages and limitations of each strategy in managing conflicts and maintaining a clean commit history.
-
-
--
-
Reviewing Commit History and Interactive Rebase:
-
-- Analyze commit history using
git log.
-- Learn about interactive rebase by renaming a commit, squashing, or rearranging commits based on a previous commit.
-
-
-
-
-In this section, we'll cover the process of cloning repositories from GitHub onto your local machine.
-
-Each participant sitting at the same table should share their repository URL. This URL can be obtained from their GitHub repository.
-
-If the repositories are private, participants need to grant access permissions to others. Ask the repository owner to grant access to you by having him follow these steps:
-
- Granting Permissions for Private Repositories on GitHub
-
-- Go to the repository on GitHub.
-- Click on "Settings" in the repository.
-- Select
Collaborators and Teams from the sidebar.
-- Under
Manage access, click on Add people.
-- Enter the GitHub username or email of the person you want to grant access to.
-- Select the write permission level.
-- Finally, click on "Add [username/email] to this repository".
-
-
-
-For each participant, follow these steps to clone the repository to your local machine:
-a. Open your terminal or Git Bash.
-b. Use the git clone command followed by the repository URL:
- git clone <repository_URL>
-
-c. This will create a local copy of the repository on your machine.
-If all went well with adding an SSH key in the previous exercise, you should be able to clone the repository without any issues. If you're still having trouble, ask for a volunteer to help you out.
-
-Once the cloning process is completed, navigate into the cloned directory using the cd command in the terminal to ensure the repository has been successfully cloned.
-Following these steps will allow each participant to clone repositories onto their local machines, enabling them to contribute and collaborate effectively.
-
-
-The objective of this task is to update the gdsc.txt file in repositories owned by other participants with your personal information.
-
-
--
-
Create a New Branch:
-
-
-
--
-
Open gdsc.txt:
-
-- Navigate to the repository of a fellow participant in your preferred code editor.
-
-
--
-
Update gdsc.txt:
-
-- Locate the
gdsc.txt file and add your name, occupation, favorite hobby, and something you dislike in the existing format, similarly to how the repository owner has added their information.
-
-
--
-
Save Changes:
-
-- Save the modifications made to the
gdsc.txt file.
-
-
--
-
Stage Changes:
-
-- In your terminal, add the modified file to the staging area using the command:
-
-git add gdsc.txt
-
-
--
-
Commit Changes:
-
-- Commit the changes with a descriptive message:
-
-git commit -m "Added personal information to gdsc.txt"
-
-
--
-
Push to Repository:
-
-- Push the changes to the repository using the command:
-
git push --set-upstream origin {branch-name}
- What is --set-upstream?
- The `--set-upstream` flag is used to set the default remote branch for the current local branch. This allows you to use `git push` without specifying the remote branch name.
-
-
-
-
-
-
--
-
Navigate to the Repository:
-
-- Visit the repository on GitHub where you've made changes that you want to merge.
-
-
--
-
Switch to the Branch:
-
-- If the changes are in a specific branch, ensure you're on that branch.
-
-
--
-
Click on "New Pull Request":
-
-- On the repository page, click on the "Pull Requests" tab.
-
-
--
-
Compare Changes:
-
-- Click the "New pull request" button. GitHub will automatically detect the base and compare branches. Ensure the changes you want to merge are displayed correctly.
-
-
--
-
Add Title and Description:
-
-- Provide a title and description that clearly explains the purpose of the Pull Request. Describe the changes and why they are being made.
-
-
--
-
Review Changes:
-
-- Review the changes tab to ensure all modifications you expect are included.
-
-
--
-
Assign Reviewers (If Required):
-
-- Assign specific reviewers for your Pull Request, if necessary. Reviewers will be notified to assess your changes.
-
-
--
-
Labels, Milestones, and Projects (Optional):
-
-- Apply labels, associate milestones, or link the PR to a project board if your repository uses these features.
-
-
--
-
Create the Pull Request:
-
-- Click "Create Pull Request" to officially open the Pull Request.
-
-
-
-
-
-Ideally, each repository should have at least two Pull Requests (PRs) modifying the same gdsc.txt file with contact information. Merging one PR may cause a conflict in the other PR due to changes in the same file.
-
-The primary difference between the merge and rebase approaches lies in how they integrate changes.
-Merge combines the commit histories of the divergent branches, creating a new commit that reflects the combined changes. It retains the original commit history but may result in "merge commits," which explicitly mark where the branches were merged.
-Rebase, on the other hand, rewrites the commit history by placing the local branch's commits on top of the latest changes from the main branch. This creates a linear and cleaner history but alters commit timestamps.
-
-
--
-
Checkout the Branch:
-
-- Navigate to the branch with the conflicting changes:
-
git checkout branch-name
-
-
--
-
Pull the Latest Changes:
-
-- Fetch the latest changes from the remote repository:
-
git pull origin main
-
-
--
-
Resolve Conflicts Locally:
-
-- Open the
gdsc.txt file in your code editor.
-- Resolve conflicts using the editor's conflict resolution tool or manually by editing the file.
-
-- The conflict markers (
<<<<<<<, =======, >>>>>>>) denote conflicting sections.
-- Use the editor's tool to choose and retain the desired changes from both versions.
-- Ensure the file reflects the intended information after conflict resolution.
-
-
-
-
--
-
Stage and Commit Changes:
-
-- After resolving conflicts, stage the modified file:
-
git add gdsc.txt
-- Commit the changes:
-
git commit -m "Resolved merge conflict in gdsc.txt"
-
-
--
-
Push Changes to the Branch:
-
-- Push the changes to the branch:
-
git push origin branch-name
-
-
--
-
Merge the PR:
-
-- Proceed to merge the conflicted PR through the repository interface.
-
-
-
-
-
--
-
Checkout the Branch:
-
-- Navigate to the branch with the conflicting changes:
-
git checkout branch-name
-
-
--
-
Rebase from Main:
-
-- Rebase the branch with the latest changes from the main branch:
-
git rebase main
-
-
--
-
Resolve Conflicts:
-
-- Address any conflicts encountered during the rebase process by following the prompts in your terminal.
-
-
--
-
Continue Rebase:
-
-- After conflicts are resolved, continue the rebase:
-
git rebase --continue
-
-
--
-
Push the Rebased Changes:
-
-- Push the rebased changes to the remote branch:
-
git push --force origin branch-name
-
-
- What is --force?
- The `--force` flag is used to force push the rebased changes to the remote branch. This is required because the rebase process rewrites the commit history of the branch.
-
-
--
-
Merge the PR:
-
-- Proceed to merge the rebased PR through the repository interface.
-
-
-
-
-
-In the GitHub Pull Request UI, when merging a pull request, three distinct options are available:
-
--
-
Create Merge Commit:
-
-- This option generates a merge commit to incorporate changes. It retains the commit history of the merged branch, clearly marking the point where the branches were merged.
-
-
--
-
Squash and Merge:
-
-- Choosing this option condenses multiple commits from the feature branch into a single commit before merging. It helps maintain a clean commit history, especially for branches with numerous smaller commits.
-
-
--
-
Rebase and Merge:
-
-- The rebase and merge option applies the changes from the feature branch onto the base branch by restructuring the commit history. It produces a linear history, placing the feature branch's commits atop the base branch's latest changes.
-
-
-
-Each option offers a distinct approach to integrating changes from a pull request, allowing contributors to select the method that best aligns with the project's version control and commit history preferences. These options significantly influence the resulting commit history and repository structure.
-
-
-Git's log command offers a view of the commit history. It displays a list of commits in reverse chronological order, showcasing commit messages, authors, and commit hashes. Participants can enter git log in the terminal to view the commit history of a repository, gaining insights into the project's development timeline.
-
-Let's explore the functionality of interactive rebase, focusing on squashing commits, changing messages, deleting, and picking commits in a simple exercise.
-
--
-
Squashing Commits:
-
-- Demonstrate squashing multiple commits into a single one using
git rebase -i HEAD~N, where N is the number of commits to include.
-
-
--
-
Changing Commit Messages:
-
-- Show how to alter commit messages during an interactive rebase by marking commits as
reword.
-
-
--
-
Deleting Commits:
-
-- Guide participants on how to delete specific commits from the commit history using
git rebase -i.
-
-
--
-
Picking Commits:
-
-- Illustrate how to pick specific commits and rearrange their order using
git rebase -i.
-
-
-
-
-
- 
+
+## Steps
+
+Before we begin writing the code, let's analyze the website and identify the structure of the page. We can see that the website has a list of upcoming events. Each event has a title, a date, a location, a description, an image, and tags. For each unique aspect of the event, we will try to identify the corresponding selector.
+
+An event can have multiple tags that should be stored in an array.
+
+At the bottom of the page, we have a button that loads more events, so no navigation is required. We see that if we click it multiple times, we can load all the events. Consider having all the events loaded before scraping the data.
diff --git a/src/images/is_this.webp b/src/images/is_this.webp
deleted file mode 100644
index e8c0e59..0000000
Binary files a/src/images/is_this.webp and /dev/null differ
diff --git a/src/images/olx.png b/src/images/olx.png
new file mode 100644
index 0000000..085f0f9
Binary files /dev/null and b/src/images/olx.png differ
diff --git a/src/images/programmer_move.jpeg b/src/images/programmer_move.jpeg
deleted file mode 100644
index acb4962..0000000
Binary files a/src/images/programmer_move.jpeg and /dev/null differ
diff --git a/src/images/progremr.jpeg b/src/images/progremr.jpeg
deleted file mode 100644
index 2f8af30..0000000
Binary files a/src/images/progremr.jpeg and /dev/null differ
diff --git a/src/images/upcoming_events.png b/src/images/upcoming_events.png
new file mode 100644
index 0000000..ec0f7a8
Binary files /dev/null and b/src/images/upcoming_events.png differ
diff --git a/src/web-scraping.md b/src/web-scraping.md
new file mode 100644
index 0000000..dc9c707
--- /dev/null
+++ b/src/web-scraping.md
@@ -0,0 +1,148 @@
+# Setting up constants
+
+Welcome to the first chapter of the Web Scraping section! In this section, we will find all the anouncements listed on OLX for the apartments in Cluj-Napoca. We will use Puppeteer to scrape the data from the website and save it to a JSON file, for easier analysis.
+
+In this chapter, we will set up the constants that we will use throughout the project. These constants will help us maintain consistency and readability in our code. We will define the URLs we want to scrape, the selectors we want to target, and other configuration options. Let's get started!
+
+To begin with, we'll analyze the website, and identify the structure of the page.
+
+
+
+We can see that the website has a list of announcements for apartments in Cluj-Napoca. Each announcement has a title, a price, a location (always Cluj Napoca), and a surface area. For each unique aspect of the announcement, we will try to identify the corresponding selector.
+
+```javascript
+import puppeteer from "puppeteer";
+
+const URL =
+ "https://www.olx.ro/imobiliare/apartamente-garsoniere-de-inchiriat/cluj-napoca/?currency=EUR";
+
+// the values of the selectors are just placeholders, we will update them live during the workshop
+const SELECTORS = {
+ ANNOUNCEMENT: ".offer-wrapper",
+ TITLE: ".offer-title",
+ PRICE: ".price",
+ SURFACE: ".surface",
+};
+```
+
+In the code snippet above, we have defined the URL of the OLX page we want to scrape and the selectors for the different aspects of the announcement. We will use these constants in the subsequent chapters to scrape the data from the website.
+
+## Launching the browser
+
+Now that we have set up the constants, we can proceed to launch the browser and navigate to the OLX page. We will use Puppeteer to automate this process. Let's write the code to launch the browser and navigate to the OLX page.
+
+```javascript
+async function run() {
+ const browser = await puppeteer.launch();
+ const page = await browser.newPage();
+
+ await page.goto(URL);
+
+ await browser.close();
+}
+```
+
+In the code snippet above, we have defined an `async` function called `run` that launches the browser and navigates to the OLX page. We will call this function in the subsequent chapters to scrape the data from the website. The `run` function will be the entry point for our web scraping project, and we will build upon it in the upcoming chapters.
+
+## Retrieving data
+
+Now, we want to retrieve all the announcements from the page. For each announcement, we will extract the title, price, and surface.
+
+If we look carefully at all the annoucements, we can see that not all contain information about the surface. If we access an annoucement that does not contain the surface, we will get an error (puppeteer errors are worse than C Compiler Messages). For the surface, we will wrap everything in a try-catch block, and if we get an error, we will just set the surface to "N/A".
+
+```javascript
+// continue from the previous code snippet, after page.goto(URL)
+const data = [];
+
+await page.waitForSelector(SELECTORS.ANNOUNCEMENT);
+
+const announcements = await page.$$(SELECTORS.ANNOUNCEMENT);
+
+for (const announcement of announcements) {
+ let surface = "N/A";
+
+ const title = await announcement.$eval(
+ SELECTORS.TITLE,
+ (el) => el.textContent
+ );
+ const price = await announcement.$eval(
+ SELECTORS.PRICE,
+ (el) => el.textContent
+ );
+ try {
+ surface = await announcement.$eval(
+ SELECTORS.SURFACE,
+ (el) => el.textContent
+ );
+ } catch (e) {
+ // do nothing
+ }
+
+ data.push({
+ title: title,
+ price: price,
+ surface: surface,
+ });
+}
+```
+
+Alright! We have successfully retrieved the data from the first page. But there are a lot more pages to scrape, and with the rate of rentals in Cluj, even the announcement from the last page are at most 2 weeks old. We will need to navigate to the next page and retrieve the data from there as well.
+
+## Navigation to another page
+
+OLX paginates data. We will need to navigate to the next page and retrieve the data from there as well. At the bottom of the page, there is an arrow that moves to the next page. Inside the arrow, there is an anchor tag that we can click to navigate to the next page, for which we'll want to retrieve the url. Furthermore, on the last page, the arrow does not exist, so this is a good way to check if we still have pages to scrape.
+
+First step is to update the selectors with the new ones.
+
+```javascript
+const SELECTORS = {
+ ANNOUNCEMENT: ".offer-wrapper",
+ TITLE: ".offer-title",
+ PRICE: ".price",
+ SURFACE: ".surface",
+ NEXT_PAGE: ".pager .next a",
+};
+```
+
+Next, we need a cycle to navigate through all the pages. We will use a `while` loop to navigate to the next page until the "Next" button is no longer available.
+
+```javascript
+while (true) {
+ console.log("Scraping page " + page.url());
+ // code is exactly the same as before
+
+ // navigate to the next page
+ // since the element might not be available, we will wrap it in a try-catch block
+ try {
+ const nextPageURL = await page.$eval(SELECTORS.NEXT_PAGE, (el) => el.href);
+ await page.goto(nextPageURL);
+ } catch (e) {
+ console.log(e);
+ break;
+ }
+}
+```
+
+Cool! Now we have all the data from all the pages. We can save it to a JSON file for further analysis.
+
+## Saving the data
+
+Finally, we will save the data to a JSON file. We will use the `fs` module to write the data to a file named `data.json`.
+
+```javascript
+// at the beginning of the file
+import fs from "fs";
+
+async function run() {
+ // code to scrape the data
+
+ fs.writeFile("data.json", JSON.stringify(data), (err) => {
+ if (err) throw err;
+ console.log("File Saved");
+ });
+
+ await browser.close();
+}
+```
+
+That's it! We have successfully set up the constants, launched the browser, retrieved the data, navigated to the next page, and saved the data to a JSON file.
diff --git a/src/web-scraping/constants.md b/src/web-scraping/constants.md
new file mode 100644
index 0000000..27db1a3
--- /dev/null
+++ b/src/web-scraping/constants.md
@@ -0,0 +1 @@
+# Setting up constants
diff --git a/src/web-scraping/launching-browser.md b/src/web-scraping/launching-browser.md
new file mode 100644
index 0000000..48b6808
--- /dev/null
+++ b/src/web-scraping/launching-browser.md
@@ -0,0 +1 @@
+# Launching the browser