Keyword Scraper Web App Using Flask

by

Here are the steps to create a Flask app that scrapes a webpage for a specific keyword in Visual Studio Code:

Use a web framework like Flask to translate the code into a webpage where a form takes the term as input. Here is an example of code for a straightforward Flask application that presents a form for the user to enter the term before scraping a website and returning the keyword’s occurrences:

Here are the steps to create a Flask app that scrapes a webpage for a specific keyword in Visual Studio Code:

  1. Install Python and Visual Studio Code on your computer, if you haven’t already.
  2. Create a new folder for your Flask app, and open it in Visual Studio Code.
  3. Create a new Python file named app.py in the root of the folder.
  4. Install the required packages for your app by opening a terminal window in Visual Studio Code (press Ctrl+Shift+ to open a new terminal) and running the following command:
pip install flask requests beautifulsoup4

This command will install Flask, Requests, and BeautifulSoup packages.

Open the app.py file and paste the following code:

from flask import Flask, render_template, request
import requests
from bs4 import BeautifulSoup

app = Flask(__name__)

@app.route('/', methods=['GET', 'POST'])
def index():
    if request.method == 'POST':
        keyword = request.form['keyword']
        url = 'https://www.example.com/'

        # Send a request to the webpage and get its HTML content
        response = requests.get(url)
        html_content = response.text

        # Parse the HTML content using BeautifulSoup
        soup = BeautifulSoup(html_content, 'html.parser')

        # Find all instances of the keyword in the webpage
        keyword_occurrences = soup.findAll(text=lambda text: text and keyword in text)

        return render_template('index.html', occurrences=keyword_occurrences, keyword=keyword)

    return render_template('index.html')

if __name__ == '__main__':
    app.run(debug=True)
  • This code defines a Flask app that handles both GET and POST requests at the root URL (“/”). If the request method is GET, it renders an HTML template that contains a form for the user to input the keyword. If the request method is POST, it retrieves the keyword from the form, scrapes the specified URL, and returns the occurrences of the keyword using a rendered HTML template.
  • Create a new folder named templates in the root of the folder, and create a new HTML file named index.html inside the templates folder.
  • Paste the following HTML code into the index.html file:
<!DOCTYPE html>
<html>
  <head>
    <title>Web Scraper</title>
  </head>
  <body>
    <h1>Web Scraper</h1>
    <form method="post">
      <label for="keyword">Keyword:</label>
      <input type="text" name="keyword" id="keyword">
      <input type="submit" value="Search">
    </form>
    {% if occurrences %}
      <p>Found {{ occurrences|length }} occurrences of '{{ keyword }}':</p>
      <ul>
        {% for occurrence in occurrences %}
          <li>{{ occurrence }}</li>
        {% endfor %}
      </ul>
    {% endif %}
  </body>
</html>
  • This HTML template contains a form with a single input field for the user to input the keyword. When the user submits the form, the Flask app will send a POST request to the same URL and handle it with the index() function. If there are any keyword occurrences found, they will be displayed in an unordered list on the same page.
  • Save the app.py and index.html files.
  • Run the Flask app by opening a terminal window in Visual Studio Code and running the following command:
python app.py

WhatsApp Bot Using Python – Mini Project with code Tutorial

Leave a Comment