Here are the steps to create a Flask app that scrapes a webpage for a specific keyword in Visual Studio Code:
Use a web framework like Flask to translate the code into a webpage where a form takes the term as input. Here is an example of code for a straightforward Flask application that presents a form for the user to enter the term before scraping a website and returning the keyword’s occurrences:
Here are the steps to create a Flask app that scrapes a webpage for a specific keyword in Visual Studio Code:
- Install Python and Visual Studio Code on your computer, if you haven’t already.
- Create a new folder for your Flask app, and open it in Visual Studio Code.
- Create a new Python file named
app.py
in the root of the folder. - Install the required packages for your app by opening a terminal window in Visual Studio Code (press
Ctrl+Shift+
to open a new terminal) and running the following command:
pip install flask requests beautifulsoup4
This command will install Flask, Requests, and BeautifulSoup packages.
Open the app.py
file and paste the following code:
from flask import Flask, render_template, request
import requests
from bs4 import BeautifulSoup
app = Flask(__name__)
@app.route('/', methods=['GET', 'POST'])
def index():
if request.method == 'POST':
keyword = request.form['keyword']
url = 'https://www.example.com/'
# Send a request to the webpage and get its HTML content
response = requests.get(url)
html_content = response.text
# Parse the HTML content using BeautifulSoup
soup = BeautifulSoup(html_content, 'html.parser')
# Find all instances of the keyword in the webpage
keyword_occurrences = soup.findAll(text=lambda text: text and keyword in text)
return render_template('index.html', occurrences=keyword_occurrences, keyword=keyword)
return render_template('index.html')
if __name__ == '__main__':
app.run(debug=True)
- This code defines a Flask app that handles both GET and POST requests at the root URL (“/”). If the request method is GET, it renders an HTML template that contains a form for the user to input the keyword. If the request method is POST, it retrieves the keyword from the form, scrapes the specified URL, and returns the occurrences of the keyword using a rendered HTML template.
- Create a new folder named
templates
in the root of the folder, and create a new HTML file namedindex.html
inside thetemplates
folder. - Paste the following HTML code into the
index.html
file:
<!DOCTYPE html>
<html>
<head>
<title>Web Scraper</title>
</head>
<body>
<h1>Web Scraper</h1>
<form method="post">
<label for="keyword">Keyword:</label>
<input type="text" name="keyword" id="keyword">
<input type="submit" value="Search">
</form>
{% if occurrences %}
<p>Found {{ occurrences|length }} occurrences of '{{ keyword }}':</p>
<ul>
{% for occurrence in occurrences %}
<li>{{ occurrence }}</li>
{% endfor %}
</ul>
{% endif %}
</body>
</html>
- This HTML template contains a form with a single input field for the user to input the keyword. When the user submits the form, the Flask app will send a POST request to the same URL and handle it with the
index()
function. If there are any keyword occurrences found, they will be displayed in an unordered list on the same page. - Save the
app.py
andindex.html
files. - Run the Flask app by opening a terminal window in Visual Studio Code and running the following command:
python app.py