Python Web Scraping with Selenium
Selenium is a powerful tool for controlling web browsers through programs and performing browser automation. Unlike BeautifulSoup, which works with static HTML, Selenium can interact with the page (click buttons, fill forms) and scrape dynamic content rendered by JavaScript.
Installation
pip install seleniumYou will also need a WebDriver for your browser (e.g., ChromeDriver for Google Chrome).
Basic Usage
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
# Initialize the driver (e.g., Chrome)
driver = webdriver.Chrome()
# Open a page
driver.get("http://www.python.org")
# Get the title
print(driver.title)
# Close the browser
driver.quit()Locating Elements
Selenium provides various strategies to locate elements on a page using the By class.
# Find element by name attribute
elem = driver.find_element(By.NAME, "q")
# Find element by ID
content = driver.find_element(By.ID, "content")
# Find element by CSS Selector
logo = driver.find_element(By.CSS_SELECTOR, "div.header")Interacting with Elements
You can interact with elements by typing into them, clicking them, etc.
# Type into an input field
elem.clear()
elem.send_keys("pycon")
elem.send_keys(Keys.RETURN) # Press EnterWaiting for Elements
Since web pages often load content dynamically, it's important to wait for elements to appear before interacting with them.
Explicit Waits
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
try:
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.ID, "myDynamicElement"))
)
finally:
driver.quit()