custom-web-scraper
Web Element Inspector
Pulling every link, or every heading, off a web page normally means writing a program for that one page. This desktop tool makes it point-and-click: type an address, pick how to search, and it drives a real Chrome browser to find every match and outline it on the page. A live scrape against a real site is not demonstrated: the sandbox it was checked in had no internet.
Solo work by Salman Adnan.
Overview
A desktop GUI for pulling elements off a live web page without writing a scraping script for each new site. Type a URL, pick one of seven Selenium locator strategies, and the tool drives a real Chrome browser to find every matching element, landing results in a sortable table with a "Highlight in Browser" button to confirm the match visually.
Most small scraping jobs don't need a script, they need five minutes of poking at a page to see what's there. Writing a one-off Python script for each of these checks gets old, so this tool turns that workflow into a GUI: load a page once, then try locator after locator against it without touching code. It drives an actual Chrome instance through Selenium rather than parsing raw HTML, so it also works on pages that render content with JavaScript.
Key features
- Loads any URL into a real Chrome browser window from inside the GUI.
- Looks up elements by seven Selenium locator strategies: class name, ID, name, tag name, link text, partial link text, and XPath.
- Results open in a separate scrollable table (
ttk.Treeview) showing tag name, visible text, and a summary of key attributes per matched element. - A status bar reports every step ("Loading page...", "Found N elements", or a specific error), colored by severity.
- Session history: the URL and locator fields remember up to 10 recent entries each.
- "Highlight in Browser" outlines every matched element in the live Chrome window and scrolls it into view.
Verification
This is a GUI that drives a real browser, so a full end-to-end run needs a display and a live target site. What was actually checked: main.py compiles cleanly; the pure result-formatting functions (summarize_element, build_result_rows) pass all 7 cases in a standalone unit test suite against fake duck-typed elements, covering truncation, whitespace collapsing, missing attributes, and a simulated stale-element exception; and the full Tkinter window, including the history comboboxes and the status bar, constructs and runs against a fake driver on a live X display. A live Chrome session against a real site was not exercised here, since this sandbox has no outbound network access for Selenium Manager to fetch a chromedriver; the Selenium calls used are standard 4.x API and match documented usage.
Tech stack
A challenge worth noting
Switching from BeautifulSoup's CSS-selector scraping to Selenium's strict locators meant a nonexistent match no longer raises a parse error, it just times out. The fix was wrapping every lookup in WebDriverWait(driver, 10).until(...) and catching TimeoutException specifically, so a missing element surfaces a plain "please enter a valid X" message instead of the GUI hanging. A related fix: the original hardcoded a Windows-only C:\Development\chromedriver.exe path, which now resolves through Selenium Manager by default so the same code runs on Linux, macOS, or Windows without editing a path in source.