/EuroSecond-web-Scrape

# EuroSecond Web Scraper & Data Pipeline

A complete, production-ready data extraction and data cleaning pipeline developed under a Linux environment. This project demonstrates the ability to bypass raw, cluttered site structures, perform deep automated crawling, and output fully refined business intelligence data.

## Project Structure & File Descriptions

Inside this repository, the files represent the complete lifecycle of a data engineering pipeline:

### 1. index.html

* *Role:* Raw Source Data Anchor.

* *Description:* This is the initial raw HTML structure of the target website's homepage, captured directly via the Linux Terminal (wget). It serves as the baseline blueprint used to study the Document Object Model (DOM) and extract initial routing pathways before automating requests.

### 2. lite_scraper.py

* *Role:* Phase 1: Structural Extraction Scraper.

* *Description:* A lightweight Python script optimized for low-resource environments. It connects to the target host to scan and extract all primary headers (H2, H3) and embedded hyperlinks (href). It maps out the entire website's raw footprint, ensuring zero data loss at the outer layer.

### 3. eurosecond_data.csv

* *Role:* Raw Unprocessed Dataset.

* *Description:* The direct output generated by Phase 1. This file contains the complete, unfiltered raw data dump. It captures everything visible to the script, including chaotic repeating elements, structural UI text (like "Read More" / "يقرأ"), redundant contact headers, and internal navigation endpoints.

### 4. update_data.py

* *Role:* Phase 2: Deep Scraping & Data Cleaning Algorithm.

* *Description:* The core intelligent engine of the pipeline. It reads the raw routing URLs from eurosecond_data.csv, applies strict conditional filters to completely purge repeating UI text, and isolates authentic product nodes (/product/). It then dynamically initializes safe HTTP connection loops to deeply crawl individual product pages, parsing localized HTML components to isolate product names and actual retail prices.

### 5. cleaned_products.csv

* *Role:* Final Refined Business Dataset.

* *Description:* The ultimate delivery asset of this pipeline. A completely normalized, noise-free spreadsheet structured into explicit business features: [Name, Price, Description, URL]. All redundant structural artifacts are eradicated, leaving only direct, actionable wholesale apparel product lines, exact weights, and Euro-denominated pricing models ready for commercial analysis.

---

## Technical Specifications

* *Environment:* Linux Lite Terminal

* *Core Language:* Python 3

* *Primary Libraries:* BeautifulSoup 4 (bs4), Requests, CSV Module

* *Architecture:* Modular ETL (Extract, Transform, Load) Pipeline