Web Scraping and Image Collection with Python

For many projects, you need to extract data and images from websites programmatically. This article demonstrates a complete Python solution for web scraping that downloads images and stores them in Azure Blob Storage.

Prerequisites

You'll need to install the following Python libraries:

pip install requests beautifulsoup4 lxml azure-storage

Overview of the Solution

The script performs the following operations:

1. Parse HTML: Use BeautifulSoup to parse web pages
2. Extract image URLs: Find all image links in the HTML
3. Download images: Download images locally with filtering
4. Upload to Azure: Transfer images to Azure Blob Storage
5. Add metadata: Attach metadata to uploaded blobs

Credential Configuration

Create a credentials.json file to store your Azure Storage Account details:

{
  "DC_LOCATION": "westeurope",
  "GROUP_NAME": "my-resource-group",
  "storagacc": {
    "Accountname": "mystorageaccount",
    "AccountKey": "your-account-key-here",
    "Container": "images-container",
    "AZURE_CLIENT_SECRET": "your-client-secret",
    "AZURE_SUBSCRIPTION_ID": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
  }
}

Complete Python Script

1. Imports and Configuration

import requests
from bs4 import BeautifulSoup
from os.path import basename
import os
import sys
import urllib
import urllib2
import urlparse
import argparse
import json
import config
import random
import base64
import datetime
import time
import string
from azure.storage import CloudStorageAccount, AccessPolicy
from azure.storage.blob import BlockBlobService, PageBlobService, AppendBlobService
from azure.storage.models import CorsRule, Logging, Metrics, RetentionPolicy, ResourceTypes, AccountPermissions
from azure.storage.blob.models import BlobBlock, ContainerPermissions, ContentSettings

CURRENT_DIR = os.getcwd()
STORING_DIRECTORY_NAME = "storage_scrapped_images"
STORING_DIRECTORY = CURRENT_DIR + "/" + STORING_DIRECTORY_NAME

if not os.path.exists(STORING_DIRECTORY):
    os.makedirs(STORING_DIRECTORY)

2. Utility Functions

def randomword(length):
    """Generate a random word of specified length"""
    letters = string.ascii_lowercase
    return ''.join(random.choice(letters) for i in range(length))

def randomword(length):
    """Generate random alphanumeric strings"""
    letters = string.ascii_lowercase
    return ''.join(random.choice(letters) for i in range(length))

3. Azure Storage Setup

# Load credentials from JSON file
startdate = time.clock()

metadata_loaded = {
    'Owner': 'Your-Name',
    'Date_Of_Upload': startdate,
    'VAR_2': 'VAL_VAR_2',
    'VAR_3': 'VAL_VAR_3',
    'VAR_4': 'VAL_VAR_4'
}

with open("credentials.json", 'r') as f:
    data = json.loads(f.read())
    StoAcc_var_name = data["storagacc"]["Accountname"]
    StoAcc_var_key = data["storagacc"]["AccountKey"]
    StoAcc_var_container = data["storagacc"]["Container"]

block_blob_service = BlockBlobService(
    account_name=StoAcc_var_name,
    account_key=StoAcc_var_key
)

4. Azure Operations

def copy_azure_files(source_url, destination_object, destination_container):
    """Copy files directly from URL to Azure Blob Storage"""
    blob_service = BlockBlobService(
        account_name=StoAcc_var_name,
        account_key=StoAcc_var_key
    )
    blob_service.copy_blob(
        destination_container,
        destination_object,
        source_url
    )

def upload_func(container, blobname, filename):
    """Upload a local file to Azure Blob Storage"""
    start = time.clock()
    block_blob_service.create_blob_from_path(
        container,
        blobname,
        filename
    )
    elapsed = time.clock() - start
    print("*** DEBUG *** Time spent uploading API %s is: %s in Bucket/container: %s" 
          % (filename, elapsed, container))

5. Web Scraping Core Logic

# Target URL for scraping
URL_TARGET = "https://www.cdiscount.com/search/10/telephone.html"
# Alternative: URL_TARGET = "https://mouradcloud.westeurope.cloudapp.azure.com/blog/blog/category/food/"

base_url = URL_TARGET
out_folder = '/tmp'

# Fetch the webpage
r = requests.get(URL_TARGET)
data = r.text

# Parse HTML with BeautifulSoup
soup = BeautifulSoup(data, "lxml")

# Process all images found in the page
for link in soup.find_all('img'):
    image_url = link.get("src")

    while image_url is not None:
        if 'http' in image_url:
            blocks = []

            # Filter: Only download image files (PNG, JPG, JPEG)
            if image_url.endswith(('.png', '.jpg', '.jpeg')):
                print("->>>>>>>>>>>>>> THIS IS AN IMAGE ... PROCESSING")

                # Download image locally
                file_name_downloaded = basename(image_url)
                file_name_path_local = STORING_DIRECTORY + "/" + file_name_downloaded

                with open(file_name_path_local, "wb") as f:
                    f.write(requests.get(image_url).content)

                # Upload to Azure Blob Storage
                filename_in_clouddir = "uploads/" + file_name_downloaded
                copy_azure_files(
                    image_url,
                    filename_in_clouddir,
                    StoAcc_var_container
                )

                break
            else:
                print("->>>>>>>>>>>>>>  THIS NOT AN IMAGE ... SKIPPING")
                break
        else:
            # Skip local images (those without http in URL)
            print("->>>>>>>>>>>>>> THIS IS A LOCAL IMAGE ... SKIPPING")
            break
        continue

Key Features Explained

1. HTML Parsing

Uses BeautifulSoup with lxml parser for robust HTML parsing:

soup = BeautifulSoup(data, "lxml")
for link in soup.find_all('img'):
    image_url = link.get("src")

2. Image Filtering

Only processes images with specific extensions:

if image_url.endswith(('.png', '.jpg', '.jpeg')):
    # Process image

3. Dual Storage

• Local storage: Downloads to storage_scrapped_images/ directory
• Cloud storage: Uploads directly to Azure Blob Storage

4. Azure Integration

Two methods for Azure storage:

• Direct copy: copy_blob() - copies directly from URL without local download
• Upload from path: create_blob_from_path() - uploads local file

5. Metadata Attachment

Add metadata to Azure blobs:

metadata_loaded = {
    'Owner': 'Your-Name',
    'Date_Of_Upload': timestamp,
    'Custom_Field': 'Custom_Value'
}

Usage Examples

Basic Usage

python scraper.py

Target Specific Website

Modify the URL_TARGET variable:

URL_TARGET = "https://example.com/images"

Configure Storage

Update credentials.json with your Azure Storage details.

Performance Considerations

Optimization Tips

1. Batch Processing: Group multiple uploads for efficiency
2. Parallelization: Use threading for concurrent downloads
3. Error Handling: Add try-except blocks for robustness
4. Rate Limiting: Add delays between requests to avoid blocking
5. Filtering: Implement size checks to skip large files

Enhanced Version with Error Handling

def scrape_with_error_handling(url):
    try:
        r = requests.get(url, timeout=10)
        r.raise_for_status()
    except requests.exceptions.RequestException as e:
        print("Error fetching URL: %s" % e)
        return None

    try:
        soup = BeautifulSoup(r.text, "lxml")
        images = soup.find_all('img')
        print("Found %d images" % len(images))
    except Exception as e:
        print("Error parsing HTML: %s" % e)
        return None

    return images

Advanced Customization

Add URL Validation

def is_valid_image_url(url):
    valid_extensions = ('.png', '.jpg', '.jpeg', '.gif', '.webp')
    valid_domains = ['example.com', 'trusted-source.com']

    if not url.endswith(valid_extensions):
        return False

    parsed = urlparse.urlparse(url)
    if parsed.netloc not in valid_domains:
        return False

    return True

Implement Retry Logic

def download_with_retry(url, max_retries=3):
    for attempt in range(max_retries):
        try:
            response = requests.get(url, timeout=5)
            return response.content
        except requests.RequestException as e:
            if attempt == max_retries - 1:
                raise
            time.sleep(2 ** attempt)  # Exponential backoff

Add Logging

import logging

logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s'
)

logger = logging.getLogger(__name__)
logger.info("Starting scraping process")

Legal and Ethical Considerations

⚠️ Important:

• Check website's robots.txt and terms of service
• Respect Crawl-Delay and User-Agent directives
• Add appropriate delays between requests
• Use a descriptive User-Agent header
• Don't overload servers with rapid requests
• Cache results to avoid repeated downloads

Common Issues and Solutions

Issue: SSL Certificate Verification Failed

# Solution: Disable verification (not recommended for production)
r = requests.get(URL_TARGET, verify=False)

Issue: Images Not Found

# Ensure parser is robust
soup = BeautifulSoup(data, "html.parser")  # Use html.parser instead of lxml

Issue: Azure Authentication Failed

# Verify credentials are correct and account key hasn't expired
block_blob_service = BlockBlobService(
    account_name=StoAcc_var_name,
    account_key=StoAcc_var_key
)

Conclusion

This Python web scraping solution demonstrates:

• HTML parsing with BeautifulSoup
• Image downloading and filtering
• Azure Blob Storage integration
• Error handling and logging
• Performance optimization

Use this as a foundation for building more sophisticated web scraping pipelines for your cloud-based data collection needs.

Repository: Original implementation at Mourad's blog

Web scraping is powerful for data collection, but always respect website policies and rate limits to be a good internet citizen.