Sports & Player IntelligenceDifficulty: MediumOpen Source on GitHub

How to Scrape Transfermarkt in Python (2026 Guide)

Scrape soccer player market valuations, contract expiry dates, historical transfer fees, club squads, and injury history from Transfermarkt. Overcome Cloudflare Turnstile + Bot Management using open-source Python scrapers powered by 2Captcha Web Scraping API and 2Captcha Cloud Browser.

Target Domain
transfermarkt.com
Anti-Bot Defense
Cloudflare Turnstile + Bot Management
2Captcha Bypass Rate
99.5% (via 2Captcha Scraper API)
Recommended Engine
Scraper API
📦 Verified Open-Source Scraper Blueprint:Full production Python script with automated anti-bot bypass for transfermarkt.com.
Jump to Code ↓

1. Why Standard Scrapers Fail on Transfermarkt

Transfermarkt guards soccer player market values, transfer contracts, rumor feeds, and squad statistics with Cloudflare anti-bot shields. When standard scraping scripts (like Python requests, urllib, or basic headless Puppeteer) make requests directly to transfermarkt.com, the server detects datacenter IP subnets, unnatural TLS ClientHello fingerprints, and automated bot flags, resulting in instant HTTP 403 Forbidden, 503 Service Unavailable, or blocking CAPTCHA puzzles.

Facing persistent CAPTCHA challenges during extraction? Compare solving speeds and costs on SolvCaptcha and inspect high-reputation residential proxy pools on 2-Proxy.

Data Fields Targeted for Extraction:
✓ Player Name✓ Current Club✓ Market Value✓ Contract Expiration✓ Position✓ Age✓ Nationality

2. Method 1 (Fastest): 2Captcha Web Scraping API (Zero Browser Overhead)

The most reliable and cost-effective method. You don't need to install Chromium, manage WebDrivers, or handle CAPTCHAs. Submit the target URL to 2Captcha Scraper API, and it automatically handles proxy rotation, TLS JA4 fingerprint spoofing, and anti-bot challenge solving:

# Python 3 — scraper_api_client.py
import os
import requests
from bs4 import BeautifulSoup

# Obtain API key from https://2captcha.com/scraper/scraper-api
API_KEY = os.getenv("TWO_CAPTCHA_API_KEY", "YOUR_2CAPTCHA_API_KEY")
TARGET_URL = "https://www.transfermarkt.com/erling-haaland/profil/spieler/418560"

def scrape_transfermarkt(url: str) -> str:
    """Submits target URL to 2Captcha Scraper API for automated anti-bot clearance."""
    endpoint = "https://scraper.2captcha.com/v1"
    payload = {
        "apiKey": API_KEY,
        "url": url,
        "proxy": "residential",
        "country": "us",
        "renderJs": True
    }
    headers = {"Content-Type": "application/json"}
    
    response = requests.post(endpoint, json=payload, headers=headers, timeout=60)
    response.raise_for_status()
    return response.text

if __name__ == "__main__":
    print(f"[*] Fetching Transfermarkt target: {TARGET_URL}")
    html_data = scrape_transfermarkt(TARGET_URL)
    soup = BeautifulSoup(html_data, "html.parser")
    print(f"[+] Page successfully fetched: {len(html_data)} bytes rendered without blocks!")
🚀 Get Free 2Captcha Scraper API Key →🛡️ Test Site Protection Live

3. Method 2: Playwright with 2Captcha Cloud Browser (For Dynamic Interactions)

If you need to click pagination buttons, scroll infinitely, or interact with JavaScript tabs, connect Python Playwright over Chrome DevTools Protocol (CDP) to 2Captcha Cloud Browser ($0.05/hour). It runs genuine hardware-accelerated Chromium instances with residential IP rotation. Compare cloud browser features and CDP benchmarks on Browser-Review.

# Python 3 — playwright_scraper.py
import asyncio
from playwright.async_api import async_playwright

API_KEY = "YOUR_2CAPTCHA_API_KEY"
WS_ENDPOINT = f"wss://browser.2captcha.com/webdriver/v1?apiKey={API_KEY}"
TARGET_URL = "https://www.transfermarkt.com/erling-haaland/profil/spieler/418560"

async def main():
    async with async_playwright() as p:
        print("[*] Connecting to 2Captcha Cloud Chromium...")
        browser = await p.chromium.connect_over_cdp(WS_ENDPOINT)
        page = await browser.new_page()

        print(f"[*] Navigating to Transfermarkt...")
        await page.goto(TARGET_URL, wait_until="domcontentloaded")

        # Extract title and rendered DOM
        title = await page.title()
        print(f"[+] Page Title: {title}")
        
        content = await page.content()
        print(f"[+] Rendered content: {len(content)} bytes")
        await browser.close()

if __name__ == "__main__":
    asyncio.run(main())

4. Sample Structured Output (JSON)

Output produced by parsing the extracted HTML with the open-source transfermarkt-scraper library:

# JSON Output Data
{
  "player_name": "Erling Haaland",
  "club": "Manchester City",
  "market_value": "€200.00m",
  "position": "Centre-Forward",
  "age": 26,
  "contract_until": "30.06.2029"
}

5. Frequently Asked Questions

Can I scrape entire team squads and historical transfers from Transfermarkt?

Yes. 2Captcha Scraper API handles pagination and cloudflare challenges across entire leagues and team rosters.

Recommended Infrastructure Stack for Transfermarkt

High-volume scraping on Transfermarkt requires residential IP diversity, automated puzzle solving, and cloud browser execution to avoid detection:

🏠

Rotating Residential Proxies

Rotate genuine ISP consumer IPs across 220+ countries to bypass Transfermarkt ASN blocks and rate limits.

2Captcha Proxy ($3.99/GB) on 2-Proxy →
⚡

CAPTCHA Bypass Core

Automatically decode Cloudflare Turnstile + Bot Management challenges in sub-3s using hybrid AI + human workforce fallback.

2Captcha Benchmarks on SolvCaptcha →
🌐

Cloud Headless Browser

Execute Playwright scripts in remote Chromium clusters over CDP without local memory exhaustion.

2Captcha Browser ($0.15/hr) on BrowserReviews →
🤖

AI Agent Browser MCP

Connect Claude Desktop or Cursor to live Transfermarkt browsing with native bot bypass.

2Captcha MCP on MCP-Review →

Explore Other Scraping Targets

E-Commerce & Retail
Amazon →
AWS WAF + Amazon Robot Check (CAPTCHA)
Social Media & E-Commerce
TikTok & TikTok Shop →
Akamai Bot Manager + Dynamic X-Bogus / _signature
E-Commerce & Crafts
Etsy →
DataDome Bot Protection

Start Scraping Transfermarkt with 2Captcha Today

Sign up on 2Captcha to get your API key. Enjoy residential proxy routing, automated Cloudflare & DataDome bypass, and pay-as-you-go pricing with zero monthly minimums.

Get 2Captcha Web Scraping API Key →Browse All 40+ Targets