Web Scraping
jamditis/claude-skills-journalismGetting information from websites can be tricky when they try to block automated tools. This method uses a smart scraping cascade of different techniques to overcome blocks. It starts with fast methods and falls back to slower JavaScript-rendering ones when needed.
You can extract text, titles, and other content from most pages. The code includes anti-bot bypass features like rotating user agents and stealth mode. It also handles paywalls and social media sites.
The tools used are Trafilatura for simple pages, Requests for basic HTML, and Playwright with stealth mode for complex JavaScript sites. This approach ensures you get the data you need reliably and ethically.
Add Web Scraping skill to your workflow
Global
mkdir -p ~/.claude/skills/web-scrapingProject
mkdir -p .claude/skills/web-scrapingSource Repository
More Browser Automation & Scraping Claude Code Skills
Agent Browservercel-labs/agent-browser
Fast browser automation tool for AI agents to control any website
Develop Userscriptsxixu-me/skills
Build and debug browser userscripts for Tampermonkey and ScriptCat
Just Scrapescrapegraphai/just-scrape
Search, scrape, crawl, and monitor web pages with an easy command line tool
Browser Actbrowser-act/skills
AI agents use browser-act to automate web tasks like fetch forms screenshots and complex workflows
Firecrawlfirecrawl/cli
Search, scrape, and crawl websites with one simple command line tool
Playwright Climicrosoft/playwright-cli
Automate browser tasks and test web pages with simple terminal commands
Browser Usebrowser-use/browser-use
Automate web testing, form filling, and screenshots with simple CLI commands
Browser Act Skill Forgebrowser-act/skills
Explore any website once and create reusable skill packages forever