Web Crawler
starchild-ai-agent/official-skillsMany websites try to block automated data gathering. This skill gets around those blocks. It can pull public posts and transcripts from popular social media platforms like YouTube, TikTok, Instagram, LinkedIn, Reddit and more.
When a normal fetch fails with errors like 403 or a Cloudflare challenge, the skill automatically switches to a special fallback method. This means you get the data you need without asking for manual screenshots.
Use it for YouTube transcripts, TikTok comments, Instagram posts, or any JavaScript-heavy page. It handles tough anti-bot cases so you can focus on your work.
Add Web Crawler skill to your workflow
Global
mkdir -p ~/.claude/skills/web-crawlerProject
mkdir -p .claude/skills/web-crawlerSource Repository
More Browser Automation & Scraping Claude Code Skills
Agent Browservercel-labs/agent-browser
Fast browser automation tool for AI agents to control any website
Develop Userscriptsxixu-me/skills
Build and debug browser userscripts for Tampermonkey and ScriptCat
Just Scrapescrapegraphai/just-scrape
Search, scrape, crawl, and monitor web pages with an easy command line tool
Browser Actbrowser-act/skills
AI agents use browser-act to automate web tasks like fetch forms screenshots and complex workflows
Firecrawlfirecrawl/cli
Search, scrape, and crawl websites with one simple command line tool
Playwright Climicrosoft/playwright-cli
Automate browser tasks and test web pages with simple terminal commands
Browser Usebrowser-use/browser-use
Automate web testing, form filling, and screenshots with simple CLI commands
Browser Act Skill Forgebrowser-act/skills
Explore any website once and create reusable skill packages forever