Robots Txt
kostja94/marketing-skillsRobots.txt files tell search engines and AI crawlers which parts of your website they can visit. This skill helps you set up, check, and fix those rules. You can block unwanted crawlers like GPTBot or allow Googlebot to reach specific pages.
Getting robots.txt right keeps your important content visible while protecting private areas. It works alongside noindex tags but controls crawling, not indexing. A mistake like blocking pages that use noindex can stop search engines from seeing your site at all.
Whether you manage a small blog or a large e-commerce site, this guide walks you through best practices, common errors, and how to handle AI crawlers. You will learn the difference between blocking a whole folder and hiding a single page.
Add Robots Txt skill to your workflow
Global
mkdir -p ~/.claude/skills/robots-txtProject
mkdir -p .claude/skills/robots-txtSource Repository
More Marketing & SEO Claude Code Skills
Twitter Automationhalt-catch-fire/skills
Automate posting, liking, retweeting, and following on Twitter from your command line
Twitter Automationqu-skills/skills
Control your Twitter account with simple commands. Post, like, follow, and more
Twitter Automationskills-shell/skills
Automate your Twitter account with simple commands for posting and engagement
Seo Auditcoreyhaines31/marketingskills
Find and fix SEO issues to boost your Google rankings and traffic
Copywritingcoreyhaines31/marketingskills
Write clear and compelling marketing copy that drives action and converts visitors
Viral Short Formvyralcontent/content-skills
Brainstorm high-retention hooks and scripts for TikTok, Reels, and Shorts
Marketing Psychologycoreyhaines31/marketingskills
Apply proven psychological principles and mental models to ethical marketing campaigns
Content Strategycoreyhaines31/marketingskills
Plan content that drives traffic, builds authority, and generates leads