Unlimited

Robots.txt Generator

Select your framework and what to block. Get a copy-paste robots.txt.

What should be blocked?

Select paths that search engines should not crawl.

API routes

/api/

Admin & dashboard pages

/admin/, /dashboard/

Auth pages

/login, /signup, /auth/

Private user pages

/settings, /account, /profile

Internal framework assets

Framework-specific build directories

Search results page

/search

Staging & preview content

/staging, /preview

Full URL to your sitemap. Leave empty to omit.

Time between crawler requests. Most sites don't need this.

robots.txt
User-agent: *

Disallow: /api/
Disallow: /admin/
Disallow: /dashboard/
Disallow: /login
Disallow: /signup
Disallow: /auth/
Where to place this file

Place this file at public/robots.txt in your Next.js project. Alternatively, use app/robots.ts for programmatic generation.

Need more than a robots.txt?
TanStart generates your complete launch kit — favicons, OG images, legal docs, and more. One workspace. Pay per project.
See all features