Robots.txt directives are few and worth knowing precisely. User-agent: names the crawler the following rules apply to (* means all). Disallow: paths the crawler must not fetch — /admin/, /cart/, /wp-admin/. Allow: exceptions inside disallowed paths — Allow: /wp-admin/admin-ajax.php is the classic. Sitemap: the directive pointing crawlers at your XML sitemap — Sitemap: https://yourdomain.com/sitemap.xml — which belongs in every robots.txt because it hands crawlers the curated URL list alongside the rules.
Billing is in INR, and every invoice shows 18% GST. Cluster uptime is on the public status page, with 90-day history. Email support@webji.services is staffed 24/7 in Hindi and English, with an average first reply under 60 minutes. Phone +91 141 4941811 is business hours only.
Robots TXT Generator — What You Get
Robots.txt directives are few and worth knowing precisely. User-agent: names the crawler the following rules apply to (* means all). Disallow: paths the crawler must not fetch — /admin/, /cart/, /wp-admin/. Allow: exceptions inside disallowed paths — Allow: /wp-admin/admin-ajax.php is the classic. Sitemap: the directive pointing crawlers at your XML sitemap — Sitemap: https://yourdomain.com/sitemap.xml — which belongs in every robots.txt because it hands crawlers the curated URL list alongside the rules.
| Directive | Meaning |
|---|---|
| User-agent: * | Rules for all crawlers |
| Disallow: /admin/ | Block that path |
| Allow: /wp-admin/admin-ajax.php | Exception inside a block |
| Sitemap: .../sitemap.xml | Declare the sitemap |
The file lives at the domain root — yourdomain.com/robots.txt — or it does not exist as far as crawlers care. Generate, deploy, then verify in Search Console; the XML sitemap guide covers the sitemap half and the knowledgebase carries the deployment steps.
Who is this for?
Disallow versus noindex is the most consequential distinction in robots.txt work. Disallow stops crawling — the crawler never fetches the page — but it does not remove the page from search results; Google can still list the URL (without content) if links point at it. Noindex is the removal instruction — it must be crawled to be read, which is exactly what Disallow prevents. Block a page in robots.txt that carries a noindex, and you have guaranteed Google can never see the removal order.
The rule: pages you want out of search get meta noindex and stay crawlable; pages that merely waste crawl budget (carts, admin, internal search) get Disallow. The generator above writes the crawl rules; the noindex decision belongs in the page's meta settings, and Search Console verifies which state each URL is actually in.
Why WebJi® for Robots TXT Generator
- 21+ years — serving Indian businesses since 2005 through every market cycle.
- 12,000+ clients from solo bloggers to enterprises, most arriving by referral.
- Everything billed in INR with proper GST invoices for your books.
- 24/7 email support in Hindi/English — average first reply under 60 minutes.
- Free migration and a money-back window on most hosting plans.
Frequently Asked Questions
How do I create a robots.txt file for my website?
The file lives at the domain root — yourdomain.com/robots.txt — or it does not exist as far as crawlers care. Generate, deploy, then verify in Search Console; the XML sitemap guide covers the sitemap half and the knowledgebase carries the deployment steps.
What should I disallow in robots.txt?
Robots.txt directives are few and worth knowing precisely. User-agent: names the crawler the following rules apply to (* means all). Disallow: paths the crawler must not fetch — /admin/, /cart/, /wp-admin/. Allow: exceptions inside disallowed paths — Allow: /wp-admin/admin-ajax.php is the classic. Sitemap: the directive pointing crawlers at your XML sitemap — Sitemap: https://yourdomain.com/sitemap.xml — which belongs in every robots.txt because it hands crawlers the curated URL list alongside the rules.
Where do I upload the robots.txt file?
The file lives at the domain root — yourdomain.com/robots.txt — or it does not exist as far as crawlers care. Generate, deploy, then verify in Search Console; the XML sitemap guide covers the sitemap half and the knowledgebase carries the deployment steps. The question here is: Where do I upload the robots.txt file.
Is robots.txt disallow the same as noindex?
For “Is robots.txt disallow the same as noindex”: Disallow versus noindex is the most consequential distinction in robots.txt work. Disallow stops crawling — the crawler never fetches the page — but it does not remove the page from search results; Google can still list the URL (without content) if links point at it.
How do I block AI bots like GPTBot in robots.txt?
Disallow versus noindex is the most consequential distinction in robots.txt work. Disallow stops crawling — the crawler never fetches the page — but it does not remove the page from search results; Google can still list the URL (without content) if links point at it.
Can robots.txt remove pages from Google?
For “Can robots.txt remove pages from Google”: Disallow versus noindex is the most consequential distinction in robots.txt work. Disallow stops crawling — the crawler never fetches the page — but it does not remove the page from search results; Google can still list the URL (without content) if links point at it.
Is WebJi® an Indian company?
Yes. WebJi® is headquartered in Jaipur, Rajasthan and has operated since 2005. Billing, support and governance all run from India.
Do you provide GST invoices?
Every purchase and renewal gets a GST invoice with our GSTIN, and hosting prices are quoted exclusive of 18% GST. Claim input credit as applicable.