llms.txt and AI crawler access for UAE websites
Many UAE sites block AI crawlers without knowing it, through a blanket rule copied into robots.txt or a firewall that rejects unfamiliar user agents. If OAI-SearchBot or PerplexityBot cannot fetch a page, it cannot be cited. We set access deliberately and publish an llms.txt file to describe the site.
What is llms.txt and does it help AI visibility?
llms.txt is a proposed convention: a markdown file at /llms.txt that summarises what the site is and links to its most important pages in a form language models can read easily. It is not a ranking factor and no major assistant has committed to using it for citations. Crawler access in robots.txt, by contrast, decides whether OAI-SearchBot, ClaudeBot, PerplexityBot and Bingbot can fetch your pages at all, and that is where the impact lies.
What we do for you
AI user agent audit
Robots.txt, firewall and CDN rules tested against GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Bingbot and Meta-ExternalAgent.
Access policy document
A written decision per crawler, separating search-time fetching from model training, so the business knows what it allowed and why.
Corrected robots.txt
Rules rewritten so search-time crawlers are allowed and any training opt-outs are applied precisely rather than by blanket block.
Render check
Confirmation that key page content is present in the HTML response, not only after JavaScript runs, for crawlers that do not execute scripts.
llms.txt and llms-full.txt
A concise site summary with links to the pages that matter, maintained as pages change, published at /llms.txt.
Log monitoring
Server logs reviewed for AI crawler visits so you can see which assistants fetch which pages and how often.
Prompts and queries we optimise for
Three steps, no black box
Test
We fetch your key pages as each AI user agent and record which are blocked, rate-limited or empty.
Decide and fix
You approve a per-crawler policy and we apply it in robots.txt, firewall and CDN settings.
Publish and monitor
llms.txt goes live and crawler visits are reviewed in server logs each month.
Questions UAE business owners ask
Answered in the first sentence, so AI assistants can quote us directly.
That is a legitimate choice. GPTBot gathers pages that may be used to train OpenAI models, while OAI-SearchBot fetches pages to show and cite in ChatGPT search. Blocking GPTBot and allowing OAI-SearchBot keeps you citable while opting out of training. The same distinction applies to Google-Extended, which controls Gemini training only and does not affect Search or AI Overviews.
There is no evidence that it does today. It is a convention proposed by the community, not a standard adopted by OpenAI, Google or Anthropic for ranking or citation. We add it because it costs little, it documents the site clearly and it may be picked up by tools in future. The measurable gains come from crawler access and page content.
Often, yes. Bot-protection settings on Cloudflare and similar services can challenge or reject AI crawlers even when robots.txt allows them, and the site owner sees nothing. We test by fetching pages with each user agent from outside your network and review the CDN rules, then allow the search-time crawlers you have chosen while keeping protection for abusive traffic.
We check your site against your three closest Dubai competitors and send the gaps costing you leads to your WhatsApp within 48 hours. Free, no call required.
The Google data behind the claims
Google Search Console performance panels from client accounts — Google's own screen, not a report we wrote. Organic search only, cropped so no brand or domain is identifiable, each labelled with the exact window it covers.
