Should you unblock AI crawlers before checking what they can read?
You may be looking at your Shopify `robots.txt` because an AI assistant cannot seem to find your products.
The tempting fix is simple: unblock the crawler.
That may be the right change. But it should not be the first conclusion.
A crawler can be allowed through the front door and still find a product page with no useful details, a broken price, or a policy page it cannot reach. The better order is to inspect the door, then inspect the rooms behind it.
We took this approach while building our free scanner. It reads only public URLs: `robots.txt`, the catalogue endpoint, product markup, and policy pages. No install and no login. That constraint was useful. If a public reader cannot get a clear answer from those pages, changing a crawler rule alone will not fix it.
Start with the live robots.txt file
Open this URL in a browser:
`https://yourstore.com/robots.txt`
Do not start in the theme editor. Read the file your live store actually sends.
Shopify's standard rules generally allow public product, collection, page, blog, and policy pages to be crawled. They block areas such as `/admin`, `/cart/`, `/checkout`, `/account`, and `/orders`. They can also block duplicate collection URLs created by filters or sorting.
That is a sensible default. Crawlers do not need access to a customer's cart or account area.
Look for two things:
- Rules aimed at a specific crawler, such as `OAI-SearchBot` or `GPTBot`
- Broad rules under `User-agent: *` that block paths you expect to be public
For example, this blocks one named crawler from all pages:
```text User-agent: ExampleBot Disallow: / ```
This allows it:
```text User-agent: ExampleBot Allow: / ```
Treat `robots.txt` as a request, not a lock. Shopify says AI crawler rules are advisory, and not every crawler is guaranteed to follow them. It is still the right place to state your preference.
There is one detail worth checking carefully: policy pages. Shopify's current default-rule summary does not list `/policies/` as blocked. But Shopify also notes that an older custom `robots.txt.liquid` template may still output `Disallow: /policies/`.
So search the live file for `policies` before assuming your returns, shipping, or privacy pages are available.
Decide what you mean by “AI crawler”
Not all bot rules do the same job.
OpenAI separates ChatGPT search visibility from training controls. `OAI-SearchBot` controls eligibility for ChatGPT search results. `GPTBot` concerns content that may be used for training. They are separate settings.
That means blocking `GPTBot` does not automatically mean you have blocked ChatGPT search. The reverse is also true.
If you change a rule for OpenAI search, OpenAI says it can take about a day for the change to adjust. That is another reason to check the pages first. You can use that time to fix what the crawler will encounter.
Open a real product page without being logged in
Next, open one product URL in a private browser window. Use a normal, currently available product.
You are checking the page as a new visitor would see it. No customer account. No Shopify admin session. No preview link.
Look for these details on the page itself:
- Product name
- Price
- Availability
- Main features
- Detailed specifications
- Materials
- Sizing or dimensions
- Care instructions, where relevant
- A useful description
Shopify recommends these details on product pages for AI discovery. They also help a person decide whether the product suits them.
Say your store sells a linen shirt. “Summer shirt” is not much for a system to work with. “Relaxed linen shirt, 100% linen, button front, available in five sizes, machine wash cold” gives it something concrete.
Then check the page source or use a structured-data test tool to see whether the product markup is present. Shopify requires product templates to include Google's rich product snippets. Shopify also says supported themes include product microdata for details such as price, availability, and reviews.
You do not need to become a schema specialist for this first check. Search the page source for one of these terms:
```text price availability Product ```
If the visible product details exist but the markup is absent or clearly wrong, that is a theme issue worth investigating.
Check the page response, not just what your browser shows
A product page must be publicly accessible to be crawled. Google lists a successful HTTP response, such as `200`, and indexable content among its technical requirements. A page that requires a login is not crawlable by Googlebot.
For a quick check, paste a product URL into a browser while logged out. If it redirects to an account page, shows an access message, or fails to load, solve that before changing crawler permissions.
This matters for products with special access rules. Shopify says products that require customer authentication, products in private stores, and some B2B-only products can be excluded from agentic storefronts.
Also check the product's status in Shopify admin. An Unlisted product is hidden from internet search, Shopify Catalog, the sitemap, storefront search, collections, and recommendations. A product can look fine when you open it directly and still be missing from the places that help it get discovered.
Check whether the catalogue can be found
Open:
`https://yourstore.com/sitemap.xml`
Shopify automatically creates this root sitemap. It links to separate sitemaps for products, collections, blogs, and webpages.
Click through to the product sitemap and search for the product handle you just reviewed. If it is missing, do not jump straight to `robots.txt`. Check whether the product is published, active, and not Unlisted.
This is also where a quiet technical mistake can cause trouble. While building our own `llms.txt` builder, we found that Shopify's `/products.json` returns at most 250 products per page. A catalogue read can look complete when it is not. We had to handle pagination rather than treating the first response as the whole store.
The lesson is broader than that endpoint: never call a catalogue complete because one page of results looked plausible.
Shopify also now serves `/agents.md`, `/llms.txt`, and `/llms-full.txt` on every store. By default, these include the store name, store URL, sitemap link, policies, and discovery endpoints.
Open those live URLs too. They are not a substitute for complete product pages. They are a quick way to see what your store is publishing as a starting map.
Keep crawler access separate from Shopify Catalog
There are two related but different questions:
1. Can a web crawler read your public pages? 2. Is your product data available through Shopify Catalog for agentic storefronts?
Unblocking a web crawler does not control Shopify Catalog distribution. Shopify says Catalog is the primary product-data source for agentic storefronts, and blocking AI crawlers through `robots.txt` or a network layer does not stop activated Catalog sharing.
That distinction prevents an unhelpful change. Do not alter bot rules expecting it to manage Catalog sharing. Check the relevant Shopify settings and product eligibility instead.
A better order of work
Before changing a crawler rule, spend a few minutes on this sequence:
1. Open `/robots.txt` and read the live rules. 2. Check whether named bots have different rules from `User-agent: *`. 3. Open a live product page while logged out. 4. Confirm price, availability, specs, materials, and sizing are visible. 5. Check that product markup includes useful product details. 6. Open policy URLs and confirm they are public. 7. Open `/sitemap.xml` and find the product sitemap. 8. Check whether the product is active, published, and not Unlisted. 9. Open `/agents.md` and `/llms.txt` to see the store's published discovery map. 10. Only then decide whether a `robots.txt` change is needed.
Opening the door matters. But first make sure there is a readable store on the other side.