Search and AI visibility¶
Every public page of a SightPress website is ready for search engines and for AI search engines and assistants, with nothing to set up. This page explains what SightPress does for each page, and the settings that change it.
Which pages are shown¶
Search engines and AI readers are only ever offered pages that are:
- published (drafts and pages waiting for review never are);
- public;
- not marked "Ask search engines and AI search engines not to list this page";
- not a copy of a page on another website (a page with an "Address of the original").
While SightPress runs anywhere other than production (the APP_ENV setting, for example on your own computer), every
page asks search engines not to list it, and robots.txt keeps crawlers out of the whole website.
What every page has¶
- A title: the page's "Title in search results" if it has one, otherwise its title followed by the website's name, such as "About us - Riverside Library".
- A description: the page's "Description for search engines", otherwise its summary, otherwise its first words. Longer text is cut at a whole word, at 160 characters. Social networks get the same description.
- One address (the canonical address). An address ending in
/moves permanently to the one without it, and the website's other domains move permanently to its first domain. - Structured data that search engines read: the page, where it sits in the website, who runs the website (from the site settings) and the pictures, videos and recordings on it.
- "Last updated" under the content, in the reader's time zone.
- A Markdown version for AI readers: add
.mdto the page's address, such as/about.md, or/index.mdfor the home page. It has the title, the summary, every block as plain text (pictures with their alternative text, videos and recordings with their transcripts, files with their type and size) and the page's address and date. Contact forms are left out.
Files for search engines and AI readers¶
| Address | What it is |
|---|---|
/sitemap.xml |
Every page search engines may list, in every language, with the date it last changed. |
/robots.txt |
Keeps crawlers out of the admin panel, accounts and sign-in pages, points them to the sitemap, and names the AI crawlers the website keeps out. |
/llms.txt |
The website's name and summary, then a link to the Markdown version of every page that may be listed, with its description. |
/llms-full.txt |
The same pages in full, as Markdown, up to 500 KB. If it stops early, it says how many pages were left out. |
The files are made again each time a page is created, saved, moved, published or unpublished.
Settings¶
Admins change these under Site settings, in the Search & AI section.
AI crawlers¶
Whether AI services may read the public pages, to answer people's questions with a link back (AI search and assistants) and to train their models.
| Choice | What robots.txt asks |
|---|---|
| Allow all (the default) | Nothing extra: every crawler may read the public pages. |
| AI search only | Crawlers that train AI models, such as GPTBot, ClaudeBot, Google-Extended and CCBot, stay out. AI search and assistants may read the pages and link to them. |
| Block all | Every AI crawler stays out: those that train models, and AI search and assistant crawlers such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot. |
Search engines are never affected. robots.txt is a request that well-behaved crawlers follow, not a lock.
Search console codes¶
To prove to a search engine that the website is yours, it gives you a code. Paste only the code (the part in
content="…") into Google Search Console verification code, Bing Webmaster Tools verification code or
Yandex Webmaster verification code. SightPress adds it to the home page.
IndexNow¶
Tell search engines at once when a page is published or unpublished (IndexNow) is off by default. When it is on, the website tells the search engines that use IndexNow, such as Bing and Yandex, each time a page is published or unpublished.
Who runs the site¶
The Who runs the site section (a description, full name, contact details and profiles on other websites) goes
into the structured data, and the description also goes at the top of llms.txt. Leave a field empty to keep it
private.
Settings on each page¶
In the page editor's Search & AI section:
| Field | What it does |
|---|---|
| Summary | One or two sentences, shown in lists of pages, used as the description when there is none, and read by AI search engines. |
| Title in search results | Replaces the title in search results only. |
| Description for search engines | Shown under the title in search results. About 160 characters are shown. |
| Picture shown when the page is shared | A picture from the media library. Leave it empty to use the website's share image. Shown only when the media library has pictures. |
| Ask search engines and AI search engines not to list this page | Keeps the page out of search results, the sitemap, llms.txt and llms-full.txt. |
| Address of the original | Only for a copy of a page on another website: its full address. The copy then points search engines to the original, and is left out of the sitemap. |
Before publishing, the page's checks suggest a description and a summary when they are missing, and warn about a description longer than 160 characters or a title another page already has.
Check a website from the command line¶
It lists titles longer than 60 characters, missing or too long descriptions, pages left out of search and why, and whether SightPress runs in production. Two pages with the same title count as a problem, and the command ends with an error.