Cited rather than ranked

A search engine returns a list and your page occupies a position in it. An answer engine does something structurally different: it retrieves passages from a number of documents, writes prose, and attributes the passages it relied on. There is no list, no position, and no stable thing to hold. The goal is therefore to be cited rather than ranked, and two identical questions asked an hour apart can cite different sources without either answer being wrong.

That difference is not a detail about reporting. It changes the unit of work from the page to the section, and it makes a large part of what is currently sold as AI search optimization either unmeasurable or already false. It also makes some of the work cheaper than the search era, because the decisions that matter most are readable from files you already have.

Which crawler decides whether you are cited?¶

Citation is decided by retrieval at the moment the question is asked, not by anything in a model's weights, and those two things are reached by different crawlers. A training crawler collects text that may eventually inform a model and arrives with no link attached. A search or retrieval crawler builds the index an engine queries while composing an answer, and that is the one that produces an attribution.

The useful consequence is that the two decisions are separable. An organization can refuse every training crawler and keep every citation, which is what a good many publishers want and few have configured. The failure we see most often is that decision made backwards. The well-known training agent is blocked because somebody read an article about it, and the retrieval agent that decides citation is left unexamined. The model learns from the site and no answer ever links to it.

Some of the tokens most often added to a robots file do not do what their names suggest, and this is worth checking rather than assuming. Of Google-Extended, Google writes (opens in a new tab) that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search", and describes it as controlling whether crawled content may train future Gemini models. It is a training control, so blocking it protects nothing about citation and costs nothing either. A block is worth what the vendor wrote down and no more.

There is a second failure that a robots file cannot show you, and it is common enough to check every time. The file permits the fetch and the edge refuses it, because a firewall rule treats an unfamiliar agent as a bot and returns a 403. Nothing in the configuration is wrong on paper. The engine simply never gets a page.

Why is the section the unit rather than the page?¶

The section is the unit because retrieval works on passages, and a passage is lifted away from everything around it. A section that begins “This is handled automatically” had a referent in the paragraph above it, and the reader of a retrieved chunk never saw that paragraph. What arrives is a fragment that does not say what it is about, and an engine composing an answer will prefer one that does.

So the edits that matter are unglamorous and structural. Make each heading a question somebody would actually type rather than a label like Overview. Put the answer in the first sentence rather than building to it. Name the subject inside the body of each section and not only in the page title, or every passage retrieved from the page is anonymous. Write so that any section can be read alone, because that is the only way it will be read.

This is the same discipline that makes documentation usable by a person who arrived from a search result, which is why it rarely feels like optimization work. Nothing here asks anybody to write worse.

What has actually been measured?¶

One controlled experiment underwrites the content side of this, and it is worth citing precisely because so much of what surrounds the subject is assertion. Aggarwal and colleagues, at KDD 2024 (opens in a new tab), tested nine content edits across a benchmark of diverse queries and multiple domains, and report that their methods “boost visibility by up to 40% in generative engine responses”.

Two of their results are more useful than the headline. Quoting somebody named and adding a statistic with its source attached were among the strongest edits, and citing sources was close behind. Keyword stuffing scored below making no change at all, which is the one finding that should settle an argument with anybody proposing it.

The result that changes strategy, though, is that the same edit was worth opposite amounts depending on where a page already stood: substantially positive for a page ranking fifth and negative for one already ranking first. Generative answers are a leveler. If you are the incumbent, this work defends a position rather than extending it, and expecting the headline figure is how the budget gets misjudged.

One caution about reading the paper or anything quoting it. Its figures are reported in the units of its own metric against a do-nothing baseline, and the secondary writing about it routinely converts them into percentage improvements they are not. A number quoted with no baseline beside it has usually been through that conversion.

Does any of this need new files or markup?¶

No, and Google says so in as many words. Its own guidance on AI features (opens in a new tab) states that "you don't need to create new machine readable files, AI text files, or markup to appear in these features", and adds that there is no special structured data required for it. That is worth knowing before paying anybody to build it.

Publishing an llms.txt is a reasonable thing to do if you want a tidy index of your own documentation, and it should be described as exactly that. It is not a ranking lever, no major engine has committed to consuming it, and presenting it as answer-engine work is the kind of claim that is checkable and embarrassing later.

Structured data still earns its place for a different reason. It disambiguates which entity a page is about, which matters a great deal when an organization shares a name with something better known. That is an identity problem rather than a visibility lever, and it is the one markup job worth doing here.

How would you know it worked?¶

You would know by measuring a citation rate, per engine, over repeated runs, with an interval reported beside it. An engine holds no ranking and answers differently on the next run, so a single observation is a sample of one from a distribution nobody has characterized.

That has an uncomfortable implication worth stating before somebody else does. Establishing a level to any useful precision takes more runs per query than most reporting budgets assume, which is why comparing two states is far cheaper than quoting one. Measure before and after on the same queries and the same engines, report the direction, and say plainly when the intervals overlap rather than claiming a move.

It is also worth separating three problems that look identical in a report. A query you are never cited on at all is a coverage problem, and no editing of an existing page closes it. A query where a comparison site nobody can edit leads every answer is an off-domain problem, and the effort belongs somewhere other than your own pages. Only the third case, where you are cited but rarely first, is the one that page work actually fixes.