Written on 12/12/2014 in Web development

Hi again, This will be Yet Another Small Post about an update to The Vorpal. I received a mail from the Google webmaster tools, smiting me because my robots.txt file wasn't successfully retrieved. One thing to note, I didn't have one, and I thought that not having a robots.txt file, meant there are no disallowed sections. Still, I got this error, and I thought to myself: 'Why not add this little bugger, shouldn't take more than a minute.'. So I made the simplest robots.txt file, effectively saying: 'Hey, you can visit my site any time, robot friends, but stay away from this one url!'.

I added it to my Nancy project's views folder, because I don't like opening physical paths on the server1. And returned it using the Nancy module like this:

Get["/robots.txt"] = _ => { return Response.AsFile("views/robots.txt"); };
Get["/humans.txt"] = _ => { return Response.AsFile("views/humans.txt"); };

This returned the file. So I went back to the all-knowing Google webmaster tools and asked/begged them to reconsider my updated robots file. Google is a merciful god and gave it another shot. This time I didn't get a 404, but an error in the file itself. Which was strange because it was a very simple file, no typo's, no strange characters or quirks. So I went to Google yet again, praying to him for a solution2. Google sent me a scroll depicting the location of a solution to this conundrum and I was grateful beyond believe. It led me straight to milevis' solution.

I followed his commands and opened the robots file in Notepad++, opened the 'Encoding' menu and clicked upon 'UTF-8 without BOM'. After that, I asked The Google webmaster tools for another chance and they granted my request. No errors were left. My struggle ended and all was well in the world.

Since I added a robots file, I also added a humans file, because of reasons... Until next post!

P.S.: I'm sorry for the style in which this post was written. I truly am.3