I suggest a list of general rules and seo tool for analyzing technical
condition and features for compiling a file: a text document robots.txt should
be in UTF-8 or ASCII; The name of the bots can be written with a capital
letter or a small one (Google = google); After the User-agent, you can specify
more than 1 robot (User-agent: GoogleMedia GoogleImage); if the bot after the
User-agent finds its name, it already ignores the common commands with "*";
file weight should be a maximum of 500 KB (according to the requirements of
Google);