Been getting some questions about my robots.txt file and what certain things do.
Thankfully some regular expressions are supported in the robots.txt (but not many).
$ in regex means the end of the file. So if you do .php$ it your robots.txt that means it will match anything that ends in .php
This is really handy when [...]