Is there any way to block crawlers from crawling the appliance? On our main websites, we use robots.txt files, is there anything like that we can use with the appliance?
robots.txt
robots.txt
Settable robots.txt and meta-robots are on the wish list. For now you can completely block crawler machines' access to the appliance using the firewall settings if you know their IP's. Be careful to not block your own access. If you mess up the firewall settings and block yourself you can clear the firewall from the console.