Vulnerability: Robots.txt file must not be accessed and should be blocked

Issue

• Encountered a vulnerability issue with the robots.txt file and the vulnerability test suggests preventing the robots.txt file from being accessed.

Environment

• Liferay DXP 7.3
• Liferay DXP 7.4

Resolution

• The robots.txt file is not itself a security threat and it might be a false positive.
• If you want to block access to the Robots.txt file then it can be done on the Web Server level. As a result, kindly contact your Web Server Administrator to get help in blocking access to the Robots.txt file.
• Please note that there is no option available in Liferay OOTB to block the access to Robots.txt file.

Additional Information

• Understand the limitations of a robots.txt file (This is an unofficial article in which the section 'password-protecting private files on your server' explains methods to keep information secure from web crawlers)