Unable to read the content for custom_robots.txt file

Issue

I am getting the following message in my log file:

[RobotsUtil:47] Unable to read the content for custom_robots.txt

However, the same does not happen in other environments. What can be the reason?

That exception is thrown in RobotsUtil.java class. Either for robots.txt without sitemap, or with sitemap.
In the first case (without sitemap):
- It means that layoutset is null when reaching RobotsUtil.java#L40
- Therefore, it tries to get the file set in robots.txt.without.sitemap property.
  - If the file cannot be found, it prints the aforementioned error message.
In the second case (with sitemap):
- It means that layoutset is not null in RobotsUtil.java#L40
- It retrieves the robots.txt with sitemap here:
  - From the layoutset's settings, if they have been already modified.
  - Otherwise, from the file indicated by the robots.txt.with.sitemap property.
    - If the file cannot be found, it prints the aforementioned error message.

See RobotsAction.java. Basically, it depends on the virtual host used to retrieve that information, and the configuration you have set for them (at company and site level). See Configuring Virtual Hosts Site URLs and Configuring Virtual Instances.

If you want to leverage robots.txt.without.sitemap or robots.txt.with.sitemap properties, take into account that Liferay tries to get those files using the ClassLoader as base path. (see RobotsUtil.java#L43).
For example, when using Tomcat, your custom robots.txt file should be placed in tomcat/webapps/ROOT/WEB-INF/classes/ (or a relative path from there).