Submit Feedback
Legacy Knowledge Base
Published Jun. 30, 2025

Large Amounts of Semrush Bot requests

Written By

Justin Mann

How To articles are not official guidelines or officially supporteddocumentation. They are community-contributed content and may not alwaysreflect the latest updates to Liferay DXP. We welcome your feedback toimprove How to articles!

While we make every effort to ensure this Knowledge Base is accurate, itmay not always reflect the most recent updates or official guidelines.We appreciate your understanding and encourage you to reach out with anyfeedback or concerns.

Legacy Article

You are viewing an article from our legacy "FastTrack"publication program, made available for informational purposes. Articlesin this program were published without a requirement for independentediting or verification and are provided "as is" withoutguarantee.

Before using any information from this article, independently verify itssuitability for your situation and project.
Note: please note that Liferay has renamed its Liferay Experience Could offerings to Liferay SaaS (formerly LXC) and Liferay PaaS (formerly LXC-SM).

Issue

  • Several requests coming from the SemrushBot/7~bl web crawler are observed on the site, with the potential of hitting 500,000 requests within 1 week
  • The large amount of requests may lead to stability concerns during the scan

Environment

  • Liferay DXP 7.2
  • Liferay PaaS

Resolution

  • These bots are harmless, but they are scanning some Liferay Cloud sites
  • More information on these bots can be found here: https://www.semrush.com/bot/
  • To prevent them from crawling, use the robots.txt file, as instructed by the site, adding the correct content and syntax, depending on the use case of the sit and tools that are applied
  • If no tools are applied, use the following to block SemrushBots from crawling the site: 
    User-agent: SiteAuditBot
    Disallow: /
Did this article resolve your issue ?
Legacy Knowledge Base
Did this article resolve your issue ?