OpenAI Crawler Updates: User-Agent String Changes

OpenAI has updated its crawler documentation. The company clarified that the version number of its user-agent string might change. It may also use a user-agent string with an extra robots.txt marker. For example, the documentation now states "the version number may change" within the user-agent string example. OpenAI also added a section explaining that this robots.txt marker helps site owners identify requests for the robots.txt file more easily. This is especially useful when their logs don't include paths. The updated user-agent string example includes "robots.txt" within it. These updates aim to provide clearer information for site owners.