Why is Reddit not allowing Microsoft to crawl their site?

An image showing the Reddit logo on a red and white background

Reddit is not allowing Microsoft to crawl their site due to an updated robots.txt file implemented on July 1, 2024, which blocks all web crawlers that lack an agreement with Reddit[2][3][4]. This change aligns with Reddit's revised Content Policy, which prohibits the use of its content for AI training without explicit consent[1]. A Reddit spokesperson indicated that they have struggled to reach agreements with some entities regarding the use of Reddit content, highlighting concerns about user privacy and enforceability[1][3][4]. Consequently, Microsoft confirmed that Bing has halted crawling Reddit following these updates[2][4].

Follow Up Recommendations