Reddit is not allowing Microsoft to crawl their site due to an updated robots.txt file implemented on July 1, 2024, which blocks all web crawlers that lack an agreement with Reddit [2][3][6]. This change aligns with Reddit's revised Content Policy, which prohibits the use of its content for AI training without explicit consent [1]. A Reddit spokesperson indicated that they have struggled to reach agreements with some entities regarding the use of Reddit content, highlighting concerns about user privacy and enforceability [1][3][6]. Consequently, Microsoft confirmed that Bing has halted crawling Reddit following these updates [2][6].

Why is Reddit not allowing Microsoft to crawl their site?

Follow Up Recommendations

Related Content You May Like