Up-to-date information and analysis on recent happenings in news, politics, science, and culture worldwide.

Reddit is not allowing Microsoft to crawl their site due to an updated robots.txt file implemented on July 1, 2024, which blocks all web crawlers that lack an agreement with Reddit. This change aligns with Reddit's revised Content Policy, which prohibits the use of its content for AI training without explicit consent. A Reddit spokesperson indicated that they have struggled to reach agreements with some entities regarding the use of Reddit content, highlighting concerns about user privacy and enforceability. Consequently, Microsoft confirmed that Bing has halted crawling Reddit following these updates.

Non-Google search engines blocked from showing recent Reddit results

Reddit is now blocking major search engines and AI bots — except the ones that pay

Bing blocked by Reddit, breaking search results for everyone but Google

Microsoft confirms Reddit blocked Bing Search

The web archive contains over 866 billion web pages according to archived data, and as of January 3, 2024, the Wayback Machine has archived more than 860 billion web pages as noted on Wikipedia.

Current Events

Why is Reddit not allowing Microsoft to crawl their site?

How many web pages are in web archive?