Major Websites Blocking Content from AI Crawlers

Amazon, Quora and Indeed are three major websites that prohibit ChatGPT’s bot.

By Rick Richardson
Technology This Week

According to recent data from content detector Originality.AI, nearly 20 percent of the top 1000 websites in the world are restricting crawler bots that collect web data for AI services.

MORE: What Is an HEIC File? | AI + MRI = Diseases That Doctors Might Miss | Your Boarding Pass Could Onboard Hackers | ChatGPT Passes CPA Exam on Second Try | Cyber Insurance Costs Rise in Health Care as Attacks Soar | Adobe Announces “Creator-Friendly” Generative AI Tools
Exclusively for PRO Members. Log in here or upgrade to PRO today.

Big and small websites are taking matters into their own hands because there are no clear legal or regulatory guidelines limiting AI’s usage of intellectual information.

Early in August, OpenAI unveiled its GPTBot crawler, claiming that the information obtained “might improve future models,” assuring that paywalled content will be omitted and providing instructions on how to block the crawler on websites.

Posted on October 17, 2023

Rick Richardson

About the Author

Rick Richardson, CPA.CITP, CGMA, is the CEO and founder of Richardson Media & Technologies and editor and publisher of Technology This Week, regularly featured at CPA Trendlines under special arrangement.

See more by Rick Richardson

Technology This Week provides an easy to read digest of the technology that tax and accounting practitioners need to know to better serve their clients. Professionals rely on Richardson to tell them in a timely and informative manner what’s new in technology and how it will impact their business and personal lives. Subscribers also have access to the entire library of prior issues with an easy-to-use search.

With all of the news sources available today, it's next to impossible to keep up with what's new in the world of technology. Technology This Week is a 20-minute read designed for an average reader rather than the technologist. Richardson had a 28-year career in technology with Ernst & Young, the last twelve years of which he served as National Director of Technology. He has been named to the "Technology 100" – the annual honors list of the 100 key achievers in technology in America. He has also been honored by the American Institute of CPAs with two lifetime achievement awards for his contributions to the profession in the field of technology. In 2012, Rick was inducted into the Accounting Hall of Fame by CPA Practice Advisor Magazine. He has also been named to the 100 most influential individuals in the accounting profession in America by Accounting Today magazine. He is a sought-after speaker around the world, providing his annual forecast of future technology trends to thousands of business executives, professionals, community leaders, educators, and students.

Click here for more by Rick Richardson

TO READ THE FULL ARTICLE

Continue reading your article with a CPA Trendlines Pro membership.