Technology

Blocklisting Revealed the Dark Side of Ad Blocking

By Trey Grainger, CTO of Presearch

Blocklisting is common on the internet, but not many people know what it is or how it works. A blocklist refers to a list of websites, portions of websites, or other materials to be blocked from view. The most popular use case online is ad blocking. More than 42 percent of internet users worldwide now use an ad blocker.

Most people who turn on ad blockers just assume the software seamlessly “protects” them from content they wouldn’t want to see online or from being unnecessarily tracked. In reality, most ad blockers primarily operate by checking blocklists that have been manually compiled over time, listing websites or portions of websites the person modifying the blocklist wants to censor from view.

As the Chief Technology Officer of a search engine, I’m pretty familiar with this technology. But it wasn’t until recently that I learned how few people control these increasingly omnipresent blocklists. Presearch, the decentralized, community-driven search engine with 2.3 million users that I help run, ran into major cascading issues caused by bad blocklisting the spring of 2021. 

A Blocklisting Headache 

Beginning on March 31, many Presearch users started reporting that they couldn’t connect to the site, or that when they connected, the site’s functionality was broken. People flooded our internal community channels and Reddit with their concerns, saying things like: “My searches just hang, what’s the deal? Reinstalled and tried multiple browsers. I’ve temporarily switched back to Google, sadly.” 

Obviously, this was a big problem, but it didn’t affect everyone, and we couldn’t initially find any reasonable source of the issue within the platform. After several days of reported problems from people using Brave, uBlock, Windscribe VPN, and similar services, we discovered that Presearch’s entire website was put on a blocklist. We scoured Reddit, Discord, and Github, downloading and combing through dozens of blocklists to find out which one added Presearch and why, and trying to reach out to each service and get the problem fixed.

We discovered that Presearch had been added to a blocklist maintained publicly by just four people. One of them had seen a referral banner on another unrelated website, and hastily added a blanket block for ALL requests to Presearch from across the web. This one person’s overzealous edit created a cascading effect, as other unrelated services also pulled in the public blocklist as part of their systems and also began blocking Presearch.

In the case of one prominent VPN, Presearch’s website was unreachable and appeared to have vanished from the internet. In the case of the ad blockers and privacy browsers like uBlock and Brave, the website would load, but search results were blocked, making users think our website was badly broken. We lost millions of searches and a week of dev time hunting down the issue and building emergency countermeasures. Our traffic and revenue took a big hit. I’m sure some users left and never came back, and that’s a punch to the brand reputation we’ve worked years to build. 

We reached out to one of the developers maintaining the blocklist to request help. To be removed, we were required to prove that we aren’t an ad network, and to show that the blocklist had broken our website since user’s couldn’t access it correctly. The developer fixed the issue by adding a more selective block that only applied to the referral banner they’d originally seen. We were warned not to work around it, or they’d apply a more severe filter. 

I reached out to another ad blocking service that was clearly related to the original list, but the developer in charge wouldn’t disclose any details. Although their process was a black box, they eventually helped us resolve the issue by adding Presearch to an “exception” list. 

Bringing Transparency to the Web 

Soon, Presearch was back up and running. But after a week spent navigating the blocklisting rabbit hole, I started to re-examine this ecosystem and how subjective it all is. Why are blocklists that censor content on the internet from hundreds of millions of users controlled by so few people? With these blocklists being integrated by default in many services now, are consumers even aware of what content is being censored from them? What gives a few people the right to decide what should and shouldn’t be accessible online, using any criteria they choose, with little to no due diligence? 

I don’t believe these developers are malicious. Blocklists serve a helpful purpose. However, I do believe that the ad blocking industry is overzealous in its efforts to “clean up” the web and decide what should and shouldn’t be viewed, breaking or hiding portions of many websites in the process. What if this practice were to be infiltrated by bad actors? It’s easy to imagine how things could quickly go downhill. Entire websites could start disappearing based on the whims of a small group of powerful blocklisters. 

One of the founding principles behind Presearch is a resistance to “control by a few over the many.” We are trying to build a decentralized search engine, where decisions around content and algorithms are open and made by the community, not a small group of companies or developers. 

My hope is that as people start to see what’s behind the curtain, we can begin to build a better future. A future where search engines become open and reveal their algorithms, developers share how they relegate ads, blocklists are community-powered, and we all benefit from a more open, transparent, and decentralized web. 

Trey Grainger

Trey Grainger

About the Author

Trey Grainger is the Chief Technology Officer of Presearch, a decentralized search engine. He’s also the founder of Searchkernel, an AI-powered search and advisory development firm. He lives in South Carolina. 

Sources: 

https://www.adexchanger.com/data-driven-thinking/ad-blocking-theft-or-fair-use/

https://www.marketingterms.com/dictionary/ad_blocking/

https://www.reddit.com/r/Presearch/comments/mhjtt0/is_presearch_down/

https://backlinko.com/ad-blockers-users

The views and opinions expressed herein are the views and opinions of the author and do not necessarily reflect those of Nasdaq, Inc.