A bot, or to be more accurate, an “internet bot”, is a software solution or programming script that is programmed to perform automated processes or tasks.
These tasks are typically pretty simple but repetitive, although there are bots that perform relatively advanced tasks. The key is, bots can execute these processes at a much faster rate than humans ever could.
So, these bots aren’t inherently malicious by nature. There are bots owned and operated by legitimate companies (i.e. Google and Facebook) that are actually beneficial for most websites.
There are, however, bots owned and operated by hackers and cybercriminals that are deployed with malicious purposes: account takeover (credential stuffing and brute force), web scraping, and DDoS attacks, among other cybersecurity threats.
Around 40% of all internet traffic today comes from bots, and even good bots can disrupt your site’s performance when not managed well. This is why now so many companies and even individuals are looking for effective anti bot strategies to manage bot traffic coming to their websites, apps, and servers.
Why We Need an Anti Bot Strategy
Both good bots and bad bots can negatively affect your business in several different ways:
-
Bot traffic can skew your site’s analytics
Unauthorized bot traffic can skew analytics metrics for your site: dwell time, bounce rate, page views, and even conversion rates. If your business heavily relies on monitoring your site’s analytics (i.e. eCommerce), then being flooded with unauthorized bot activities can negatively impact your business. For instance, it’s going to be very difficult to improve your site via A/B testing since your metrics will be skewed.
-
Bot traffic can ruin your site’s performance
All bots, even good bots will eat your server’s resources, which are actually intended for human users. In turn, this will affect user experience, and according to Google, more than 50% of people will leave a website that loads in more than 3 seconds for them. Worse still, many people won’t come back to a site that has performed badly for them.
Malicious bots can also launch DDoS attacks, directing a massive volume of traffic at the website to deliberately overload the server. DDoS attacks will significantly slow down the website or render it completely unavailable for its legitimate users.
-
Bots can steal and use your valuable data
Cybercriminals can use bots to perform web scraping and data scraping attacks on your website.
While web scraping isn’t entirely illegal and can actually be beneficial, bots can, for example, re-publish your content on other websites, creating a duplicated content issue and negatively affect your site’s SEO performance.
Bots can also attempt to gain access to legitimate user accounts (account takeover attacks), for example via credential stuffing, and steal data within the account or use the account to launch various types of attacks like phishing and spam.
-
Bot attacks can directly and indirectly affect your business
Successful DDoS attacks and data breachers, for example, can cause long-term or even permanent damage to your company’s reputation besides financially crippling your business.
For sites that serve advertising, bots can perform click fraud attacks by clicking on the ads, and while this might initially boost your site’s ad revenue, the advertising network (Google Ads) can suspect your site from performing the click fraud and may penalize or even ban your site.
Also, eCommerce sites can be targeted by hoarding/scalping bots that will automatically put a high amount of merchandise to the shopping cart, and thus making the product unavailable for actual shoppers. For businesses that are very price-sensitive (i.e. ticketing, hotel reservation), bots can also spy on your price information and leak it to your competitors so they can undercut you, eliminating your competitive advantage.
Different Approaches to Anti-bot Mitigation
Due to all these negative effects that can be caused by bot traffic, it might seem like blocking the bot activities once they are identified are always the best approach. However, blocking is not always the best option, especially for persistent attackers.
When a bot operated by persistent hackers is blocked, they will simply modify it to bypass your security measures, so the bot will return stronger to haunt you.
Instead, we should focus on managing the bot activities, and here are some anti bot mitigation strategies to consider:
Rate Limiting
Since blocking a bot will give the bot operator a signal that the bot isn’t strong enough, then it’s typically to keep the bot for as long as possible on your site to waste its resources without letting it fulfill its objectives.
Rate limiting, or throttling is essentially about limiting the bandwidth served to these bots, so they’ll be much slower in executing its operations. The hope is that by slowing them down enough, the operator will give up and move on from your site.
Feeding Fake Data
Another common approach is to keep the bot active by feeding it with fake/thin content to poison its database. We can, for example, redirect the bot to a similar page or app where content is modified so it won’t be able to access your original content.
Again, bots run on resources that can be very expensive for the operator. So, by letting it waste its resources to steal fake, worthless data, hopefully the attacker will move to other targets.
CAPTCHA
This is the common approach when we are not 100% whether the user agent is a bot or a legitimate user. However, today’s sophisticated bots are really good at solving CAPTCHA challenges, and if we make the CAPTCHAs more difficult, it will hurt user experience.
Not to mention, the presence of CAPTCHA farms have rendered most CAPTCHA challenges useless as an anti bot solution.
However, CAPTCHAs solutions can be quite effective in certain situations to defend against less sophisticated bots.
Closing Thoughts: Blocking as The Most Effective Anti Bot Approach
Blocking can still be the most effective anti-bot approach provided we have a powerful enough anti-bot detection solution that can effectively and consistently differentiate between good bots and bad bots. By blocking the traffic in real-time, we won’t use any resources to reply to the bot’s requests, and we also don’t need to worry about various mitigation and filtering techniques.
It’s crucial to use an advanced, AI-based anti bot detection solution to leverage behavioral analysis to ensure blocking stays effective in protecting your website and system.
