Bots Are Getting Good at Mimicking Engagement

When a company puts up a billboard or an ad on the bus, they don't care if the ad is seen by dashcams and dogs. All that matters is impact on the bottom line.

We at engineering decided to filter out bots. Figures fell dramatically by more than 50%.

In less that a day business mandated us to remove the filter.

Bots are real people after all

Why not use Microsoft Clarity?

> The bots are getting creepily good at mimicking engagement.

You would think Google Analytics would help a lot with this, but they seem to not care.

The word "revenue" above doesn't sound right. Did you mean "ad impressions", "clicks", "visits", or something else?

If the bot traffic is giving you 40% more revenue, well that's excellent news, right? If it's revenue, then keep doing it.

How does the advertising money get to them to make it worth their while to run bots at such a scale?

I mean sure, I suspect that people in the advertisers are doing it, but surely thats massively risky. there must be a grey market for this kind of transaction?

With that said, what do you think the error is here? Could your script have had some false positives? Enough to move the dial?

I was never really a punk-rebel kid, but a certain part of me rooted in the optimism of the early-internet kinda wants to see ad-models crash and burn.

Even advertising "working normally" always had a psychic odor of exploitation and deceit. Ex: "You absolutely need this product or else your peers will hate you."

> During my investigation, a source from the e-commerce data industry provided a crucial piece of the puzzle. He explained that his former company was responsible for scraping 70 million retailer web pages every single day. This is a legitimate and massive source of automated traffic.

> Why do they do this? For vital business intelligence. Major retailers like Amazon do not always notify vendors when they run out of stock. So, brands pay for data scraping services to monitor their own products. These "good bots" check inventory levels, see who is winning the "buy box," ensure product descriptions are correct, and track search result rankings. They even scrape from different locations and mobile device profiles to analyze what banner ads are being shown to different audiences.

I think a lot of the players involved in this would say those are bad bots. Having your competitors scrape your site for data would probably be something most website owners wouldn’t like, but getting data about THEIR competitors would be something they WOULD like.

All bot traffic is good from SOMEONE’S perspective, otherwise it wouldn’t be happening. Someone had to program the bot, someone has to be running it. They obviously think they are good bots.

The people running AI scraping bots think they are good bots. Many content creators think those are bad bots. Price comparison sites scraping retailers think their bots are good. The sites being scraped often think they are bad bots.

I just don’t think we can clearly separate good and bad bot traffic without specifying whose perspective we are talking about.

Bot traffic has been an issue for years. It has given rise to a new cottage industry of ad fraud detection services, none of which I’ve found particularly valuable. It always comes down to “so what do we do about it?” and no one seems to know how to get bots to stop viewing or clicking on paid media placements. Consumers use Google search and are on FB, IG, TikTok, LinkedIn, etc., and there aren’t really competing ad networks with “fewer bots”. So they keep buying fake traffic knowing that a significant chunk of it is invalid.

I don’t see anything changing unless big tech companies making billions in ad revenue have a large enough incentive to do so. At the moment they have plenty of incentives to keep things as they are.

“Half of the money I spend on advertising is wasted; the trouble is I don’t know which half.” - John Wanamaker

[1] https://jimalytics.com/industry/youre-buying-bots-an-inconve...

This is not likely to distort the basic numbers in your story, but it makes the premise questionable

If your script can correctly segregate bot traffic from human traffic, and website operators can conclude that bot traffic does not make purchases, then what -- you still don't want to blackhole that traffic right ?

I've nicked that sentence to use as the title above, since it's more neutral and representative of what the article is about.

The numbers that Google Ads shows as invalid clicks is dramatically different from what our platform detects.

In our cases, most fraudulent clicks come from mobile devices and mobile networks. Also, bots don't load page resources in full, making them easier to detect.

If your organization is struggling with fraud click detection, don't hesitate to reach out by email. We are ready to help.

1. https://github.com/TirrenoTechnologies/tirreno

In Google Ads you can just turn off the option to run your ads on non-Google sites; I think it's called their Display Network. Just run your campaign only on Google's search pages.

I'm surprised the article doesn't mention this rather common solution.

I'm none of those things, 10x or otherwise, and would be ashamed if I were. I do run a personal website, though, and most of the traffic is bots that ignore robots.txt. I wish those developers, founders, and marketers would stop doing that to me.

Too often I have been working on ecommerce gigs where the traffic comes from internal 'SEO' tools. Normally the person in charge is marketing, not technical, so it has always been difficult to get past identifying the problem to fixing it.

Often ecommerce companies are very siloed, so one person in one part of the 'team' is scraping the site with their special tools, only for another person to be doing another scrape with their special tools. You can have the guy doing the newsletter doing his thing, the guy doing organic search doing their thing, the guy doing paid ads doing their thing and someone in sales doing their own thing.

The sad thing is that they are typically just a few SQL joins away from exactly the data they want in a format they can digest. However, due to silo-ing, it can be hard to have that conversation.

On top of that, you do get new bots that need to be dealt with. The Huawei bot will scrape everything yet the store might not be delivering to China. So there are legit bots not doing ad fraud that need to be dealt with.

What I also find interesting is that nobody is interested in the server logs. They come for free, and, although not having CDN cache hits, they still record checkout transactions and any pages that don't have a freshly cached page.

Ad fraud also goes on in companies. I worked for a very successful company once and we only measured sales and what was out of stock. We didn't need open rates for emails or click through rates, our main problem was selling too much, which was a nice problem to have. Note that if you sell too much then you aren't going to get it all out the door in a timely fashion, or you run out of big lorries to put the orders in.

Since then I have not worked on a site that is as successful. Instead we have people getting praise for 'false metrics'. Anything an SEO person creates or a marketeer measures will always have some nonsense aspect to it. Or the accounting is mixed with brick and mortar sales even though free shipping and discounts have been given on each sale, with adwords used to get people through the 'door'.

Since sales manager has to report to someone on the board, if the sales numbers aren't good, the nonsense stats can be used to obfuscate the facts. The board only ever care about profit, so I don't like the way this goes down with false metrics of nonsense.

Another fundamental problem in ecommerce is a lack of basic salesmanship. If you work the shop floor doing specialist sales where you have to listen to the customer's needs, then you gain experience in the art of sales. You haven't got to be good at it, in fact it can be better to know your limitations, for me that is big ticket items where I don't have the product knowledge, however, I could always hand those sales over to a much more capable colleague.

You don't win every sale, but, in retail, you can have some really good streaks where no customer leaves empty handed. Your conversion rate is going to be more like 90% in face to face sales if you have the right product at the right price, with customers that don't buy coming back the following day or week to splash the cash.

In High Street retail there is no way you give customers 15%+ off just for stepping through the door. Yet this is table stakes in ecommerce, particularly for small to medium size shops. This instantly devalues the product.

Often there will be chatbots for whatever reason, and I am sure the likes of Dell can get that right, but your typical small ecommerce site will fluff this up too, so any customer daring to use the chatbot will not get instant help from a sales person.

So what to do?

It depends on your product, however, the goal is to get customers for life, not to churn through them. To achieve this it comes down to product, price, availability, shipping times, customer service and incentives for the customer to advertise for you, with reviews, word of mouth and all that hard stuff that needs real human skill. Sometimes it will only be a one-off sale, for example, if someone is buying a mattress. But, even then, the customer service basics matter.

What is also silly is how, with everything you buy, you will get adverts and incentives to buy what you have just purchased. I don't know why this is not considered career limiting, but nobody seems to have fixed this.

Next time, I will find out what data everyone needs and have my own script to collect just that data, then have a live backup that can be used for all internal purposes such as report generation. There will be no mystery script containers downloading 150 scripts with cookies, outside of developer supervision. My hunch is that the only numbers that really matter are sales.

As for why the sales aren't happening, so long as functionality is as it should be on all devices, then you have to dig deeper, to the knowledge level that only someone that has worked the showroom floor understands. Really, a website should be the sales person's knowledge condensed into HTML. Yet nobody asks the guy serving customers what they upsell or what product they recommend if a customer baulks at a given product. Instead we have mystery-meat AI scripts that manage these things.

Thanks for the heads up on ad fraud, normally I am a long way from that due to internal scraping efforts being far more damaging to website performance, and, as a developer, page load times matter to me far more than how much was paid for the traffic.

  One rep I had known for years finally admitted the truth off the record. "Dude, we know," he said. "Everyone knows. But if we filtered it all out properly, our revenue would drop 40% overnight, and investors would have a meltdown."

The irony of complaining about bot traffic with bot copy is too funny.

> It was not a sophisticated piece of software, just a tool designed to observe how "users" actually interacted with the website. I was not just counting clicks; I was watching behavior

I'm working right now on an inventory management system for a clinic which would really benefit from pulling the prices and availability from a very specialised online shop. I wish I could just get a large, fully cached status of all items in a json/CSV/whatever format. But they're extremely not interested, so I'm scraping the html from 50 separate categories instead. They'll get a few daily bot hits and neither of us will be happy about it.

If people are scraping data that you're not selling, they're not going to stop - just make it trivially accessible instead in a way that doesn't waste resources and destroy metrics.

The second is best "blocked" by doing nothing at all and instead silently dropping them from analytics.

Lawsuit time?