Digital advertising depends on one fundamental assumption:

There is a real person on the other side of the impression or click.

When that assumption breaks, the entire advertising ecosystem starts to lose value.

Advertisers pay for impressions and clicks that may never have been seen or intentionally generated by a human. Publishers may receive revenue for traffic that advertisers don’t value. Ad platforms have to process enormous volumes of requests that provide little or no legitimate value. And analytics teams end up making decisions based on traffic that was never human in the first place.

This is the world of bot detection, invalid traffic (IVT), and ad fraud.

The good news is that not all invalid traffic requires sophisticated machine learning or expensive fraud-detection platforms.

A meaningful portion can be identified with relatively simple techniques using:

• Web server logs
• IP addresses
• User-Agent strings
• Request frequency
• HTTP status codes
• Referrers
• Known bot lists
• Data-center IP intelligence
• Cookies or session identifiers
• Basic behavioral rules

The harder problem is detecting traffic that looks human.

That is where sophisticated invalid traffic — SIVT — comes into play.

---

1. What is a bot?

A bot is software that performs actions automatically rather than being operated directly by a human for every action.

Bots are not inherently malicious.

Search engines use bots to crawl websites.

Monitoring systems use bots to check whether websites are working.

Security scanners use bots to identify vulnerabilities.

Content aggregators use bots to collect information.

These are legitimate uses of automation.

The problem begins when automated activity is presented as human activity in an environment where human presence has economic value.

For example:

Human:

User → Browser → Website → Ad → Impression

Bot:

Bot → HTTP Request → Website → Ad → Impression

If the advertising system cannot distinguish between these two, the bot may generate an impression that looks like a legitimate advertising opportunity.

And if the bot subsequently generates clicks:

Bot
 ↓
Page request
 ↓
Ad loaded
 ↓
Ad impression
 ↓
Click
 ↓
Landing page

the economic impact becomes even larger.

This is one of the foundations of ad fraud.

---

2. What is ad fraud?

Ad fraud is the deliberate manipulation of digital advertising systems to generate economic benefit from activity that does not represent genuine advertising engagement.

At a simple level:

Ad fraud = creating advertising activity without creating equivalent human value.

This can involve:

• Fake impressions
• Fake clicks
• Automated browsing
• Click farms
• Botnets
• Hidden advertisements
• Forced redirects
• Ad stacking
• Domain spoofing
• Cookie stuffing
• Fake conversions
• Automated app activity
• Malware-generated advertising activity

Not every instance of invalid traffic is necessarily malicious.

For example, a search-engine crawler can generate an ad request even though it isn’t trying to steal money.

Similarly, a user can accidentally click an advertisement.

From an advertising measurement perspective, however, these interactions may still be invalid traffic.

Google defines invalid traffic as clicks and impressions that aren’t the result of genuine user interest and includes accidental clicks, manual fraudulent clicks, automated tools, crawlers, known data-center traffic and other irregular activity.

---

3. Why does bot traffic cost us money?

The easiest way to understand the economics is to follow the money.

Suppose an advertiser spends:

$1,000,000

on digital advertising.

Imagine that 10% of the measured traffic is invalid.

That doesn’t necessarily mean the advertiser literally loses exactly $100,000 in cash — platforms may filter or credit some invalid activity — but the invalid traffic can still create substantial economic damage.

The impact can happen at multiple levels.

Direct media waste

Money is spent reaching traffic that isn’t genuinely interested.

Lost opportunity

A fraudulent impression consumes inventory that could have been delivered to a real user.

Analytics pollution

Bots can distort:

• CTR
• Conversion rate
• Bounce rate
• Engagement
• Session duration
• Attribution
• Audience segments

Algorithmic pollution

This is increasingly important.

Modern advertising systems use machine learning to optimize campaigns.

If a campaign receives large amounts of artificial traffic, the optimization system may learn from the wrong signals.

For example:

Real users:
100,000 impressions
2,000 clicks
200 conversions

Bots:
100,000 impressions
10,000 clicks
0 conversions

The advertising platform doesn’t simply see “bots.”

It sees unusual behavioral data.

If enough fraudulent activity gets through, the resulting signals can influence campaign optimization, audience targeting and attribution.

---

4. How big is the problem?

There is no single universally accepted number for “the cost of bot traffic.”

Different studies measure different things:

• Ad fraud
• Invalid clicks
• Invalid impressions
• Bot traffic
• Programmatic advertising
• Paid search
• Display advertising
• Mobile advertising
• CTV
• Affiliate traffic

So it is better to treat industry estimates as ranges rather than a single definitive loss figure.

For example, Juniper Research has been cited as estimating that more than $84 billion of global digital advertising spend was lost to ad fraud in 2023.

Another 2025 study from Spider AF estimated more than $37.7 billion in digital ad-fraud losses during 2024.

IAS’s 2025 Media Quality Report cited an estimated $100 billion of ad spend siphoned away by fraud in 2024.

The difference between these numbers is important.

It does not mean one company is correct and another is wrong.

It demonstrates that the answer depends heavily on:

1. What traffic is measured
2. Which advertising channels are included
3. What constitutes fraud
4. Which countries are included
5. Whether the estimate represents gross fraud or measurable economic loss

The important conclusion is simpler:

At global advertising scale, even a small percentage of invalid traffic represents billions of dollars.

And at an individual advertiser or publisher level, the calculation is much easier.

If your annual advertising spend is:

$10 million

and you determine that:

5% = demonstrably invalid

then the potential exposure is approximately:

$500,000

before accounting for platform filtering, credits, indirect effects and uncertainty.

That is why measuring your own traffic is more useful than relying exclusively on a global headline number.

---

5. Not all bots are the same

This is where the industry distinction between GIVT and SIVT becomes important.

The Media Rating Council (MRC) divides invalid traffic into two broad categories:

GIVT

General Invalid Traffic

Traffic that can generally be identified using routine methods such as lists and standardized parameter checks.

SIVT

Sophisticated Invalid Traffic

Traffic that is significantly harder to identify and requires advanced analytics, multiple corroborating signals, coordination of data sources and sometimes human investigation.

Google similarly describes GIVT as traffic identifiable through known robot/spider lists and routine checks, while SIVT requires deeper analysis.

This distinction is extremely useful because it tells us something practical:

You don’t need an advanced bot-detection platform to start detecting bots.

---

6. GIVT: the easy bots

GIVT is the traffic where relatively obvious signals tell us that something isn’t normal.

Examples include:

• Known crawlers
• Known spiders
• Known bots
• Known data-center traffic associated with invalid activity
• Non-browser User-Agent headers
• Invalid or obviously automated requests
• Certain repeated activity patterns
• Pre-fetch or pre-render traffic
• Known irregular request patterns

Google’s documentation lists data-center traffic, known crawlers and certain irregular patterns among GIVT examples.

This is the first place where server logs become extremely powerful.

---

7. Your server logs are already a bot-detection dataset

Imagine your web server receives requests like:

timestamp
IP
method
URL
status_code
user_agent
referer
response_size

You might have:

12:01:01  1.2.3.4  GET /sports  200  Chrome
12:01:03  1.2.3.4  GET /news    200  Chrome
12:01:04  1.2.3.4  GET /article 200  Chrome

At first glance this looks like a normal visitor.

Now imagine:

12:01:01  5.6.7.8  GET /page1  200  Python-requests
12:01:01  5.6.7.8  GET /page2  200  Python-requests
12:01:01  5.6.7.8  GET /page3  200  Python-requests
12:01:01  5.6.7.8  GET /page4  200  Python-requests
...

We already have a strong signal.

The traffic is declaring itself as automated.

No machine learning is required.

---

8. The simplest bot detector

A basic bot detector can start with a set of rules.

For example:

def detect_bot(request):

    if user_agent_is_known_bot(request.user_agent):
        return "GIVT_KNOWN_BOT"

    if ip_is_known_data_center(request.ip):
        return "GIVT_DATACENTER"

    if is_non_browser_user_agent(request.user_agent):
        return "GIVT_NON_BROWSER"

    if request_rate_too_high(request.ip):
        return "GIVT_IRREGULAR_RATE"

    return "UNKNOWN"

This is intentionally simple.

The goal isn’t to build the world’s best bot detector.

The goal is to identify obvious invalid traffic with explainable rules.

---

9. Signal #1 — User-Agent

The User-Agent is one of the easiest signals to inspect.

Examples of obvious automation:

Googlebot
bingbot
curl
wget
python-requests
Go-http-client
HeadlessChrome
Selenium
Playwright
Puppeteer

A simple rule could be:

BOT_PATTERNS = [
    "bot",
    "crawler",
    "spider",
    "curl",
    "wget",
    "python",
    "scrapy",
    "selenium",
    "headless",
]

Then:

ua = user_agent.lower()

if any(pattern in ua for pattern in BOT_PATTERNS):
    return True

This will catch some bots.

But there is a major problem.

User-Agent strings are not trustworthy.

A sophisticated bot can simply pretend to be:

Mozilla/5.0 ...
Chrome/151...

and now the User-Agent looks like a real browser.

So User-Agent detection is useful for GIVT, but it cannot solve SIVT.

---

10. Signal #2 — Data-center IP addresses

Another useful signal is IP intelligence.

A normal residential user might originate from:

Residential ISP
Mobile carrier
Cable provider
Broadband provider

whereas automated infrastructure may originate from:

AWS
Google Cloud
Azure
DigitalOcean
Oracle Cloud
Akamai
Cloudflare
Other hosting providers

A database containing ASN/IP ownership information can therefore be useful.

For example:

IP
 ↓
ASN
 ↓
Organization
 ↓
Network type
 ↓
Datacenter / ISP / Residential / Mobile

You can then create rules such as:

Known invalid datacenter IP → GIVT

But there is an important caveat.

A data-center IP does not automatically mean a bot.

VPNs, corporate networks, cloud browsers, privacy systems and legitimate automated services can all create traffic from infrastructure that isn’t a residential ISP.

Google explicitly cautions that multiple clicks from one IP don’t automatically indicate invalid traffic because shared and rotating IPs can represent legitimate users.

Therefore:

Datacenter = signal, not proof.

---

11. Signal #3 — Request rate

Human browsing has physical limitations.

A human cannot realistically open:

500 pages

in:

30 seconds

while generating normal browser behavior.

A bot can.

So calculate requests per IP:

SELECT
    ip,
    COUNT(*) AS requests
FROM server_logs
WHERE timestamp >= NOW() - INTERVAL '1' MINUTE
GROUP BY ip
ORDER BY requests DESC;

You may discover:

IP             Requests/min

1.1.1.1        12
2.2.2.2        18
3.3.3.3        22
4.4.4.4        847
5.5.5.5        1,293

The last two deserve investigation.

But again:

High request rate ≠ automatically fraud.

Large corporate NATs, crawlers, APIs, preloaders and legitimate applications can generate high volumes.

Use rate as a signal.

---

28. A practical layered bot-detection architecture

For many organizations, a good architecture looks like this.

Layer 1 — Server-side GIVT

Use:

• Known bot lists
• User-Agent
• IP reputation
• ASN
• Data-center intelligence
• Request rate
• HTTP behavior
• Session patterns

This layer is:

Cheap + explainable + scalable

---

Layer 2 — Client-side detection

Collect:

• Browser characteristics
• Device characteristics
• JavaScript behavior
• WebGL
• Touch capabilities
• Screen properties
• Browser inconsistencies

This layer catches traffic that looks legitimate at the HTTP level.

---

Layer 3 — Behavioral analysis

Analyze:

• Mouse behavior
• Scroll behavior
• Click timing
• Session navigation
• Focus/visibility
• Ad interactions
• Page engagement

This helps identify automation that successfully mimics browsers.

---

Layer 4 — Network intelligence

Correlate:

• IP reputation
• ASN
• Proxy
• VPN
• Hosting provider
• Residential proxy intelligence
• Historical activity

---

Layer 5 — Cross-event correlation

Look across:

IP
+
device
+
cookie
+
session
+
page
+
ad
+
click
+
campaign

This is where sophisticated fraud starts becoming detectable.

---

Layer 6 — Machine learning

At large scale, models can learn relationships between hundreds of signals.

Instead of:

if IP == X:
    fraud

you can have:

risk = f(
    network,
    device,
    browser,
    behavior,
    history,
    advertising,
    geography,
    timing
)