Search is the one product where appearing at all reads as an endorsement. This page is about the four places a verified list of live phishing hostnames belongs in a search pipeline — the discovery gate, the crawl gate, index-time classification and the serve-time safety net — plus ad destination screening, and the honest limits of any known-bad list.
Excluding a hostname before you fetch it, after you have indexed it, and while a user is waiting are three genuinely different engineering problems.
The instinct is to treat phishing exclusion as a single feature — "filter the bad results" — and then argue about where it goes, but that framing hides the real design decision: each stage of a search pipeline has a different cost profile, a different latency budget and a different failure mode, so a domain check that is trivially cheap in the crawl scheduler is unacceptable in the query path, and a policy that works beautifully at index time does nothing about a page that was already served this morning.
A crawler spends most of its life deciding what to fetch next from a frontier of URLs harvested from links, sitemaps, feeds and submissions, and that frontier is where a known-bad hostname is least expensive to remove: dropping it costs one set membership test and saves a connection, a fetch, a parse, a store and every downstream unit of work that would have followed. Excluding at discovery also means the content never exists in your systems at all, which quietly simplifies questions about retention and about what you are hosting a copy of.
The frontier is not the only way a URL reaches a crawler: redirect chains, canonical declarations, hreflang alternates, embedded resources and re-crawl schedules all produce fetch targets that never passed the admission check. Re-testing the hostname immediately before the request is cheap insurance against a domain that was clean when it entered the frontier and became known-bad in the hours or days before the fetch actually happened — which, given how quickly this infrastructure appears, is not a rare case.
This is the gate that matters most for correctness, because your index contains pages fetched before the domain was known, pages on hosts that changed hands, and pages that were genuinely legitimate when crawled. The daily changelog of added domains makes it tractable: intersect the newly-added set with the distinct hosts in your index and you have a precise, small list of documents to re-evaluate, and quarantine — a flag that suppresses serving without destroying the record — is usually better than deletion because it is reversible and preserves the evidence.
Anything in the serving path competes for a latency budget measured in single-digit milliseconds, and a search engine cannot make a network call per result and remain a search engine — so if you check at query time, and there are good reasons to for freshness, it has to be a local in-memory test against a set you already loaded, applied to the small number of hosts in the result set you are about to return.
This is a known-bad lookup. A clean result means the hostname was not on the list at the moment you asked, not that the destination is safe. A domain registered and first used within the same hour will not be there yet, and no amount of pipeline placement changes that — so treat all four gates as one layer among several.
Four checks against the same in-memory structure, each one costing a hash lookup and each one covering what the previous one could not.
Discovered URLs are tested on the way into the crawl frontier. Known-bad hosts never queue, so no scheduler slot, connection or parse is ever spent on them.
Re-tested immediately before the request, which catches redirect targets, canonical hops and re-crawls of hosts that turned hostile after admission.
The daily added list is intersected with distinct hosts in the index. Matches are flagged and quarantined rather than deleted, so the action is reversible.
The final result set is tested in memory before it leaves the serving node. A handful of lookups per query, no network, no measurable latency.
Two implementation details make this cheaper than it first sounds, and the first is that all four gates read the same structure: you load the feed once per node, and the frontier scheduler, the fetcher, the indexer and the serving tier all consult it — no separate service, no fan-out, and no consistency problem beyond making sure every node has the same generation of the data loaded.
Lowercase, strip the trailing dot, resolve internationalised domain names to their punycode form, and decide explicitly how you treat subdomains — whether a match on login.example.tld implies anything about example.tld, and whether a match on a bare domain should suppress every host beneath it. Both answers are defensible; what is not defensible is different components answering differently, which is how a document gets excluded at crawl time and served at query time.
A jobs board crawling employer sites, a marketplace indexing seller pages, a documentation search ingesting customer content, a product-search engine pulling supplier catalogues — every one has a discovery step, a fetch, an index and a serve. The volumes are lower by orders of magnitude, which usually means the API is a perfectly good fit rather than the feed, but the placement logic is identical.
The technical work is the same either way. The choice is about what you owe a user who was about to click, and what you owe a site owner who might be wrong about being on a list.
Silent removal is the default most engineers reach for, and for confirmed credential-harvesting infrastructure it is usually right: the page has no legitimate audience, nobody is being denied information, and a warning interstitial for a site whose entire function is to steal a password mainly gives a determined user a way through — so if the list you are acting on is genuinely a verified inventory of live phishing hosts, each entry carrying an active A record checked through rotating proxy infrastructure, removing those results is closer to spam filtering than to editorial judgement.
The clearest case for a label rather than a removal is the compromised-but-legitimate host: a real business whose site was breached and is temporarily serving a phishing kit at one path. Removing the entire domain punishes the victim and takes down content people are genuinely looking for, whereas a plainly worded warning explaining that the site may currently be serving a fraudulent page gives the user what they need without erasing a whole site over one compromised directory.
If a user searches for the exact name of a brand and every result you would have shown is a look-alike, returning nothing is confusing rather than protective. A short annotation with the genuine destination promoted above it teaches the user something durable; silence teaches nothing and often sends them straight back to a different search box, where the same look-alike is waiting.
Site owners will contact you, some of them will be right, and the ones who are right will be furious and public about it. Keep the quarantine flag reversible, log which list generation triggered the action and on what date, and be able to answer "why was I excluded on the fourteenth?" with a specific record rather than a shrug — storing last_checked and the build date alongside your own action makes that a lookup rather than an investigation.
The confidence model is deliberately binary — 0.98 for a confirmed match and 0.0 for no match, with nothing in between — which is honest precisely because it refuses to imply a spectrum. Rendering the absence of a match as a positive safety signal converts a narrow factual statement into a guarantee you cannot support, and the first time it is wrong it will be wrong in public.
The asymmetry worth designing around: removing a phishing result costs a user nothing, because no one was looking for it. Removing a legitimate site costs its owner their traffic and costs you their trust. Build the pipeline so that the first action is automatic and the second is reversible, and every argument about thresholds becomes an argument about a reinstatement queue instead.
The numbers settle this argument quickly, and they settle it the same way for a web-scale crawler and for a busy vertical search product.
Start with the rate limit, because it is the ceiling everything else runs into: ten requests per second per key, with /batch carrying up to a hundred domains in a single POST, puts the practical ceiling around a thousand domain verdicts per second per key — comfortable for a site-search product checking a few thousand ingested URLs a day, and not in the right universe for a crawler making decisions about millions of frontier entries an hour, which no amount of key sharding fixes.
A search result page has a total budget most teams measure in tens of milliseconds across query understanding, retrieval, ranking and rendering. Sub-50ms is a good number for an HTTP API and a catastrophic number for a component that must run once per result inside that budget, so any design putting a network call between a user's query and their results will be removed the first time it has a bad afternoon.
The /feed endpoint returns the entire database as CSV with domain,category,dns_status or as JSON, with unlimited downloads on a subscription. A few hundred thousand hostnames is a small structure by search-infrastructure standards — a hash set on every node, or a Bloom filter in front of one for a flatter footprint — so membership tests cost a memory access and the check is cheap enough to run at all four gates instead of just the convenient one.
Refreshing is the only ongoing operation, and it is another dataset on a daily distribution schedule — a problem every crawler team has solved a dozen times. The build lands at 04:30 UTC with a diff of domains added and removed, so nodes apply a delta rather than reloading a full file, and the same artefact hands the indexer exactly the set of documents it needs to re-evaluate at no extra cost.
Pull once centrally, validate, distribute internally, and raise one alert if the loaded generation is older than forty-eight hours. Hit counts vary for uninteresting reasons; a silently stale set is the realistic failure mode, it degrades protection without producing any other symptom, and it is invisible unless you measure it directly.
Single /check calls remain the right tool for an abuse analyst confirming a report, for a manual review queue, and for the ad-review path in the next section, where volumes are low and a live answer beats a cached one. Keep both shapes rather than treating the feed as a replacement for everything.
The /stats endpoint reports the current database size and last update time and requires no API key, no credits and no authentication at all, which makes it a convenient independent check for a monitoring job comparing against the generation your own nodes believe they have loaded.
An approved ad is a standing promise about a destination you checked once. Destinations do not stay checked.
/check and sub-50ms, this fits inside an automated review step without a human ever waiting on it.is_phishing, category, confidence and last_checked. This is what makes step four a query rather than a project.Step four costs almost nothing because it runs against data you already hold and a diff you already download. It is also the step most ad platforms skip, which is why "the ad was clean when we approved it" remains such a common and unsatisfying incident summary.
Paid placements deserve stricter handling than organic results for a reason that has nothing to do with security engineering: you took money for them. An organic result is your ranking system's opinion, and users broadly understand that an opinion can be wrong, whereas a paid placement is a commercial transaction in which you accepted payment to put a destination in front of a user above the results you organically believe in — so when that destination harvests credentials, the position is materially worse for the user, for the advertiser whose brand was imitated, and for you.
Advertisers legitimately use click trackers, affiliate networks, agency-managed redirect layers and measurement platforms, so the hostname in the creative is routinely not the hostname the user lands on, and any screening that only looks at the visible URL is screening the least interesting hop. Follow the chain as far as your policy allows, screen each hostname you observe, and treat an advertiser who refuses to declare their final destination as a risk signal rather than a technical inconvenience.
Ads approved months ago, running at low budget in narrow geographies, are invisible to everyone until something goes wrong, and a daily intersection against your stored destinations catches those without anyone having to remember they exist. Teams building this alongside broader ad-quality tooling will find the patterns on our adtech and advertising page apply directly, and redirect chains are covered under URL shorteners.
Three failure modes that are worth understanding precisely, because each one changes what you should expect the list to do for you.
Cloaking is the oldest of the three and the most directly aimed at you: the page serves one thing to a crawler identifying itself as a search engine and something else to a browser arriving from a result, so that it is indexed on the strength of content the operator never shows a human — which is devastating against a content-based classifier, because the classifier is reading exactly the content prepared for it, and largely irrelevant against a hostname list, because the domain is what it is regardless of what the server chooses to render.
Because the unit is the hostname, this data says nothing about a specific path: a large legitimate site with one compromised directory is not on the list as a whole, and should not be. If your threat model is dominated by path-level compromise on otherwise-good domains, a hostname list is the wrong primary tool and you will need content-level signals working alongside it.
Phishing pages hosted as subdomains or user paths on trusted platforms — site builders, notebook and document services, form builders, cloud storage with public sharing, code-hosting pages, and the various "publish your app" platforms — inherit ranking signals that took the parent platform years to earn, so a ranking system treating domain authority as a proxy for trustworthiness will actively promote them.
When the phishing host is a distinct subdomain that resolves and serves the kit, it can be verified and listed as that hostname without touching the parent platform, which is exactly the granularity you want. When the deception lives at a path on a shared hostname also serving millions of legitimate pages, no hostname list can help and nobody should pretend otherwise — that case needs platform abuse reporting, path-level signals and pressure on the host, as described on our hosting providers page.
Search engines are frequently the first system in the world to encounter a brand-new domain, sometimes within minutes of it going live, because discovery is what they do — and a verified list is by construction a record of what has already been observed, so there is a window in which a domain is live, harvesting credentials, discoverable by your crawler and absent from every list including this one. The twenty-four-hour rebuild narrows it; your freshness heuristics and reporting channels cover the rest.
Against reused, mass-deployed infrastructure; against cloaking, because the verdict does not depend on what the server showed your crawler; and at the gates where you need an answer in microseconds rather than a classification pipeline. Also against the same host reappearing in a different vertical or a different language market.
Against path-level compromise on a legitimate domain; against deception living on a shared hostname you cannot suppress wholesale; and in the first hours of a brand-new domain's life. These are real gaps, they are shared by every known-bad list, and the correct response is to run this as one layer rather than as the answer.
Including several where the useful answer is a limitation rather than a capability.
No, and vertical and site-search products are probably the better fit. A jobs board crawling employer career pages, a marketplace indexing seller storefronts, a comparison engine pulling supplier catalogues, or a documentation search over customer-hosted content all have the same four gates at a fraction of the volume. At those volumes the API is usually the right tool: /batch takes up to a hundred domains per request at one lookup each, which covers a daily ingest of a few thousand URLs for very little. The threshold for switching to the feed is not a headcount or a company size, it is whether your check volume makes per-lookup pricing dominate, or whether you need the test in a latency-sensitive path. Either of those is a reason to hold the list locally.
For a hostname that exists solely to harvest credentials, remove it. There is no legitimate audience being denied anything, and an interstitial mainly gives a determined user a route through. For a legitimate site that appears to be temporarily compromised, label it — removing an entire domain because one directory is serving a kit punishes the victim and hides content people genuinely want. The engineering is the same either way; the difference is policy plus a reinstatement path. Whichever you choose, keep the action reversible and log which build of the list triggered it, so that a site owner's appeal is answered with a date and a record rather than a guess.
Where the phishing lives on its own resolvable subdomain, that specific hostname is what gets verified and listed — the parent platform is untouched, which is exactly the granularity you want. Your pipeline suppresses one host, not a service millions of people use legitimately. Where the deception lives at a path on a shared hostname, this data cannot help and will not claim to. That case needs path-level signals of your own, the platform's abuse reporting channel, and the kind of pressure described on our hosting providers page. It is worth deciding in advance which of the two situations your suppression logic is actually looking at, because conflating them produces either over-blocking or a false sense of coverage.
Much less, and that is one of the better reasons to have hostname-level intelligence alongside whatever content analysis you run. The verdict attaches to the domain rather than to a rendering of a page, so it does not matter what the server chose to show your crawler. If the host was observed and verified serving a credential-harvesting page to an ordinary client, it is on the list. The corresponding limitation is that the unit is the hostname, so this says nothing about a specific path on a domain that is otherwise fine. Use it for what it is good at and keep content signals for path-level problems.
The database is rebuilt every twenty-four hours with the daily build landing at 04:30 UTC, and every entry is DNS-verified through rotating proxy infrastructure so only hosts with an active A record are included. Domains that stop resolving fall out rather than accumulating, which is why the live count stays workable instead of growing without bound. You can confirm this independently without any commitment: the /stats endpoint reports the current database size and the last update time, and it requires no API key, no credits and no authentication. Point a monitoring job at it and compare against the generation your own nodes have loaded — that comparison is a better staleness alarm than anything internal to your pipeline.
It makes crawl-time filtering incomplete, which is different. A domain registered and first used within the same hour will not be there, and search engines are unusually likely to meet exactly those domains because discovery is their job. That gap is real and no known-bad list closes it. What the gap does not do is remove the value of the rest. A large share of phishing infrastructure is reused across many targets and many markets before anybody reports it, which means by the time it reaches your crawler it has frequently already been observed elsewhere. Excluding that portion at the frontier saves real crawl budget and keeps confirmed hostile hosts out of the index entirely, which leaves your freshness heuristics and reporting channels to handle a much smaller residue.
Yes, and it is one of the cheapest wins available if you stored the destination hostname with the ad record. Each day's changelog gives you the domains newly added to the database. Intersect that set with the distinct destinations of your live ads and you get a short list of campaigns to pause. It runs against your own database and a small diff, so it costs no credits at all beyond the feed you already have. If you did not store destinations, that is the first thing to fix — the re-screen is trivial once the data exists and impossible until it does. The adtech and advertising page covers the wider ad-quality pipeline this sits in.
If you are holding the list locally — which any crawler at meaningful volume should be — the Daily Threat Feed is $499 per month. The annual option saves $1,989 and adds historical archive access, priority support, custom format options, a dedicated account manager and 100,000 API credits, which conveniently covers the ad-review and analyst-lookup workloads without a second purchase. Feed downloads are unlimited, so the cost does not move with your node count or your crawl rate. If you are a vertical or site-search product with bounded ingest, monthly plans are the simpler answer: Growth at $99/month for 25,000 lookups, Growth at $99/month for 25,000 lookups , Professional at $249/month for 100,000 lookups, and down to per lookup on the Enterprise plan at the largest package. Credits are monthly subscriptions via PayPal and billed monthly. Larger arrangements — SFTP or S3 delivery, STIX/TAXII, a custom update frequency, an SLA or on-premise deployment — are quoted rather than listed. The full table is on the pricing page.
One dataset loaded per node, consulted at four gates, refreshed daily from a changelog that doubles as your index re-evaluation work list. Ad destinations screened at submission and again every morning.
