Phishing domains turn up in DNS answers, proxy sessions, mail gateway verdicts and endpoint process telemetry. Turning them into detections does not require ingesting anything new — it requires a reference dataset sitting next to the data you already have, refreshed daily and diffed so the reload stays cheap.
Every phishing interaction leaves a trace in something your SIEM is receiving. The reason those traces are not detections is that nothing in the pipeline knows the hostname is hostile.
Trace a single successful phishing click through a normal log estate and count the systems that saw it. The recursive resolver logged an A query and returned an answer. The proxy or secure web gateway logged an outbound session with a hostname, a client address, a user agent and a byte count. The mail gateway logged a delivery verdict on the message that carried the link, probably with the URL preserved in a header or a rewritten click-tracking record. The endpoint agent logged a browser process making a network connection. Four independent witnesses, all retained, all searchable, and not one of them raised anything — because to every one of those systems the hostname was simply a string it had not seen before.
Once a list of verified-live phishing hostnames exists as a queryable object inside the platform, every one of those four log sources becomes a detection surface without any change to collection, parsing or licensing. The correlation search does the work at search time, and the data was always there. That is the entire gap this integration closes, and being precise about its shape is what keeps the project small.
Expensive because most SIEM licensing is a function of daily volume. Wrong because indicators are not events: an indicator is a fact that is true until it stops being true, whereas an event is something that happened at a timestamp. Modelling the first as the second produces an index full of records with no meaningful time dimension, a search that must deduplicate before doing anything useful, and a bill that grows with somebody else's database rather than with your activity.
Splunk calls it a lookup. Microsoft Sentinel calls it a watchlist. Elasticsearch calls it an enrich policy, backed by an index and applied through an ingest pipeline. QRadar calls it a reference set. The vocabulary and the operational details differ, but the concept is identical — a compact reference collection held beside the event data, refreshed on its own schedule, and joined at query time or ingest time depending on which end of the trade-off you want.
An authenticated download, a file landing somewhere your platform can reach, a load into the native reference object, and correlation searches that reference it. That is the whole chain, set out below. The parts that need genuine engineering thought are the refresh mechanics and the alert semantics — not the plumbing, which is a scheduled job and a file.
Local reference data, per-event API calls and full ingestion have very different cost, latency and freshness profiles. Most estates end up with the first plus a little of the second.
| Decision factor | Local lookup / reference set | Per-event API call | Ingest the feed as events |
|---|---|---|---|
| Effect on licensed daily volume | None — reference data is not indexed events | None, but adds outbound calls | Adds the whole database, every day |
| Search-time latency | Local join, no network hop | Sub-50ms per call, but per call | Local, though the index is large |
| Freshness ceiling | As of the last pull — up to 24 hours | Current at the moment of the call | Same as the local lookup |
| Behaviour when the internet path is down | Keeps working on yesterday's copy | Enrichment fails or the pipeline stalls | Keeps working |
| Rate-limit exposure | One download per day | Ten requests per second per key is a real ceiling | One download per day |
| Suits retro-hunting over historical logs | Yes — join the new list against old events | No, unless you replay every historical domain | Yes, but you are storing snapshots |
| Where it genuinely earns its place | The default for all bulk correlation | Analyst-initiated lookups and SOAR enrichment steps | Almost nowhere; avoid unless a platform forces it |
The practical answer for most estates: a local reference object carrying every domain, plus an on-demand call from the investigation workflow for the hostnames that are not in it.
The middle column deserves a note, because per-event API calls are frequently proposed and rarely survive contact with volume. A mid-sized organisation resolves a very large number of distinct hostnames per day, and even if you cache aggressively and only ask about names you have not seen, the residual query rate will exceed ten requests per second during business hours. The rate limit is not a pricing lever to be negotiated around; it is a property of the service, and building a pipeline that depends on exceeding it produces a fragile detection layer that stalls exactly when traffic is heaviest.
An analyst pivots on a hostname from an alert and wants a current verdict. A SOAR playbook enriches a case with the response fields before deciding whether to auto-contain. A detection engineer validates a candidate rule against a handful of names. All of those are single-digit call counts with a human or a case behind them, and sub-50ms responses make them feel instantaneous inside a console rather than like a lookup you waited for.
A lookup file is portable across your whole detection estate. The same file backing a Splunk lookup can back an enrich index in a parallel Elastic deployment, feed a resolver blocklist, and populate the reference set on an appliance somebody inherited from an acquisition. Once the daily pull exists as a job, adding a consumer is a configuration change — whereas per-event API calls couple every consumer directly to one dependency.
Nothing here needs a custom app or a bespoke connector. Each platform already has the primitive; the work is choosing the right one and setting the refresh correctly.
The feed is delivered as CSV with three columns — domain, category and dns_status — or as JSON if that suits your tooling better. Three columns is not a limitation you have to work around; it is what makes this trivially loadable everywhere, because every reference-data primitive in every SIEM accepts a flat table with a key column. What follows is the shortest sensible path in each of the four platforms most teams are running.
Land the CSV in a lookup file, define a lookup definition over it keyed on domain, and — if you want enrichment without editing every search — attach it as an automatic lookup on the relevant sourcetypes so category and dns_status simply appear as fields on DNS and proxy events.
A watchlist is the right primitive for a list this size: it is queryable from KQL with _GetWatchlist(), it does not consume analytics ingestion, and it can be refreshed from a scheduled Logic App or an Azure Function that pulls the CSV and replaces the items.
DnsEvents, your proxy table, EmailUrlInfo or whatever custom tables your collectors write. Threat-intelligence indicator ingestion introduces its own expiry semantics that you must then reconcile with the feed's removals — a watchlist keeps the source of truth in one place.Index the feed into a small source index, define an enrich policy with domain as the match field, execute it to build the enrich index, and add an enrich processor to the ingest pipelines handling DNS, proxy and mail data. Documents then arrive already carrying the indicator fields, mapped wherever your schema expects threat indicator data to live.
Create a reference set of alphanumeric type keyed on the hostname and populate it from the daily file. Custom rules then test whether an extracted domain property is contained in that set, and an offence can be raised with elevated magnitude so it surfaces above routine traffic in the offence list.
Whichever object you use, the failure mode is identical and it is silent: the load job stops working and nothing tells you. The reference set still exists, the searches still run, the dashboards still render, and the detection coverage quietly ages out as the live threat landscape moves past a frozen snapshot. Nothing errors, because from the platform's point of view everything is fine.
/stats endpoint needs no API key, credits or authentication, so a lightweight monitor can also compare the upstream database size against the row count you actually loaded — a mismatch is the earliest signal that a truncated download made it through. Pair that with the deployment sequence and the incident response hand-off further down this page.// 1. Any internal host resolved a listed domain in the last hour dns_logs | join phishing_domains on query == domain // 2. A proxy session actually completed to a listed host proxy_logs | join phishing_domains on host == domain | where bytes_out > 0 // 3. More than one distinct user hit the same listed host in 30 minutes stats dc(user) as users by domain, bin(_time, 30m) | where users > 1 // 4. Message carrying a listed URL was delivered, not quarantined mail_logs | join phishing_domains on url_host == domain | where verdict == "delivered" // 5. Resolution of a listed host, then an authentication from the same user dns_logs | join phishing_domains on query == domain | join auth_logs on user within 2h
Most feeds hand you a number between zero and one and leave the interpretation to you. This one gives 0.98 or 0.0, which makes the routing decision considerably easier.
A confirmed match returns a confidence of 0.98 with a category of phishing/malware and a dns_status of resolves. A domain that is not in the database returns 0.0. There is nothing in between, and that is deliberate rather than a limitation. Every entry has been verified as having an active A record through rotating proxy infrastructure, so membership is a factual test rather than a heuristic judgement, and a graded score would be inventing precision that the underlying method does not possess.
For a detection engineer this removes an entire class of work. There is no threshold to argue about in a rule review, no drift as a vendor recalibrates a scoring model, and no ambiguous middle band that ends up routed to a queue nobody watches. A match is a match: raise it, enrich it, route it. What varies between your rules is not the indicator's confidence but the severity you assign based on what happened around it.
Resolution-only matches make excellent low-severity notables that a hunter reviews in batch — they are common, frequently the result of a mail client prefetching a link, and paging somebody at three in the morning for one will destroy your on-call rota's willingness to trust the whole detection. Reserve the high-severity path for correlations implying a human interacted: bytes transferred, a form posted, a credential used shortly afterwards from an unfamiliar address.
A 0.0 response means the hostname is not on the list. It does not mean the hostname is safe, and any dashboard tile, playbook branch or analyst runbook that treats it as an all-clear is wrong. A domain registered and first used within the same hour will not be present, because the verification that makes the data trustworthy also means an entry cannot exist before the host has been observed as resolving. Write that sentence into the runbook next to the enrichment step, because at three in the morning somebody will read a green tick and close a case on it.
checked and phishing_found returnedEvery day a set of domains enters the database that were unknown yesterday. Your logs already contain the sessions that touched them.
This is the capability that justifies the whole integration for most security operations teams, and it is the one that gets designed in last. A phishing domain is registered, stood up, used in a campaign and only afterwards observed, verified and added to a feed. During the interval between first use and first publication — which may be hours or may be days — your users were unprotected and your controls saw nothing worth flagging. But your logs recorded everything, and they are still there.
The feed ships with a daily changelog of domains added and removed. Its primary purpose is operational — it lets you apply an incremental update rather than replacing a large reference object wholesale. Its second use is the interesting one: take today's additions, a much smaller set than the full database, and search them across thirty, sixty or ninety days of retained DNS and proxy records. Anything that matches is a host in your estate that touched infrastructure now confirmed hostile, at a time when nothing could have told you.
That search is cheap because the input set is small, so it can run on a schedule rather than as an occasional heroic effort. You know the host, the user, the timestamp and whether the session completed. From there the questions are the ordinary ones — did an authentication follow, was a session token used from an unfamiliar address, has that account's behaviour changed since — and the retained data answers most of them without anyone touching the endpoint.
The accumulated series is what lets you answer "when did we first know about this" months later, which is a question incident reviews ask constantly. The day a domain entered your reference object is a fact about your detection coverage rather than about the domain, and only you can record it. Between that and the last_checked date on a check response, you can reconstruct the timeline of what was knowable when — precisely what a post-incident review needs and almost never has.
DNS logs are frequently the first thing dropped when storage gets tight, because they are voluminous and individually uninteresting. If retro-hunting matters, that retention decision is worth revisiting — resolution records are compact when summarised, and a rolled-up table of distinct hostnames per client per day is a fraction of the raw volume while remaining perfectly adequate for this kind of hunt. Teams running this alongside a broader intel programme usually pair it with the threat intelligence workflows.
An afternoon of work if the platform is already collecting DNS and proxy data. Longer if this is the conversation that finally gets those sources onboarded.
Find which sourcetypes actually carry a hostname field and what it is called in each. Normalise to one field name and to lower case before anything else.
A job shortly after 04:30 UTC, an authenticated download, a size and format check, then an atomic swap. Archive the changelog alongside it.
Lookup, watchlist, enrich index or reference set. Rebuild whatever derived structure the platform needs, and record the build date as a sentinel.
Resolution, completed session, multi-user cluster, delivered mail, and resolution-then-authentication. Grade severity by what happened, not by the indicator.
Enrich the case, block at the resolver, notify the user, and open the retro-hunt over retained logs. Then add the staleness alert and walk away.
The SOAR hand-off deserves more than one line, because it is where the value of a binary indicator becomes obvious. A playbook triggered by a confirmed match can act without waiting for a human to interpret a score. Push the hostname to the resolver blocklist and the proxy denylist so no further host in the estate reaches it. Pull the mail platform's record of who else received a message referencing the same domain, since a click is rarely a single delivery. Notify the affected user with a short, factual message rather than a scolding one. Open the retro-hunt automatically over the retained window. Attach the full check response to the case so the analyst who picks it up starts with the evidence rather than having to gather it.
Credential invalidation. Forcing a password reset and revoking sessions is the correct response to a probable compromise and a genuinely disruptive response to a false assumption, and the distinction between "resolved the hostname" and "posted a password to it" is not visible in DNS data alone. Automate everything up to that decision point, present the evidence clearly, and let a person make the call — the automation has already removed the twenty minutes of gathering that would otherwise have delayed it.
A new indicator source that fires overnight and pages somebody for a prefetched link will be muted within a fortnight, and a muted rule is worse than no rule because it still looks like coverage on a report. Start every new correlation search in a non-paging state, watch a fortnight of real volume in your own environment, and promote only the ones whose firing pattern justifies waking somebody. For the enforcement side of the same feed, see the DNS filtering and email security pages.
Including the two objections that should be raised and usually are not.
A three-column CSV with several hundred thousand rows is a small file by SIEM standards — comfortably in the tens of megabytes, and far smaller than a single hour of proxy logs in most estates. Every platform's reference primitive is built and indexed for exactly this: a large key set joined against a much larger event stream.Where people do run into trouble is by doing the join badly rather than by the size of the data. Joining on a non-normalised field, or on a full URL instead of a hostname, or without lower-casing both sides, turns an index-backed match into something far more expensive. Normalise the hostname once at ingest and the join is essentially free.
This is the objection worth taking seriously, and the honest answer is that it depends entirely on what your existing feeds contain. Many broad intel feeds carry a mixture of indicator types across a wide range of threat categories, with varying and often unstated verification standards, and phishing hostnames tend to be a thin slice of them that ages badly.What is different here is narrowness and the verification rule. Every entry has an active A record confirmed through rotating proxy infrastructure, and entries that stop resolving fall out rather than accumulating. That produces a smaller, denser set with fewer dead indicators, which matters directly to your false-positive rate — a domain that was taken down eight months ago and is now parked or re-registered by somebody legitimate is a classic source of confusing matches. If your current feeds already give you verified-live phishing coverage, you may not need this. Check the overlap before you buy: pull a sample of your existing phishing indicators and run them through /batch.
Yes, and in Elastic that is the natural shape — an enrich processor in the ingest pipeline stamps the indicator fields onto documents as they arrive, so the detection rule downstream is a simple field test with no join at all. Splunk's automatic lookups achieve something similar at search time without needing every search to know about the lookup.The trade-off to understand is that ingest-time enrichment records what was known at ingest time and never revisits it. A document processed on Monday will not gain an indicator field because the domain was added to the feed on Wednesday. That is precisely why you also want the search-time retro-hunt described above — the two approaches answer different questions and you want both.
It leaves the database on the next build and appears in that day's changelog as a removal, because the DNS verification step stops finding an active A record. If your reference object is refreshed by full replacement, it simply disappears; if you apply incremental updates, the changelog removals are what you process.Importantly, this does not retroactively invalidate anything. An alert raised last Tuesday for a session to that host was correct at the time, and the case should not be closed as a false positive on the grounds that the domain no longer resolves. Record the indicator state at alert time in the case itself, because the reference object is a moving picture of now and cases need a fixed picture of then.
For SIEM correlation you need the feed. Credits are priced per lookup, and bulk correlation is not a lookup workload — it is a join, and it needs the whole set present locally. The Daily Threat Feed is $499 per month, with feed downloads unlimited on a subscription, and all plans include priority support.Those bundled credits are genuinely useful in this context: they cover the analyst-initiated and SOAR-playbook calls that sit on top of the correlation layer, so you do not need a separate credit purchase to run the interactive half. If you only ever wanted the interactive half, monthly plans start at Growth at $99/month for 25,000 lookups. Both are on the pricing page, and delivery options including SFTP, S3 and STIX/TAXII are covered on the daily feed page.
Mostly it changes who does the work rather than what the work is. The load job has to live somewhere with outbound network access and a place to put the file, which in a managed arrangement usually means either your provider hosts the job or you deliver the file to a location they collect from. Enterprise delivery over SFTP or S3 exists partly for this reason.What you should insist on regardless of who operates it is visibility into the staleness check. A managed provider who cannot tell you the build date currently loaded in the reference object cannot tell you whether your detection coverage is current, and that is not an acceptable answer. See the managed security page for how this tends to be structured.
Possibly, and you should assume so and plan for it rather than discovering it live. The first run against retained logs frequently surfaces a backlog of historical matches, most of which are old, already-remediated or benign artefacts such as a mail client prefetching links in a quarantined message.Run the first pass as a report rather than as alerts, triage the backlog once as a batch, and only then enable scheduled searching in a paging configuration. Doing it the other way round — enabling alerts first and cleaning up afterwards — is how a genuinely useful detection source acquires a reputation for noise in its first week that it never quite recovers from.
It is a known-bad lookup and nothing more. A domain has to be observed and verified as resolving before it can be in the database, so anything registered and used within the same hour is not there yet. A 0.0 result means "not on this list", never "safe", and building a control that treats absence as an all-clear is a real design error rather than a theoretical one.It also covers domains rather than URLs, so a hostile page hosted on a legitimate compromised site will not appear — that is a genuinely different detection problem requiring content inspection, and no domain feed solves it. Treat this as one high-confidence layer among several: excellent at removing a large volume of confirmed-hostile traffic from your analysts' attention, and silent about everything it has not seen.
Read the current database size from the open statistics endpoint, load one day's file into a test lookup, and run it against a week of your own DNS and proxy data before you commit to anything. The interesting number is not how many domains are in the file — it is how many of them your own logs already touched.
