Every short link you mint lends your domain's reputation to a destination somebody else chose. Screening that destination — at creation, and again at click time — is the difference between a link service and an unwitting laundering service for other people's phishing pages.
Nothing about the mechanics is unusual. What is unusual is who bears the cost when the destination turns out to be hostile.
The product you actually sell is not compression. Nobody needs eight characters instead of ninety. What a short link provides is a stable, memorable, trackable handle that inherits whatever standing your domain has accumulated: it fits in a message, it survives being printed, it does not break when the destination changes, and it looks like it came from you. That last property is the valuable one, and it is also the one an abuser is buying when they sign up for a free tier.
What the abuser is actually purchasing
Because the visible hostname is yours, the destination's own reputation is hidden at exactly the moment a recipient decides whether to click. Every judgement a person or a filter would normally make about a URL — is this domain plausible, is it new, does it look like a lookalike of a bank — is deferred until after the redirect has already happened. You have not just failed to warn anyone; you have supplied the mechanism that suppresses the warning.
This is why screening is not an optional trust-and-safety nicety for a link service. It is a load-bearing part of the product. A shortener that does not look at destinations is, functionally, a reputation laundering service: it takes an input with poor standing, wraps it in a domain with good standing, and emits something that passes checks the original would have failed. Nobody sets out to build that. Plenty of services become it by omission.
The good news is that the check is small. You already have the destination URL in hand at creation time, because storing it is the entire point of the service. Extracting the registrable host and asking whether it is a known, currently-live phishing domain is one GET /api/v1/check call costing a single credit. Against a database of more than 200,000 DNS-verified active domains, that one call intercepts the large, unoriginal middle of the abuse distribution — the operators recycling known infrastructure rather than standing up something new for every campaign.
The rest of this page is about where to put that call, how often to repeat it, what to do with the answer, and how to keep the cost sane once you are minting hundreds of thousands of links a month. It is also honest about the limit: this is a known-bad lookup, not a verdict on an unknown page, and a mature link service pairs it with signals only it can see.
Four things have to happen in the create handler, in this order, and none of them are expensive.
When a user submits a destination, you are holding the whole decision. Nothing has been printed, nothing has been posted to a social account, no recipient has seen anything, and no mailbox provider has formed an opinion about your domain. Rejecting the request costs you one credit and one error message. Rejecting the same link three days later costs you an abuse report, a support thread and a small dent in your sending reputation.
Do the check inside the create call rather than behind a queue. Asynchronous screening feels safer for latency but it opens a window in which a live short link points at a known phishing page, and that window is exactly when a campaign fires. If you must keep creation fast, allocate the slug in a pending state and refuse to serve redirects until the check completes; a link that 404s for 200 milliseconds is far better than one that resolves to a credential form.
Strip the scheme, lowercase the host, drop any port, resolve internationalised domain names to their punycode form, and check the registrable domain rather than a deep subdomain string. The check endpoint will strip http://, https:// and path segments for you, but it cannot guess which of several encodings a submitted string was meant to be, and inconsistent normalisation is the most common reason a screening layer quietly misses things it should have caught.
Store is_phishing, category, dns_status, confidence and last_checked next to the destination. This costs nothing at write time and becomes the backbone of every abuse conversation you will have later, because it lets you state precisely what was known about a destination on the day you accepted it.
last_checked date, and enrol the link in the nightly re-screen.A short link is a promise you keep indefinitely, and the thing it points at is under somebody else's control the entire time. Screening at creation validates the destination on the day it was submitted. It says nothing about the destination on the day somebody clicks, which may be nine months later, after the domain has changed hands twice.
The bait-and-switch is the deliberate version and it is the reason destination editing deserves the same scrutiny as destination creation. If your product lets a user change where a link points — and most link services do, because that flexibility is a headline feature — then every edit is a new submission and must be re-screened. Treating edits as a metadata update rather than a security event is one of the most common design gaps in this category.
The slow burn is the accidental version and it is more common. Perfectly legitimate sites get compromised. Hosting accounts get taken over. A marketing microdomain gets abandoned, expires, and is picked up by somebody who noticed it still receives traffic. In all of these the destination string never changed, so no edit event fires, and only a periodic re-check will notice.
Re-screening at click time closes the gap entirely, because the verdict is then never older than the request. Keep a short-lived local cache keyed on the destination host so that a viral link does not generate one API call per click, fail open on a lookup error rather than breaking every redirect you serve, and set a timeout tight enough that the check can never become the reason your redirects are slow.
A screening layer is only as good as the list underneath it, so it is worth being precise about what that list contains and how often it changes.
Every domain in the database has been verified by DNS through rotating proxy infrastructure, and only domains with an active A record are included. That constraint is what makes the list usable for a shortener. A blocklist that accumulates every domain ever reported grows without bound and drifts towards false positives as abandoned infrastructure is recycled by legitimate owners; a list that drops entries when they stop resolving stays proportionate to the threat that exists today. The all-time total is far larger than the active count, and the gap between the two is the churn you would otherwise be carrying.
The rebuild runs every 24 hours and lands at 04:30 UTC, and the daily feed subscription ships both the full CSV and a changelog of domains added and removed since the previous build. That changelog is the piece most link services underuse. It converts "re-screen the back catalogue" from a full table scan into a delta operation: you only need to reconsider links whose destination host appears in the additions list, which for most services is a few thousand rows rather than several million.
Responses are per domain, not per URL, which suits this use case almost exactly. A shortener stores destinations by host far more usefully than by full URL, because the interesting relationships — one account creating four hundred links across nine hosts, or one host receiving links from thirty accounts — are all host-level. Keying your screening cache and your abuse analytics on the registrable domain rather than the full destination string is worth doing on day one; retrofitting it later is painful.
Rate limiting matters for click-time screening in particular. Ten requests per second per key sounds generous until a single link goes viral, which is why local caching is not an optimisation but a requirement. In practice a well-built cache keyed on host will absorb the overwhelming majority of redirect traffic, because popular links point at a small number of popular hosts, and the API is left handling the long tail of destinations nobody has seen recently.
These are not alternatives so much as layers with different cost curves. The right answer for most services is two of the three, and which two depends entirely on link volume.
| Consideration | Create-time API check | Click-time API check | Daily feed, matched locally |
|---|---|---|---|
| Blocks the link before anyone sees it | Yes — the slug is never issued | No — the link already exists | Only if you screen on create against the local copy |
| Catches a destination that goes bad later | No — verdict is frozen at submission | Yes — verdict is never older than the request | Yes — on the next sync, via the changelog |
| Cost driver | Links created | Clicks served, minus cache hits | Flat subscription, independent of volume |
| Cost at 300,000 links / month | 300,000 credits — roughly $600 at the Business rate of $0.0020 | Depends entirely on cache hit rate; unbounded without one | $499 per month flat, whatever the volume |
| Adds latency to the hot path | One call on create only | One call per cache miss on redirect | None — the match is a local lookup |
| Works during an upstream outage | Needs a fail-open or fail-closed policy | Needs a fail-open policy or redirects break | Yes — the data is already on your side |
| Re-screening the whole back catalogue | Prohibitive — one credit per stored link | Never happens for links nobody clicks | Trivial — a join against your own table |
Read the bottom row twice. A service with five million live links cannot pay per lookup to revalidate them, and revalidation is precisely what the time-of-check problem demands. Loading the feed once a day and matching locally turns an impossible bill into a database join — and the feed subscription includes 100,000 API credits, which comfortably covers the create-time path for a mid-sized service.
# create-time: one destination, one credit curl -s "https://phishingdetectionapi.com/api/v1/check?domain=$HOST&apikey=$KEY" # backfill / triage: up to 100 hosts per request, one lookup each curl -s -X POST https://phishingdetectionapi.com/api/v1/batch \ -H 'Content-Type: application/json' \ -d "{\"apikey\":\"$KEY\",\"domains\":[\"a.example\",\"b.example\"]}" # -> results[], checked, phishing_found, credits_used # nightly: pull the feed, then match in your own database curl -s "https://phishingdetectionapi.com/api/v1/feed?apikey=$KEY&format=csv" \ | psql -c "\copy pd_feed(domain,category,dns_status) FROM STDIN CSV"
An interstitial is the right default for anything uncertain. A full-page warning that names the destination host, states plainly why it is being flagged and requires a deliberate second action gives the user back the information the short link removed, without you having to be right with total confidence. It also produces a click-through rate you can measure, which is genuinely useful telemetry about how your users respond to warnings.
Hard blocking is the right response when the destination is a confirmed hit. A confidence of 0.98 against a domain that is currently resolving and categorised phishing/malware is not a judgement call you need to devolve to the person who just clicked. Serve a terminal page, do not offer a "continue anyway" affordance, and disable the link rather than only the redirect — otherwise the same slug is reactivated the moment your cache expires.
The thing that makes an abuse desk effective is not the queue software but the evidence attached to each item. Store the destination host, the full API response including last_checked and confidence, the creating account and its signup metadata, the creation timestamp and the click count at the moment of the block. That package answers the three questions every abuse conversation asks: what did you know, when did you know it, and what did you do about it.
| Field to retain | Where it comes from | Question it answers |
|---|---|---|
domain | Normalised destination host | What was the link actually pointing at? |
is_phishing | API response at decision time | What did the database say? |
confidence | API response at decision time | How firm was the match? |
dns_status | API response at decision time | Was the host live when you blocked it? |
last_checked | API response at decision time | How fresh was the verdict you acted on? |
| Account & signup metadata | Your own user table | Who created it, and how new were they? |
| Click count at block | Your own redirect logs | How much exposure happened first? |
Abuse reports frequently arrive weeks after the link stopped working, and a report you cannot reconstruct is a report you must treat as unresolved. Teams that already run structured incident response processes will recognise the pattern: the value is in the record being complete at the moment of the decision, because nobody can rebuild it afterwards.
resolves. No continue option. Disable the link, not just the redirect.Abuse costs you customers one at a time. A blocklisted short domain costs you all of them at once, and getting off a mailbox provider's list is far harder than staying off it.
Mailbox providers, browser safe-browsing lists and security vendors do not have the patience to distinguish your good links from your bad ones. If a meaningful share of traffic bearing your hostname terminates at phishing pages, the pragmatic response is to treat the hostname as untrustworthy in aggregate. That decision is made about the domain, not the slug, which means one abusive campaign can render every legitimate link your customers ever created undeliverable and unclickable. This is the failure mode that has ended link services.
The exposure is worse for a shared public shortener than for a branded one, because a single domain carries the behaviour of every user. It is one of the strongest arguments for offering branded short domains to enterprise customers: each customer's reputation is then isolated on their own hostname, so an incident is contained to the party that caused it. That isolation only helps if you still screen — a branded domain that ships phishing burns the customer's brand rather than yours, which is a different disaster, not the absence of one.
Free tiers are where the pressure concentrates, because the cost of an abusive account is whatever a disposable email address costs. The signals worth watching are behavioural rather than content-based: signup bursts from a narrow address range, accounts that create hundreds of links within minutes of registration, many distinct accounts pointing at the same destination host, or a single account fanning out across dozens of freshly registered hosts. None of those is proof, and all of them are worth combining with a screening verdict before you decide.
Then there is the case that catches everyone eventually: your shortener pointing at another shortener. Chaining is a deliberate evasion because it defeats create-time screening completely — the destination you screen is a perfectly reputable link service, and the actual payload sits one hop further on. Resolve chains server-side with a hop cap, check every host you traverse rather than only the last, and consider refusing to accept known shortener domains as destinations at all unless the account has a reason to need it. The same reasoning applies to QR codes generated from your links, where the printed artefact hides the chain even more completely.
A service creating three hundred thousand links a month and screening each one at creation spends three hundred thousand credits; on the Business package at Business at $499/month for 250,000 lookupsthat is roughly $0.0020 per link, and at the Enterprise Plus rate of $0.0010 it halves again. Trying to re-screen a live catalogue of several million links the same way is simply not affordable. Loading the daily feed once and matching locally costs a flat $499 per month regardless of catalogue size, includes 100,000 credits for the create-time path, and turns revalidation into a query against your own tables. Most services should end up running both: the API on the write path, the feed on everything else.
The questions below are the ones that come up when a link product moves from "we should screen" to "we are screening in production".
Both, with the daily feed carrying the load between them. Create-time is the only check that happens before anyone can be harmed, and it is cheap because it scales with links created rather than clicks served. Click-time is the only check whose verdict is current, which is what the time-of-check problem demands. Doing click-time checks entirely through the API is expensive without a cache, so most services cache aggressively on the host and use a locally loaded feed as the backstop for everything in the catalogue that nobody happens to click today.
Start with create-time. It is a single call in a code path you already own, it produces the account-level abuse signal you will want later, and it stops the crude bulk campaigns that make up most of the volume. Add the click-time cache once you can measure how often a stored destination changes verdict between creation and first click.
Do not use per-lookup calls for this. Take the feed subscription, load the CSV into a table with an index on the domain column, and join it against your stored destination hosts. That converts a multi-million-credit operation into a database query costing nothing per row. The feed also ships a changelog of additions and removals with each daily build, so after the first full load you only need to reconsider links whose host appears in the day's additions.
Treat it as "not currently in a database of confirmed, actively resolving phishing domains". That is a precise and useful statement, and it is not a guarantee. This is a known-bad lookup rather than a heuristic classifier, so a destination registered an hour ago for a campaign that has not yet been reported anywhere will come back clean. Combine the verdict with the signals only you can see: account age, creation velocity, how many accounts point at the same host, and whether the destination is itself a redirector.
Say "checked against known phishing domains" rather than "safe" wherever a verdict is shown to a user or a customer. The wording costs nothing, survives the day a clean destination turns out to be hostile, and keeps a support conversation about a missed link from becoming a conversation about a promise you made.
Resolve the chain on your side with a hard hop cap and a short timeout, then check every host you passed through rather than only the final one. Batch them together — up to 100 domains per request at one lookup each — so a full chain costs about the same as a handful of single lookups. Many services additionally refuse known shortener domains as destinations by default, on the grounds that a legitimate user rarely needs to point one short link at another, and make it an explicit allowance for accounts that genuinely do.
Only if you let it. Cache verdicts by host, since popular links concentrate on a small number of destinations, and serve cache hits without touching the network. On a miss, apply a tight timeout and fail open so that an upstream problem can never break your core function. If your redirect path cannot tolerate any external call at all, run purely from a locally loaded feed on the hot path and use the API for creation and for triage.
No, and you should not. The key is the username chosen at registration, and anything shipped to a browser or app can be extracted. Call the API from your server, where you are already validating and storing the destination anyway. This also lets you enforce the 10 requests per second per key limit centrally rather than hoping thousands of clients behave, and keeps a single place to add caching, normalisation and logging.
Screening does not remove an existing listing, but it addresses the reason for it and it gives you something to show. When you approach a provider about delisting, being able to demonstrate that destinations are screened at creation, re-screened against a daily-updated database, and that flagged links are disabled with a retained evidence record, is a substantially stronger position than an assurance that you will do better. Pick a package on the pricing page and measure your own back catalogue before you talk to anyone.
A batch run over your stored destination hosts gives you a defensible figure — how many links in the catalogue point at confirmed phishing domains today, and how many have already been disabled. Providers respond to a measured number and a remediation date far better than to a description of a policy.
Pick a monthly plan and screen a sample of your live destinations this week. Then decide whether the create-time path, a click-time cache or a nightly feed sync fits your volume — all three run against the same DNS-verified database, rebuilt every 24 hours.
