How to pull every email address and link out of messy text

To pull every email address out of a messy page or export, use a pattern that matches the shape of an address, then clean up the punctuation that sits next to it. A simple pattern grabs a full stop or comma at the end of a sentence, and it misses addresses written in an obfuscated way. Expect to check the list by eye before you use it.

Extract emails and linksPull emails out of a pageOpen

Why simple patterns go wrong

A common pattern is a run of non-space characters, an @ sign, and another run of non-space characters. It is easy to write, and it has two failures.

It over-matches. In the sentence "Contact [email protected], or [email protected]." the pattern "\S+@\S+" returns "[email protected]," with the comma and "[email protected]." with the full stop. Those are not valid addresses in your list, and a mail merge or a spam filter will reject them.

It under-matches. Addresses written as "ana [at] example [dot] com" or "ana at example dot com" are not matched at all. Writers use this form to avoid scraping, and some pages use it for the same reason. Replace the words first, then extract.

A better pattern allows only the characters an address can contain on the left and only a letter-based domain on the right. The pattern "[\w.+-]+@[\w-]+\.[\w.-]+" is a reasonable start. Even so, it can keep a full stop after the domain when the address ends a sentence, because the last part of the domain allows dots. Strip one trailing full stop from each match.

Here is what that pattern returns for the two examples above:

• "Write to [email protected]." gives "[email protected]." with the trailing full stop, which must be removed.
• "Contact [email protected], or [email protected]." gives "[email protected]" and "[email protected].", so the same clean-up applies.

Links behave the same way

URLs have the same problem. A link at the end of a sentence often carries a full stop, comma or closing bracket. A link inside brackets, such as (see https://example.com/page), can pick up the closing bracket if the pattern is greedy. Clean the trailing punctuation, then check that each link opens in a browser.

Links can also be relative, such as /contact, which an extraction for email addresses will ignore. Decide whether you need absolute addresses only, and say so when you run the extraction.

Steps that work on most exports

1. Replace obfuscated forms first: "[at]" becomes @ and "[dot]" becomes a full stop, but only where you can see that the text is an address.
2. Run the extraction with a pattern that matches the address shape.
3. Strip trailing punctuation from each match.
4. Remove duplicates, ignoring case, since email addresses are not case-sensitive in practice. Remove duplicate lines does this in one pass.
5. Read through the final list. Look for values that are not addresses, such as image file names that happen to include an @ sign.

A worked example: a page contains "Sales: [email protected]", "[email protected]." and "Support: [email protected]". The extraction returns three matches. After stripping the full stop and removing duplicates ignoring case, the result is two addresses: [email protected] and [email protected]. If you want the list in order, sort lines puts it from A to Z.

Keep the list clean and lawful

Email addresses are personal data in many places, and anti-spam rules exist in many countries. Collecting an address from a public page does not automatically allow you to email it. Rules such as the EU's GDPR and the US CAN-SPAM Act are two well-known examples, and they differ in what they require. This is general information, not legal advice. Before you send anything, check the rules that apply to you and to the people on the list, and keep an unsubscribe option for bulk messages.

Edge cases

• Text that is not an address. Strings such as [email protected] can match. Remove anything that ends in a file extension.
• Hidden characters. A zero-width space inside an address breaks the match, so the address is missed. Remove hidden characters before extraction if a known address is missing.
• Encoded addresses. Some pages hide addresses in HTML entities such as @ for @. Decode the page text first, or the pattern will not see the @ sign.
• Long exports. For large files, extract into a file and check the count against the number of lines you expect.

Quick answers

Why does my email extraction include a full stop at the end?

A full stop is allowed inside a domain, so a pattern that accepts dots can catch the one that ends a sentence. Strip one trailing full stop from each match.

Can a tool find addresses written as "name at example dot com"?

Not reliably without a replacement step. Convert the words to symbols first, but only where the text is clearly an address, so ordinary sentences are not changed.

Is it legal to email every address I extract from a web page?

That depends on where you and the recipients are, and on what the message is. Anti-spam and privacy rules apply, so check them first. This is general information, not legal advice.