How can I systematically and reliably identify all possible Unicode characters that might fit into a particular custom category like "apostrophes"?

Problem

Unicode categories are often too exclusive or too inclusive to function as categories for processing text consistently, regardless of whether it was typed in notepad (where keystrokes map to ASCII double/single quotes, apostrophes, etc.) or Microsoft Word (where keystrokes map to non-ASCII characters in Unicode or a Windows code page).

Desired Categories for Text Processing

I'd like to build a text processor using Regex expressions that can consi开发者_StackOverflow社区stently and exhaustively identify language elements like:

1. apostrophe (used in a contraction, presumably with a letter on each side)
2. quote (and more specifically whether each is initial/final/single/double)
3. dash (em dash, en dash, hyphen, minus, etc.) -- Unicode category: "DashPunctuation"
4. open/close grouping punctuation (parenthesis, brackets, braces, etc.) -- Unicode categories: "OpenPunctuation" and "ClosePunctuation"

However, such categories do not always correspond with any particular Unicode category or set of categories.

Quote/Apostrophe Issue

Categories like "InitialQuotePunctuation" and "FinalQuotePunctuation" don't distinguish between single and double quotes, and they actually exclude the common ASCII double and single quotes, which are neutral/straight and therefore fall into neither the initial nor final category specifically. Instead, we find them in the "OtherPunctuation" category, along with many other common but unrelated punctuation symbols. The real problem is that we need to separate out the single quote characters to get a complete set of potential apostrophe characters, since someone could use such characters as an apostrophe.

Also a problem for quotes/apostrophes, is that the grave and acute symbols, which could easily be used as a quote/apostrophe by typing in notepad, are under the "ModifierSymbol" category.

Final Thoughts on Desired Solution

I think the dash and parenthetical categories are fairly complete, although the "DashPunctuation" category excludes the "swung dash" and "dashed overline", which are found in "OtherPunctuation" category with the ASCII single/double quotes.

I imagine that detecting the language elements I've identified above would require a two stage process where a single Unicode character (such as an ASCII apostrophe) would be allowed to fall into multiple categories at first (such as apostrophe AND single quote), such that a secondary contextual check could resolve which category is the best fit for its apparent usage.

The Regex for the quote category might include a character class like this: [\p{Pi}\p{Pf}"'], with an additional check to ensure it has a non-letter character on one side or the other.

Question

How can I systematically and reliably identify all possible characters of each of the four categories listed? As I explained, Unicode categories alone aren't sufficient.

Here's an example answer in case no one knows what I'm asking.

To identify characters in category 1, apostrophe, you'll have to:

1. Manually pick out all apostrophe-looking characters from the two Unicode categories InitialQuotePunctuation and FinalQuotePunctuation
2. Include the ASCII single quote/apostrophe (U+0027)
3. Include the Unicode "modifier letter apostrophe" (http://www.fileformat.info/info/unicode/char/02bc/index.htm) and look at it's "see also section" to find related characters that resemble it and may be used as an apostrophe

To identify characters in category 3, dash, you'll have to:

1. Include characters in the Unicode "DashPunctuation" category
2. Also include the dash-like characters "swung dash" and "dashed overline" from the "OtherPunctuation" category
3. Eyeball the whole Unicode character database for anything else that resembles or might be used as a dash or open various word processors and see what they use as a dash in case it's not included in the set you've identified already

etc.

I don't think there's a simpler or more reliable way to do this. Any thoughts or alternative techniques, or reasonable already-defined character sets for this purpose are welcome.