Wildcard String Search Algorithm

In my program I need to search in a quite big string (~1 mb) for a relatively small substring (< 1 kb). The problem is the string contains simple wildcards in the sense of "a?c" which means I want to search for strings like "abc" or also "apc",... (I am only interested in the first occurence).

Until now I use the trivial approach (here in pseudocode)

algorithm "search", input: haystack(string), needle(string)

for(i = 0, i < length(haystack), ++i)
 if(!CompareMemory(haystack+i,needle,length(needle))
  return i;

return -1; (Not found)
开发者_开发知识库

Where "CompareMemory" returns 0 iff the first and second argument are identical (also concerning wildcards) only regarding the amount of bytes the third argument gives.

My question is now if there is a fast algorithm for this (you don't have to give it, but if you do I would prefer c++, c or pseudocode). I started here but I think most of the fast algorithms don't allow wildcards (by the way they exploit the nature of strings).

I hope the format of the question is ok because I am new here, thank you in advance!

A fast way, which is kind of the same thing as using a regexp, (which I would recommend anyway), is to find something that is fixed in needle, "a", but not "?", and search for it, then see if you've got a complete match.

j = firstNonWildcardPos(needle)
for(i = j, i < length(haystack)-length(needle)+j, ++i)
  if(haystack[i] == needle[j])
    if(!CompareMemory(haystack+i-j,needle,length(needle))
      return i;

return -1; (Not found)

A regexp would generate code similar to this (I believe).

Among strings over an alphabet of c characters, let S have length s and let T_1 ... T_k have average length b. S will be searched for each of the k target strings. (The problem statement doesn't mention multiple searches of a given string; I mention it below because in that paradigm my program does well.)

The program uses O(s+c) time and space for setup, and (if S and the T_i are random strings) O(k*u*s/c) + O(k*b + k*b*s/c^u) total time for searching, with u=3 in program as shown. For longer targets, u should be increased, and rare, widely-separated key characters chosen.

In step 1, the program creates an array L of s+TsizMax integers (in program, TsizMax = allowed target length) and uses it for c lists of locations of next occurrences of characters, with list heads in H[] and tails in T[]. This is the O(s+c) time and space step.

In step 2, the program repeatedly reads and processes target strings. Step 2A chooses u = 3 different non-wild key characters (in current target). As shown, the program just uses the first three such characters; with a tiny bit more work, it could instead use the rarest characters in the target, to improve performance. Note, it doesn't cope with targets with fewer than three such characters.

The line "L[T[r]] = L[g+i] = g+i;" within Step 2A sets up a guard cell in L with proper delta offset so that Step 2G will automatically execute at end of search, without needing any extra testing during the search. T[r] indexes the tail cell of the list for character r, so cell L[g+i] becomes a new, self-referencing, end-of-list for character r. (This technique allows the loops to run with a minimum of extraneous condition testing.)

Step 2B sets vars a,b,c to head-of-list locations, and sets deltas dab, dac, and dbc corresponding to distances between the chosen key characters in target.

Step 2C checks if key characters appear in S. This step is necessary because otherwise a while loop in Step 2E will hang. We don't want more checks within those while loops because they are the inner loops of search.

Step 2D does steps 2E to 2i until var c points to after end of S, at which point it is impossible to make any more matches.

Step 2E consists of u = 3 while loops, that "enforce delta distances", that is, crawl indexes a,b,c along over each other as long as they are not pattern-compatible. The while loops are fairly fast, each being in essence (with ++si instrumentation removed) "while (v+d < w) v = L[v]" for various v, d, w. Replicating the three while loops a few times may increase performance a little and will not change net results.

In Step 2G, we know that the u key characters match, so we do a complete compare of target to match point, with wild-character handling. Step 2H reports result of compare. Program as given also reports non-matches in this section; remove that in production.

Step 2I advances all the key-character indexes, because none of the currently-indexed characters can be the key part of another match.

You can run the program to see a few operation-count statistics. For example, the output

Target 5=<de?ga>
012345678901234567890123456789012345678901
abc1efgabc2efgabcde3gabcdefg4bcdefgabc5efg

@  17, de?ga  and  de3ga  match
@  24, de?ga  and  defg4  differ
@  31, de?ga  and  defga  match
Advances:  'd' 0+3  'e' 3+3  'g' 3+3  = 6+9 = 15

shows that Step 2G was entered 3 times (ie, the key characters matched 3 times); the full compare succeeded twice; step 2E while loops advanced indexes 6 times; step 2I advanced indexes 9 times; there were 15 advances in all, to search the 42-character string for the de?ga target.

/* jiw 
$Id: stringsearch.c,v 1.2 2011/08/19 08:53:44 j-waldby Exp j-waldby $

Re: Concept-code for searching a long string for short targets,
    where targets may contain wildcard characters.

The user can enter any number of targets as command line parameters.
This code has 2 long strings available for testing; if the first
character of the first parameter is '1' the jay[42] string is used,
else kay[321].

Eg, for tests with *hay = jay use command like

   ./stringsearch 1e?g a?cd bc?e?g c?efg de?ga ddee? ddee?f

or with *hay = kay,

   ./stringsearch  bc?e?  jih?  pa?j  ?av??j

to exercise program.

Copyright 2011 James Waldby.  Offered without warranty
under GPL v3 terms as at http://www.gnu.org/licenses/gpl.html
*/
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <limits.h>
//================================================
int main(int argc, char *argv[]) {
  char jay[]="abc1efgabc2efgabcde3gabcdefg4bcdefgabc5efg";
  char kay[]="ludehkhtdiokihtmaihitoia1htkjkkchajajavpajkihtijkhijhipaja"
    "etpajamhkajajacpajihiatokajavtoia2pkjpajjhiifakacpajjhiatkpajfojii"
    "etkajamhpajajakpajihiatoiakavtoia3pakpajjhiifakacpajjhkatvpajfojii"
    "ihiifojjjjhijpjkhtfdoiajadijpkoia4jihtfjavpapakjhiifjpajihiifkjach"
    "ihikfkjjjjhijpjkhtfdoiajakijptoik4jihtfjakpapajjkiifjpajkhiifajkch";
  char *hay = (argc>1 && argv[1][0]=='1')? jay:kay;

  enum { chars=1<<CHAR_BIT, TsizMax=40, Lsiz=TsizMax+sizeof kay, L1, L2 };
  int L[L2], H[chars], T[chars], g, k, par;

  // Step 1.  Make arrays L, H, T.
  for (k=0; k<chars; ++k) H[k] = T[k] = L1; // Init H and T
  for (g=0; hay[g]; ++g) {  // Make linked character lists for hay.
    k = hay[g];            // In same loop, could count char freqs.
    if (T[k]==L1) H[k] = T[k] = g;
    T[k] = L[T[k]] = g;
  }

  // Step 2.  Read and process target strings.
  for (par=1; par<argc; ++par) {
    int alpha[3], at[3], a=g, b=g, c=g, da, dab, dbc, dac, i, j, r;
    char * targ = argv[par];
    enum { wild = '?' };
    int sa=0, sb=0, sc=0, ta=0, tb=0, tc=0;
    printf ("Target %d=<%s>\n", par, targ);

    // Step 2A.  Choose 3 non-wild characters to follow.
    // As is, chooses first 3 non-wilds for a,b,c.
    // Could instead choose 3 rarest characters.
    for (j=0; j<3; ++j) alpha[j] = -j;
    for (i=j=0; targ[i] && j<3; ++i)
      if (targ[i] != wild) {
        r = alpha[j] = targ[i];
        if (alpha[0]==alpha[1] || alpha[1]==alpha[2]
            || alpha[0]==alpha[2]) continue;
        at[j] = i;
        L[T[r]] = L[g+i] = g+i;
        ++j;
      }
    if (j != 3) {
      printf ("  Too few target chars\n"); 
      continue;
    }

    // Step 2B.  Set a,b,c to head-of-list locations, set deltas.
    da  = at[0];
    a = H[alpha[0]];  dab = at[1]-at[0];
    b = H[alpha[1]];  dbc = at[2]-at[1];
    c = H[alpha[2]];  dac = at[2]-at[0];

    // Step 2C.  See if key characters appear in haystack
    if (a >= g || b >= g || c >= g) {
      printf ("  No match on some character\n");
      continue;      
    }

    for (g=0; hay[g]; ++g) printf ("%d", g%10);
    printf ("\n%s\n", hay);    // Show haystack, for user aid

    // Step 2D.  Search for match
    while (c < g) {
      // Step 2E.  Enforce delta distances
      while (a+dab < b) {a = L[a]; ++sa; } // Replicate these
      while (b+dbc < c) {b = L[b]; ++sb; } //  3 abc lines as many 
      while (a+dac > c) {c = L[c]; ++sc; } //   times as you like.
      while (a+dab < b) {a = L[a]; ++sa; } // Replicate these
      while (b+dbc < c) {b = L[b]; ++sb; } //  3 abc lines as many 
      while (a+dac > c) {c = L[c]; ++sc; } //   times as you like.

      // Step 2F.  See if delta distances were met
      if (a+dab==b && b+dbc==c && c<g) {
        // Step 2G.  Yes, so we have 3-letter-match and need to test whole match.
        r = a-da;
        for (k=0; targ[k]; ++k)
          if ((hay[r+k] != targ[k]) && (targ[k] != wild))
            break;
        printf ("@ %3d, %s  and  ", r, targ);
        for (i=0; targ[i]; ++i) putchar(hay[r++]);
        // Step 2H.  Report match, if found
        puts (targ[k]? "  differ" : "  match");
        // Step 2I.  Advance all of a,b,c, to go on looking
        a = L[a]; ++ta;
        b = L[b]; ++tb;
        c = L[c]; ++tc;
      }
    }
    printf ("Advances:  '%c' %d+%d  '%c' %d+%d  '%c' %d+%d  = %d+%d = %d\n",
        alpha[0], sa,ta, alpha[1], sb,tb, alpha[2], sc,tc,
        sa+sb+sc, ta+tb+tc, sa+sb+sc+ta+tb+tc);
  }
  return 0;
}

Note, if you like this answer better than current preferred answer, unmark that one and mark this one. :)

Regular expressions usually use a finite state automation-based search, I think. Try implementing that.