Distance Metric for clustering elements in a sparse matrix

I am attempting to cluster approximately 12000 elements based on approximately 1200 binary开发者_开发问答 variables using K-means. None of the conventional distance metrics (euclidean, manhattan, Hamming, Levenshtein) have produced satisfactory results.

I have devised the following metric.

Dist(x,y)= Min of P(x=0|y=1) P(y=0|x=1)

Has anyone used a similar approach to this type of problem? Are there any obvious flaws in using this metric? I am relatively new to data mining and would appreciate any feedback.

Thanks

What is the domain you are trying to model? Can you use insights from the domain for clustering? There is very little that can be said in this general setting. I would try dimension reduction in order to get more meaningful features first, and cluster later, but using domain knowledge can help here.

You can use Jaccard's distance over binary attributes. Assume you are comparing 2 rows A and B.

M11 - is the number of attributes where both Ai=Bi=1
M01 - is the number of attributes where Ai=0 and Bi=1
M10 - is the number of attributes where Ai=1 and Bi=0

The Jaccard coefficient will be

J = M11/(M11+M10+M01)

The Jaccard distance will be:

D=(M01+M10)/(M11+M10+M01)