Should one use distances (dissimilarities) or similarities in R for clustering?

I'm doing a cluster problem, and the proxy package in R provides both dist and simil functions.

For my purpose I need a distance matrix, so I initially used dist, and here's the code:

distanceMatrix <- dist(dfm[,-1], method='Pearson')
clusters <- hclust(distanceMatrix)  
clusters$labels <- dfm[,1]#colnames(dfm)[-1]
plot(clusters, labels=clusters$labels)

But after I ploted the image I found that the cluster result is not the way I expecte it to be, since I know what it should look like.

So I tried simil instead, and the code is like:

distanceMatrix <- simil(dfm[,-1], method='Pearson')
clusters <- hclust(pr_simil2dist(distanceMatrix))   
clusters$labels <- dfm[,1]#colnames(dfm)[-1]
plot(clusters, labels=clusters$labels)

This code computes a similarity matrix using simil, then convert it to distance matrix using pr_simil2dist, then I plot it and get the result I expected !

I'm confused about the relationship between dist and simil. According to the relationship described in the documentation, shouldn't the two code snippet has the same result?

Where am I wrong ?

Edit:

You can try my code with dfm of the following value, sorry for the bad indentation.

                             Blog china kids music yahoo want wrong
                         Gawker     0    1     0     0    7     0
                  Read/WriteWeb     2    0     1     3    1     1
                 WWdN: In Exile     0    2     4     0    0     0
           ProBlogger Blog Tips     0    0     0     0    2     0
                    Seth's Blog     0    0     1     0    3     1
 The Huffington Post | Raw Feed     0    6     0     0   14     5

Edit:

Actually the sample data is taken from a very big data frame using tail, and I get completely different matrix using dist and simil+pr_simil2dist. The full data can found here.

In case I made other silly mistakes,开发者_如何学Go here's the full code of my function:

The code I use to read in data:

dfm<- read.table(filename, header=T, sep='\t', quote='')

Code for clustering:

hcluster <- function(dfm, distance='Pearson'){
    dfm <- tail(dfm)[,c(1:7)] # I use this to give the sample data.
    distanceMatrix <- simil(dfm[,-1], method=pearson)
    clusters <- hclust(pr_simil2dist(distanceMatrix))   
    clusters$labels <- dfm[,1]#colnames(dfm)[-1]
    plot(clusters, labels=clusters$labels)
}

Matrix using dist:

           94         95         96         97         98
95 -0.2531580                                            
96 -0.2556859 -0.4629100                                 
97  0.9897783 -0.1581139 -0.2927700                      
98  0.8742800 -0.2760788 -0.1022397  0.9079594           
99  0.9114339 -0.5020405 -0.2810414  0.8713293  0.8096980

Matrix using simil+pr_simil2dist:

           94         95         96         97         98
95 1.25315802                                            
96 1.25568595 1.46291005                                 
97 0.01022173 1.15811388 1.29277002                      
98 0.12572004 1.27607882 1.10223973 0.09204062           
99 0.08856608 1.50204055 1.28104139 0.12867065 0.19030202

You can see that corresponding elements in the two matrices add up to 1, which I think is not right. So there must be something I'm doing wrong.

Edit:

After I specify names in the read.table function to read in the data frame, the dist way and simil+pr_simil2dist way give the same correct result. So technically problem solved, but I don't know why my original way of handling data frame have anything to do with dist and simil.

Any one has a clue on that ?

I'm not sure what you mean by not as per expected. If I compute the distance/similarity matrix via proxy::dist() or via simil() and convert to a dissimilarity I get the same matrix:

> dist(dfm, method='Pearson')
                                  Gawker Read/WriteWeb WWdN: In Exile ProBlogger Blog Tips Seth's Blog
Read/WriteWeb                  0.2662006                                                              
WWdN: In Exile                 0.2822594     0.2662006                                                
ProBlogger Blog Tips           0.2928932     0.5917517      0.6984887                                 
Seth's Blog                    0.2662006     0.2928932      0.4072510            0.2928932            
The Huffington Post | Raw Feed 0.1835034     0.2312939      0.2662006            0.2928932   0.2312939

> pr_simil2dist(simil(dfm, method = "pearson"))
                                  Gawker Read/WriteWeb WWdN: In Exile ProBlogger Blog Tips Seth's Blog
Read/WriteWeb                  0.2662006                                                              
WWdN: In Exile                 0.2822594     0.2662006                                                
ProBlogger Blog Tips           0.2928932     0.5917517      0.6984887                                 
Seth's Blog                    0.2662006     0.2928932      0.4072510            0.2928932            
The Huffington Post | Raw Feed 0.1835034     0.2312939      0.2662006            0.2928932   0.2312939

and

d1 <- dist(dfm, method='Pearson')
d2 <- pr_simil2dist(simil(dfm, method = "pearson"))
h1 <- hclust(d1)
h2 <- hclust(d2)
layout(matrix(1:2, ncol = 2))
plot(h1)
plot(h2)
layout(1)
all.equal(h1, h2)

The last line yields:

> all.equal(h1, h2)
[1] "Component 6: target, current do not match when deparsed"

which is telling us that h1 and h2 are exactly the same except for the matched function call (obviously as we used d1 and d2 in the respective calls).

The figure produced is:

If you set your object up correctly, then you won't need to fiddle with the labels. Look at the row.names argument to read.table() to see how to specify a column be used as the row labels when the data are read in.

All of this was done using:

dfm <- structure(list(china = c(0L, 2L, 0L, 0L, 0L, 0L), kids = c(1L, 
0L, 2L, 0L, 0L, 6L), music = c(0L, 1L, 4L, 0L, 1L, 0L), yahoo = c(0L, 
3L, 0L, 0L, 0L, 0L), want = c(7L, 1L, 0L, 2L, 3L, 14L), wrong = c(0L, 
1L, 0L, 0L, 1L, 5L)), .Names = c("china", "kids", "music", "yahoo", 
"want", "wrong"), class = "data.frame", row.names = c("Gawker", 
"Read/WriteWeb", "WWdN: In Exile", "ProBlogger Blog Tips", "Seth's Blog", 
"The Huffington Post | Raw Feed"))