calculating the density of a set

(I wish my mathematical vocabulary was more developed)

I have a website. On that website is a video. As a user watches the video, a bit of javascript stores how far they have gotten so far in the video. When they stop watching the video, that number of seconds is stored. There's no pattern to when the js will do this, unfortunately.

So if one person is watching the video, we might see this set:

3
6
8
10
12
16

And another person might get bored 开发者_如何学运维immediately:

1
3

This data is all stored in the same place, anonymously. So the sorted table with all this info would look like this:

1
3
3
6
8
10
12
16

Finally, the amount of times the video is started at all is stored. In this case it would be 2.

So. How do I get the average 'high-time' (the farthest reached point in the video) for all of the times the video was played?

I know that if we had a value for every second:

1
2
3
4
5
6
7
...
14
15
16

1
2
3

Then we could count up the values and divide by the number of plays:

(19) / 2 = 9.5

Or if the data was otherwise uniform, say in increments of 5, then we could count that up and multiply it by 5 (in the example, we would have some loss of precision, but that's ok):

5
10
15

5

(4) * 5 / 2 = 10

So it seems like I have a general function which would work:

count * 1/d = avg

where d is the density of the numbers (in the example above with 5 second increments, 1/5).

Is there a way to derive the density, d, from a set of effectively random numbers?

Why not just keep the last time that has been provided, and average across those? If you either throw away, or only pay attention to, the last number, it seems like you could just average over these.

You might also want to check out the term standard deviation as the raw average of this might not be the most useful measurement. If you have the standard deviation as well, it could help you realize that you have an average of 7, but it is composed of mostly 1's and 15's.

If you HAVE to have all the data, like you suggested, I will try and think about this a little bit more. I'm not totally certain how you can associate a value with all the previous values that came with it. Do you ALWAYS know the sequence by which numbers are output? If so, I think I know of a way you could derive the 'last' one, which might be slightly computationally expensive.

If you only have a sequence of integers, I think you may be able to increase each value (exponentially?) to 'compensate' for the fact that a later value 'contains' earlier values. I'm still working through this idea, but maybe it will give someone else a seed. What if you average over the sum of these, and then take the base2 logarithm of this average? Does that provide any kind of useful metric? That should 'weight' the later values to the point where they compensate for the sum of earlier values. I think.

In python-esk:

sum = 0
numberOf = 0  
for node in nodes:
   sum = sum + node.value ^ 2
   numberOf = numberOf + 1
weightedAverage = log(sum/numberOf, 2) 
print weightedAverage
print "Thanks Brian"

I think that @brian-stiner is on the right track in one of his comments.

Start with something like:

1
3
3
6
8
10
12
16

Turn that into numbers and counts.

1, 1
3, 2
6, 1
8, 1
10, 1
12, 1
16, 1

And then reading from the end down, find all of the points that happened more often than any remaining ones.

3, 2
16, 1

Take differences in counts.

3, 1
16, 1

And you have an estimate of stopping places.

This will not be an unbiased estimate. But if the JavaScript is independently inconsistent and the number of people is large, the biases should be fairly small.

It won't be right, but it will be close enough for government work.

Assuming increments are always around 5, some missing, some a bit longer or shorter. Then it won't be easy (possible?) to do this exactly. My suggestion: compute something like a 'moving count'. Similar to moving average.

So, for second 7: count how many numbers are 5,6,7,8 or 9 and divide by 5. That will give you a pretty good guess of how many people watched the 7th second. Do the same for second 10. The difference would be close to the number of the people who left between second 7 and 10.

To get the total time watched for each user, you'll have parse the list smallest to largest. If you have 4 views, you'll go through your list until you find that you no longer have 4 identical numbers, the last number where you had 4 identical numbers is the maximum of the first view. Then you'll look for when the 3 identical numbers stop, and so on. For example:

4 views data: 1111222233334445566778

4 views side by side:

1 1 1 1
2 2 2 2
3 3 3 3 <- first view max is 3 seconds
4 4 4 <- second view max is 4 seconds
5 5
6 6
7 7 <- third view max is 7 seconds
8 <- fourth view max is 8 seconds

EDIT- Oh, I just noticed that they are not uniform. In that case, the moving average would probably be your best bet.

The number of values roughly corresponds to the number of time periods in which your javascript sends the values (minus 1/2 if the video stop is accompanied with a obligatory time posting, since its moment is random within the interval).

If all clients have similar intervals and you know them, you may just use:

SELECT  (COUNT(*) - 0.5) * 5.0 / (SELECT counter FROM countertable)
FROM    ticktable

5.0 is the interval between the posts here.

Note that it does not even look at the values: you could as well just store "ticks".

For the max time, you could use MAX() on your field. Perhaps something like...

SELECT MAX(play_time) AS maxTime FROM video

Which would give you the longest time someone has played the video for.

If you want other things, like AVG() then you'll need more complex queries, for collecting on a per-user basis etc etc.

MySQL also contains a Standard Deviation function called STDDEV() and STD() which could help you too.