SQL: Fast Cumulative Frequency Query (postgres)
I'm looking to get Cumulative Frequency Data out of our database. I've created a simple temp table with all unique status update counts that we've seen, and the number of users that ha开发者_StackOverflow社区ve that amount of status updates.
Table "pg_temp_4.statuses_count_tmp"
Column | Type | Modifiers
----------------+---------+-----------
statuses_count | integer |
frequency | bigint |
Indexes:
"statuses_count_idx" UNIQUE, btree (statuses_count)
My current query is:
select statuses_count, frequency/(select * from total_statuses)::float, (select sum(frequency)/(select * from total_statuses)::float AS percentage from statuses_count_tmp WHERE statuses_count <= SCT.statuses_count) AS cumulative_percent FROM statuses_count_tmp AS SCT ORDER BY statuses_count DESC;
But this takes quite a while and the number of queries grows quite quickly. So with the ~50,000 rows I have, I'm looking at 50k factorial rows to be read. Sitting here watching the query grind away I'm hoping theres a better solution that I haven't through of yet.
Hoping to get something like this:
0 0.26975161 0.26975161
1 0.15306534 0.42281695
2 0.05513516 0.47795211
3 0.03050646 0.50845857
4 0.02064444 0.52910301
Should be solvable with windowing functions, assuming you have PostgreSQL 8.4 or later. I am guessing that total_statuses
is a view or temp table along the lines of select sum(frequency) from statuses_count_tmp
? I wrote it as a CTE here which should make it calculate the result just once for the duration of the statement:
with total_statuses as (select sum(frequency) from statuses_count_tmp)
select statuses_count,
frequency / (select * from total_statuses) as frequency,
sum(frequency) over(order by statuses_count)
/ (select * from total_statuses) as cumulative_frequency
from statuses_count_tmp
Without 8.4's window functions your best bet is simply to process the data iteratively:
create type cumulative_sum_type as ( statuses_count int, frequency numeric, cumulative_frequency numeric );
create or replace function cumulative_sum() returns setof cumulative_sum_type strict stable language plpgsql as $$
declare
running_total bigint := 0;
total bigint;
data_in record;
data_out cumulative_sum_type;
begin
select sum(frequency) into total from statuses_count_tmp;
for data_in in select statuses_count, frequency from statuses_count_tmp order by statuses_count
loop
data_out.statuses_count := data_in.statuses_count;
running_total := running_total + data_in.frequency;
data_out.frequency = data_in.frequency::numeric / total;
data_out.cumulative_frequency = running_total::numeric / total;
return next data_out;
end loop;
end;
$$;
select * from cumulative_sum();
精彩评论