refactoring large cursor queries by splitting into multiple cursors

Another PL/SQL refactoring question!

I have several cursors that are of the general simplified form:

cursor_1 is
  with X as (select col1, col2 from TAB where col1 = '1'),
       Y as (select col1, col2 from TAB where col2 = '3'),
  /*main select*/
  select count(X.col1), ...
  from X inner join Y on...
  group by rollup (X.col1, ...

cursor_2 is
  with X as (select col1, col2 from TAB where col1 = '7' and col2 = '9' and col3 = 'TEST'),
       Y as (select col1, col2 from TAB where col3 = '6'),
  /*main select*/
  select count(X.col1), ...
  from X inner join Y on...
  group by rollup (X.col1, ...


cursor_2 is
  with X as (select col1, col2 from TAB where col1 IS NULL ),
       Y as (select col1, col2 from TAB where col2 IS NOT NULL ),
  /*main select*/
  select count(X.col1), ...
  from X inner join Y on...
  group by rollup (X.col1, ...


...
begin
    for r in cursor_1 loop
       print_report_results(r);
    end loop;

    for r in cursor_2 loop
       print_report_results(r);
    end loop;
    ...
end;

Basically, all of these cursors (there's more than 3) are the same summary/reporting queries. The difference is in the factored subqueries. There are always 2 factored subqueries, "X" and "Y", and they always select the same columns to feed into the main reporting query.

The problem is that the main reporting query is VERY large, about 70 lines. This itself isn't so bad, but it was copy-pasted for ALL of the reporting queries (I think there's over a dozen).

Since the only difference is in the factored subqueries (and they all return the same columns, it's really just a difference in the tables they select from and their conditions) I was hoping to find a way to refactor all this so that there is ONE query for the giant report and smaller ones for the various factored subqueries so that when changes are made to the way the report is done, I only have to do it in one place, not a dozen. Not to mention a much easier-to-navigate (and read) file!

I just don't know how to properly refactor something like this. I was thinking pipelined functions? I'm not sure they're appropriate for this though, or if there's a simpler way...

On the other hand, I also wonder if performance would be signi开发者_开发知识库ficantly worse by splitting out the reporting query. Performance (speed) is an issue for this system. I'd rather not introduce changes for developer convenience if it adds significant execution time.

I guess what I'd ultimately like is something that looks sort of like this (I'm just not sure how to do this so that it will actually compile):

cursor main_report_cursor (in_X, in_Y) is
    with X as (select * from in_X),
         Y as (select * from in_Y)
  /*main select*/
  select count(X.col1), ...
  from X inner join Y on...
  group by rollup (X.col1, ...

cursor x_1 is
     select col1, col2 from TAB where col1 = '1';
cursor y_1 is
     select col1, col2 from TAB where col2 = '3'

...
begin
for r in main_report_cursor(x_1,y_1) loop
   print_report_results(r);
end loop;

for r in main_report_cursor(x_2,y_2) loop
   print_report_results(r);
end loop;
...

(Using Oracle 10g)

Use a pipelined function. For example:

drop table my_tab;
create table my_tab
(
col1 number,
col2 varchar2(10),
col3 char(1)
);
insert into my_tab values (1, 'One', 'X');
insert into my_tab values (1, 'One', 'Y');
insert into my_tab values (2, 'Two', 'X');
insert into my_tab values (2, 'Two', 'Y');
insert into my_tab values (3, 'Three', 'X');
insert into my_tab values (4, 'Four', 'Y');
commit;

-- define types
create or replace package refcur_pkg is
    --type people_tab is table of people%rowtype;
    type my_subquery_tab is table of my_tab%rowtype;
end refcur_pkg;

Create the function pipelined

-- create pipelined function
create or replace function get_tab_data(p_cur_num in number, p_cur_type in char)
return REFCUR_PKG.my_subquery_tab pipelined
IS
    v_ret  REFCUR_PKG.my_subquery_tab;
begin
    if (p_cur_num = 1) then
        if (upper(p_cur_type) = 'X') then
            for rec in (select * from my_tab where col1=1 and col3='X')
            loop
                pipe row(rec);
            end loop;
        elsif (upper(p_cur_type) = 'Y') then
            for rec in (select * from my_tab where col1=1 and col3='Y')
            loop
                pipe row(rec);
            end loop;
        else
            return;
        end if;
    elsif (p_cur_num = 2) then
        if (upper(p_cur_type) = 'X') then
            for rec in (select * from my_tab where col1=2 and col3='X')
            loop
                pipe row(rec);
            end loop;
        elsif (upper(p_cur_type) = 'Y') then
            for rec in (select * from my_tab where col1=2 and col3='Y')
            loop
                pipe row(rec);
            end loop;
        else
            return;
        end if;
    end if;
    return;
end;

MAIN procedure example

-- main procedure/usage
declare

  cursor sel_cur1 is
    with X as (select * from table(get_tab_data(1, 'x'))),
           Y as (select * from table(get_tab_data(1, 'y')))
    select X.col1, Y.col2 from X,Y where X.col1 = Y.col1;

begin
    for rec in sel_cur1
    loop
        dbms_output.put_line(rec.col1 || ',' ||  rec.col2);
    end loop;
end;

All of your various subqueries are reduced to a call to a single pipelined function, which determines the rows to return.

EDIT:

To combine all needed types and functions into 1 procedure, and also to use variables for subquery function parameters, I'm adding the following example:

create or replace procedure my_pipe
IS
    -- define types
    type my_subquery_tab is table of my_tab%rowtype;
    type ref_cur_t is ref cursor;
    v_ref_cur ref_cur_t; 

    -- define vars
    v_with_sql varchar2(4000);
    v_main_sql varchar2(32767);
    v_x1 number;
    v_x2 char;
    v_y1 number;
    v_y2 char;
    v_col1 my_tab.col1%type;
    v_col2 my_tab.col2%type;

    -- define local functions/procs
    function get_tab_data(p_cur_num in number, p_cur_type in char)
    return my_subquery_tab pipelined
    IS
        v_ret  my_subquery_tab;
    begin
        if (p_cur_num = 1) then
            if (upper(p_cur_type) = 'X') then
                for rec in (select * from my_tab where col1=1 and col3='X')
                loop
                    pipe row(rec);
                end loop;
            elsif (upper(p_cur_type) = 'Y') then
                for rec in (select * from my_tab where col1=1 and col3='Y')
                loop
                    pipe row(rec);
                end loop;
            else
                return;
            end if;
        elsif (p_cur_num = 2) then
            if (upper(p_cur_type) = 'X') then
                for rec in (select * from my_tab where col1=2 and col3='X')
                loop
                    pipe row(rec);
                end loop;
            elsif (upper(p_cur_type) = 'Y') then
                for rec in (select * from my_tab where col1=2 and col3='Y')
                loop
                    pipe row(rec);
                end loop;
            else
                return;
            end if;
        end if;
        return;
    end;

BEGIN
    ---------------------------------
    -- Setup SQL for cursors
    ---------------------------------
    -- this will have different parameter values for subqueries
    v_with_sql := q'{
    with X as (select * from table(get_tab_data(:x1, :x2))),
           Y as (select * from table(get_tab_data(:y1, :y2)))
    }';
    -- this will stay the same for all cursors
    v_main_sql := q'{
    select X.col1, Y.col2 from X,Y where X.col1 = Y.col1
    }';

    ---------------------------------
    -- set initial subquery parameters
    ---------------------------------
    v_x1 := 1;
    v_x2 := 'x';
    v_y1 := 1;
    v_y2 := 'y';
    open v_ref_cur for v_with_sql || v_main_sql using v_x1, v_x2, v_y1, v_y2;
    loop
        fetch v_ref_cur into v_col1, v_col2;
        exit when v_ref_cur%notfound;
        dbms_output.put_line(v_col1 || ',' ||  v_col2);
    end loop;
    close v_ref_cur;
    ---------------------------------
    -- change subquery parameters
    ---------------------------------
    v_x1 := 2;
    v_x2 := 'x';
    v_y1 := 2;
    v_y2 := 'y';
    open v_ref_cur for v_with_sql || v_main_sql using v_x1, v_x2, v_y1, v_y2;
    loop
        fetch v_ref_cur into v_col1, v_col2;
        exit when v_ref_cur%notfound;
        dbms_output.put_line(v_col1 || ',' ||  v_col2);
    end loop;
    close v_ref_cur;
end;

Note the benefit now is that even if you have many different cursors, you only need to define the main query and subquery SQL once. After that, you're just changing variables.

Cheers

--Create views that will be replaced by common table expressions later.
--The column names have to be the same, the actual content doesn't matter.
create or replace view x as select 'wrong' col1, 'wrong' col2 from dual;
create or replace view y as select 'wrong' col1, 'wrong' col2 from dual;

--Put the repetitive logic in one view
create or replace view main_select as
select count(x.col1) total, x.col2
from X inner join Y on x.col1 = y.col1
group by rollup (x.col1);

--Just querying the view produces the wrong results
select * from main_select;

--But when you add the common table expressions X and Y they override
--the dummy views and produce the real results.
declare
    cursor cursor_1 is
    with X as (select 'right' col1, 'right' col2 from dual),
         Y as (select 'right' col1, 'right' col2 from dual)
    select total, col2 from main_select;
    --... repeat for each cursor, just replace X and Y as necessary
begin
    for r in cursor_1 loop
        dbms_output.put_line(r.col2);
    end loop;
    null;
end;
/

This solution is a little weirder than the pipelined approach, and requires 3 new objects for the views, but it will probably run faster since there is less context switching between SQL and PL/SQL.

One possibility you could consider is using 2 Global Temporary Tables (GTTs) for X and Y. Then you just need one cursor, but you have to clear and re-populate the 2 GTTs several times - and if data volumes are large you may want to get optimiser stats on the GTTs each time too.

This is the sort of thing I mean:

cursor_gtt is
  select count(X.col1), ...
  from GTT_X inner join GTT_Y on...
  group by rollup (X.col1, ...

begin
    insert into gtt_x select col1, col2 from TAB where col1 = '1';
    insert into gtt_y select col1, col2 from TAB where col2 = '3';
    -- maybe get stats for gtt_x and gtt_y here
    for r in cursor_gtt loop
       print_report_results(r);
    end loop;

    delete gtt_x;
    delete gtt_y;
    insert into gtt_x select col1, col2 from TAB where col1 = '7' and col2 = '9' and col3 = 'TEST';
    insert into gtt_y select col1, col2 from TAB where col3 = '6'
    -- maybe get stats for gtt_x and gtt_y here
    for r in cursor_gtt loop
       print_report_results(r);
    end loop;
    ...
end;

So the same 2 GTTs are re-populated and the same cursor is used each time.

What about creating a view for the main query? That pretties up your code and centralizes the main query to boot.