Help me optimize this SQL Server 2005 query
I have this query that's running too slow. I'm not sure what all info I should provide in order to make it easy for you to help me, but I'll take a stab at it and then add more when you big-brains inevitably ask for stuff that I either didn't think to include or don't know what is.
I want to identify customers (but using only part of their address -- accommodating households and businesses) who first registered a purchase in 2006.
My first attempt was:
select
distinct a.line1 + '|' + substring(a.zip,1,5)
from
registrations r
join customers c on r.custID = c.id
join addresses a on c.addressID = a.id
where year(r.purchaseDate) = 2006
and a.line1 + '|' + substring(a.zip,1,5) not in (
select
distinct a.line1 + '|' + substring(a.zip,1,5)
from
registrations r
join customers c on r.custID = c.id
join addresses a on c.addressID = a.id
where
year(r.purchaseDate) < 2006
)
and when it was running too long, I switched a NOT EXISTS (with which I am less comfortable, but willing to experiment) like
select
distinct a.line1 + '|' + substring(a.zip,1,5)
from
registrations r
join customers c on r.custID = c.id
join addresses a on c.addressID = a.id
where
year(r.purchaseDate) = 2006
and not exists (
select
1
from
registrations r
join customers c on r.custID = c.id
join addresses ia on c.addressID = ia.id
where
ia.line1 + '|' + substring(ia.zip,1,5) = a.line1 + '|' + substring(a.zip,1,5) and
year(r.purchaseDate) < 2006
)
group by
a.line1 + '|' + substring(a.zip,1,5)
But it runs too long too. Like no results in 17 hours kind of too long. I think the first thing to consider is where my SQL might be wrong or sub-optimal but in case it's not that, I want to also give you enough info to consider the environment.
So, diagnostic info. You probably don't care, but just in case: it's running on a G6 server with four quads and 20 GB RAM; each query is limited to occupying four processors to keep performance up for request开发者_运维技巧s from the web server; when I'm running this query, we're clearing off other big imports and reports because of deadlocks but the web server is customer-facing and can't be stopped.) There are roughly: 15 million registrations, 11 million customers and 8.6 million addresses. I rebuilt all of the indexes just to be sure that fragmentation wasn't the problem. However, I'm not really sure how to index properly, so I'm totally open to this being a problem -- some of these indexes were a result of me futzing around and some were scripts that one of the MS analysis tools gave me to improve performance. I'm also not exactly sure how to convey index info to you, so I'll just give the create scripts:
ALTER TABLE [dbo].[registrations] ADD CONSTRAINT [PK_flatRegistrations_1] PRIMARY KEY CLUSTERED
(
[Id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
.
ALTER TABLE [dbo].[customers] ADD CONSTRAINT [PK_flatCustomers_1] PRIMARY KEY CLUSTERED
(
[Id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
.
ALTER TABLE [dbo].[addresses] ADD CONSTRAINT [PK_addresses] PRIMARY KEY CLUSTERED
(
[ID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
CREATE NONCLUSTERED INDEX [addresses] ON [dbo].[addresses]
(
[line1] ASC,
[line2] ASC,
[city] ASC,
[state] ASC,
[zip] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
CREATE NONCLUSTERED INDEX [deliverable] ON [dbo].[addresses]
(
[addressDeliverable] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
CREATE NONCLUSTERED INDEX [_dta_index_addresses_5_1543676547__K9_K1_6] ON [dbo].[addresses]
(
[addressDeliverable] ASC,
[ID] ASC
)
INCLUDE ( [zip]) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
CREATE NONCLUSTERED INDEX [_dta_index_addresses_5_1543676547__K1_K9_6] ON [dbo].[addresses]
(
[ID] ASC,
[addressDeliverable] ASC
)
INCLUDE ( [zip]) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
CREATE NONCLUSTERED INDEX [_dta_index_addresses_5_1543676547__K1_6] ON [dbo].[addresses]
(
[ID] ASC
)
INCLUDE ( [zip]) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
Thanks a bunch for your time!
I think your Not Exists subquery's table alias is wrong. Try this:
select r.custID,
a.line1 + '|' + substring(a.zip,1,5)
from registrations r
join customers c on r.custID = c.id
join addresses a on c.addressID = a.id
where r.purchaseDate between '2006-01-01' and '2006-12-31'
and not exists (
select 1
from registrations ir
join customers ic on ir.custID = ic.id
join addresses ia on ic.addressID = ia.id
where ia.line1 = a.line1
and substring(ia.zip,1,5) = substring(a.zip,1,5)
and ir.purchaseDate < '2006-12-31'
)
My first try would be replace:
year(r.purchaseDate) = 2006
with:
r.purchaseDate BETWEEN '2006-01-01' and '2006-12-31 23:59:59'
and also year(r.purchaseDate) < 2006
with r.purchaseDate < '2006-01-01'
.
and make sure that there is an index on purchaseDate
.
Next thing (if you have enough resources to run it):
-- create temporary table to prepare data
CREATE TABLE #addrs (yearr int, pattern varchar(100)) -- depends on a.line1 length
-- calculate all patterns for purchase before 1st Jan 2007
INSERT INTO
#addrs (yearr, pattern)
SELECT
YEAR(r.purchaseDate),
a.line1 + '|' + substring(a.zip,1,5)
from
registrations r
join customers c on r.custID = c.id
join addresses a on c.addressID = a.id
where
r.purchaseDate < `2007-01-01`
-- optionally, but could be useful in query below
CREATE INDEX idx_temp ON #addrs (pattern, yearr)
-- original query rewritten
SELECT
DISTINCT pattern
FROM
#addrs a
WHERE
a.yearr = 2006
and not exists (
select top 1 1
from
#addrs aa
where
aa.pattern = a.pattern
and aa.yearr < 2006
)
Second solution could have some typos and could not compile from first try. It's just an idea.
SubString(A.zip, 1, 5) must be causing the table scan. Is this one time query? If so, get the result of the below query and store it in a new table. Make indexes on AddressToCompare and PurchaseDate and run your subsequent query against the new table.
Select
R.ID
, R.CustID
, C.AddressID
, A.line1 + '|' + SubString(A.zip, 1, 5) As AddressToCompare
, R.PurchaseDate
From
Registrations R
Inner Join Customers C On R.CustID = C.ID
Inner Join addresses A On C.AddressID = A.ID
Where
R.PurchaseDate <= '2006-12-31'
First your logic is poor, business and customers move in and out of addresses, so comparing on the address rather than the customer is a guarantee of wrong results. Just becasue ABC copmany ordered something in 2002 does not mean that DEF company didn;t have thier first corder in 2006 as ABC comapny and DEF Company have no relationship to each other. If you need relationships of people inteh same company or household then havea table to to store them properly, don;t rely on incorrect hack.
Assuming you can't do this and this is aproces that will run more than once, then you need to persist a column inthe address table with
line1 + '|' + substring(zip,1,5)
This prevents you from having to calculate it on the fly.
精彩评论