Comments on: Combining contiguous date ranges in a SQL query – using CTE recursion http://cdonner.com/combining-contiguous-date-ranges-in-a-sql-query-using-cte-recursion.htm Sun, 29 Apr 2012 14:35:11 +0000 hourly 1 http://wordpress.org/?v=3.3.2 By: Christian Donner http://cdonner.com/combining-contiguous-date-ranges-in-a-sql-query-using-cte-recursion.htm/comment-page-1#comment-6079 Christian Donner Sat, 12 Feb 2011 19:01:53 +0000 http://cdonner.com/?p=77#comment-6079 Tom, Glad it worked for you. Tom, Glad it worked for you.

]]> By: tom http://cdonner.com/combining-contiguous-date-ranges-in-a-sql-query-using-cte-recursion.htm/comment-page-1#comment-6077 tom Sat, 12 Feb 2011 09:53:47 +0000 http://cdonner.com/?p=77#comment-6077 Aaah silly me problem solved! The culprit was in the WHERE clause of my query. I needed to put the "last.EmpID = this.EmpID" check into the ON clause instead. Looking back at the query and the penny dropped - Of course my original query was taking so long, the left outer join itself produces about 50 million rows!!! Anyway, thanks Chris for your solution again and I should put my thinking cap on first before just blatanly copying code! :) Cheers! Aaah silly me problem solved!
The culprit was in the WHERE clause of my query. I needed to put the “last.EmpID = this.EmpID” check into the ON clause instead.
Looking back at the query and the penny dropped – Of course my original query was taking so long, the left outer join itself produces about 50 million rows!!!

Anyway, thanks Chris for your solution again and I should put my thinking cap on first before just blatanly copying code! :)

Cheers!

]]> By: Christian Donner http://cdonner.com/combining-contiguous-date-ranges-in-a-sql-query-using-cte-recursion.htm/comment-page-1#comment-6074 Christian Donner Fri, 11 Feb 2011 04:02:24 +0000 http://cdonner.com/?p=77#comment-6074 Tom, I would love to, but right now I can't spare the time and I don't even know if I would be of any help. I used in recursion in larger databases, but I can't give you any specific metrics. It's been years. Tom, I would love to, but right now I can’t spare the time and I don’t even know if I would be of any help. I used in recursion in larger databases, but I can’t give you any specific metrics. It’s been years.

]]> By: tom http://cdonner.com/combining-contiguous-date-ranges-in-a-sql-query-using-cte-recursion.htm/comment-page-1#comment-6072 tom Thu, 10 Feb 2011 10:33:11 +0000 http://cdonner.com/?p=77#comment-6072 Chris, You're probably too busy, but would you mind executing this query if I provided you with the data set I'm working with? It's about 5 mb csv file. It's just that I'm very keen to see why it's so slow. Cheers. Chris,

You’re probably too busy, but would you mind executing this query if I provided you with the data set I’m working with? It’s about 5 mb csv file.
It’s just that I’m very keen to see why it’s so slow.

Cheers.

]]> By: tom http://cdonner.com/combining-contiguous-date-ranges-in-a-sql-query-using-cte-recursion.htm/comment-page-1#comment-6071 tom Thu, 10 Feb 2011 10:28:09 +0000 http://cdonner.com/?p=77#comment-6071 Hmmm, I created the indices as you suggested, but it looks like it has no effect on the slowness of the recursion. Also, I'm now just calling the recursive function with: SELECT EmpID, StartDate as start, EndDate as [end], BSB,CC, Level FROM Recursion without any grouping and ordering just to see if there is any improvement. I noticed that it straight away spits out the first 40 000 to 45000 records quite quickly and then it slows down to a crawl, spitting out only a handful of rows every couple of minutes. And this is just for Level 0. 10 minutes into the query and it is still processing Level 0 records. Is there anything else I could try? Or does recursion only work for small to medium sets of data? I'm curious to know, how many records do you have in your data set and what is the performance like? Hmmm, I created the indices as you suggested, but it looks like it has no effect on the slowness of the recursion.

Also, I’m now just calling the recursive function with:
SELECT EmpID, StartDate as start, EndDate as [end], BSB,CC, Level
FROM Recursion
without any grouping and ordering just to see if there is any improvement. I noticed that it straight away spits out the first 40 000 to 45000 records quite quickly and then it slows down to a crawl, spitting out only a handful of rows every couple of minutes. And this is just for Level 0. 10 minutes into the query and it is still processing Level 0 records.

Is there anything else I could try? Or does recursion only work for small to medium sets of data?

I’m curious to know, how many records do you have in your data set and what is the performance like?

]]> By: Christian Donner http://cdonner.com/combining-contiguous-date-ranges-in-a-sql-query-using-cte-recursion.htm/comment-page-1#comment-6068 Christian Donner Wed, 09 Feb 2011 15:14:40 +0000 http://cdonner.com/?p=77#comment-6068 Start with a composite index on the group by columns: StartDate, EmpID, BSB, CC Keep the index on EmpId and Startdate. Start with a composite index on the group by columns:

StartDate, EmpID, BSB, CC

Keep the index on EmpId and Startdate.

]]> By: tom http://cdonner.com/combining-contiguous-date-ranges-in-a-sql-query-using-cte-recursion.htm/comment-page-1#comment-6066 tom Wed, 09 Feb 2011 12:54:18 +0000 http://cdonner.com/?p=77#comment-6066 Also, here is my query: I had to include an employee id because I have multiple employees. I hope what I have done is correct. ;WITH Recursion (EmpID, StartDate, EndDate, BSB, CC, Level) AS ( -- Anchor member definition SELECT this.EmpID, this.StartDate, this.EndDate, this.BSB, this.CC, 0 AS Level FROM TestEmployees AS this LEFT OUTER JOIN dbo.TestEmployees AS last ON last.EndDate + 1 = this.StartDate WHERE (last.EmpID = this.EmpID AND last.BSB + last.CC this.BSB + this.CC) or last.BSB + last.CC is null UNION ALL -- Recursive member definition SELECT last.EmpID, last.StartDate, this.EndDate, this.BSB, this.CC, Level + 1 FROM TestEmployees AS this inner JOIN Recursion AS last ON last.EndDate + 1 = this.StartDate WHERE last.EmpID = this.EmpID AND last.BSB + last.CC = this.BSB + this.CC and Level<99 ) -- Statement that executes the CTE SELECT EmpID, StartDate as start, MAX(EndDate) as [end], BSB,CC FROM Recursion group by StartDate, EmpID, BSB,CC order by EmpID, StartDate I've created a clustered index on EmpID and StartDate but the query has been executing for 1hr 45 min! Also, here is my query:
I had to include an employee id because I have multiple employees. I hope what I have done is correct.

;WITH Recursion (EmpID, StartDate, EndDate, BSB, CC, Level)
AS
(
– Anchor member definition
SELECT this.EmpID, this.StartDate, this.EndDate, this.BSB, this.CC, 0 AS Level
FROM TestEmployees AS this
LEFT OUTER JOIN dbo.TestEmployees AS last
ON last.EndDate + 1 = this.StartDate
WHERE (last.EmpID = this.EmpID AND last.BSB + last.CC this.BSB + this.CC) or last.BSB + last.CC is null
UNION ALL
– Recursive member definition
SELECT last.EmpID, last.StartDate, this.EndDate, this.BSB, this.CC, Level + 1
FROM TestEmployees AS this
inner JOIN Recursion AS last
ON last.EndDate + 1 = this.StartDate
WHERE last.EmpID = this.EmpID AND last.BSB + last.CC = this.BSB + this.CC
and Level<99
)
– Statement that executes the CTE
SELECT EmpID, StartDate as start, MAX(EndDate) as [end], BSB,CC
FROM Recursion
group by StartDate, EmpID, BSB,CC
order by EmpID, StartDate

I've created a clustered index on EmpID and StartDate but the query has been executing for 1hr 45 min!

]]> By: tom http://cdonner.com/combining-contiguous-date-ranges-in-a-sql-query-using-cte-recursion.htm/comment-page-1#comment-6065 tom Wed, 09 Feb 2011 10:45:59 +0000 http://cdonner.com/?p=77#comment-6065 Hi Christian, Thanks for replying, I actually have no indices on any of the columns. Would I need at least an index on StartDate column? I really want to use your recursive solution as it works out perfectly, but I can't justify the 4.5 hours! :) Hi Christian,

Thanks for replying, I actually have no indices on any of the columns. Would I need at least an index on StartDate column?

I really want to use your recursive solution as it works out perfectly, but I can’t justify the 4.5 hours! :)

]]> By: Christian Donner http://cdonner.com/combining-contiguous-date-ranges-in-a-sql-query-using-cte-recursion.htm/comment-page-1#comment-6064 Christian Donner Tue, 08 Feb 2011 14:20:38 +0000 http://cdonner.com/?p=77#comment-6064 This is not normal. I would look at your indices first and see if the right ones are in place. This is not normal. I would look at your indices first and see if the right ones are in place.

]]> By: tom http://cdonner.com/combining-contiguous-date-ranges-in-a-sql-query-using-cte-recursion.htm/comment-page-1#comment-6063 tom Tue, 08 Feb 2011 11:22:27 +0000 http://cdonner.com/?p=77#comment-6063 Hi Christian, I tried using your solution but it is EXTREMELY slow! I have contiguous date ranges for my employees but I have 450 000 rows in my table. This query took 4.5 hours to complete!!! Is this normal and is there a way to speed it up? Thanks Hi Christian,
I tried using your solution but it is EXTREMELY slow! I have contiguous date ranges for my employees but I have 450 000 rows in my table. This query took 4.5 hours to complete!!! Is this normal and is there a way to speed it up?

Thanks

]]>