Beyond Sql

Multiple cascade paths to the same table

2009-01-22T18:38:00.000-08:00

This example is based on the thread:

microsoft.public.sqlserver.programming
Jan 21, 2009
'JET Referential Integrity superior than SQL'
http://tinyurl.com/d58ubl 

Sql Server can't deal with more than 1 cascade to a table. Some nonsense about
cycles. I guess if two paths are involved they call it a bi-cycle ☺ It simply is
a poor system for relationally modeling a business problem. Simply replace it 
with Dataphor and use sql server to store the data. Sql server doesn't screw
up data storage ☺ Here's how simple this example is with Dataphor (and sql server
as the backend).

The tables will be persisted in sql server but Dataphor will (mercifully) perform all
the integrity checks.

create table users
{
id:Integer,
username:String tags{Storage.Length='50'},
key{id},
key{username}
};

create table table2
{
id:Integer,
user1:String tags{Storage.Length='50'}, 
user2:String tags{Storage.Length='50'}, 
user3:String tags{Storage.Length='50'} ,
key{id},
reference user1username {user1} references users {username}
update cascade delete cascade,
reference user2username {user2} references users {username}
update cascade delete cascade,
reference user3username {user3} references users {username}
update cascade delete cascade
};

users:= table{
              row{1 id,'john' username},
              row{2,'steve'},
              row{3,'judy'},
              row{4,'larry'},
              row{5,'bill'},
              row{6,'rita'}};

This row would violate the user3username constraint.
insert row{12 id,'john' user1,'rita' user2, 'stan' user3} into table2;              
//Error: The table users does not have a row with username "stan".

table2:=table{
              row{10 id,'john' user1,'steve' user2, 'judy' user3},
              row{11,'larry','rita', 'bill'},
              row{13,'larry','steve', 'steve'},
              row{14,'judy','steve', 'john'},
              row{15,'bill','judy','rita'},
              row{16,'steve','bill', 'judy'},
              row{17,'john','bill','john'}};
              
select table2;
id user1 user2 user3 
-- ----- ----- ----- 
10 john  steve judy  
11 larry rita  bill  
13 larry steve steve 
14 judy  steve john  
15 bill  judy  rita  
16 steve bill  judy  
17 john  bill  john  

Check the cascading updates. In table table2 all entries
of 'john' are replaced with 'paul'.

update users set {username:='paul'} where username='john';

select table2;
id user1 user2 user3 
-- ----- ----- ----- 
10 paul  steve judy  
11 larry rita  bill  
13 larry steve steve 
14 judy  steve paul  
15 bill  judy  rita  
16 steve bill  judy  
17 paul  bill  paul  

Check the cascading deletes. Any row in table2 with an entry
of 'steve' is deleted.

delete users where username='steve';

select table2;
id user1 user2 user3 
-- ----- ----- ----- 
11 larry rita  bill  
15 bill  judy  rita  
17 paul  bill  paul  

Dataphor is open source, get a copy here:
http://databaseconsultinggroup.com/downloads/

Extracting numbers from a string

2008-12-20T17:36:00.000-08:00

This example is based on the question asked in the thread:

microsoft.public.sqlserver.programming
Dec 18, 2008
Pulling a number out of an nvarchar field
http://tinyurl.com/3quyap

The OP was interested in pulling out the 1st occurrence of a number in a
string. The string has numbers and letters. So if the string is 'XY40A3' 
we want the number 40 which is the 1st of two numbers in the string.

This is very easy to do in Dataphor. Just as a table of numbers comes in
handy for solving many different problems so does a list of letters. 
The idea here is to treat each letter in the string as a 'delimiter'. We
then split the string using the list of letters as the delimiters so
what results are the number(s) in the string.

We can store the list (the letters of the alphabet) in a temporary table
for easy access.

create session table AlphaTable
 { AlphaList:list(String),
   key{ }
 };
AlphaTable:=
       table
            {
             row
             { {'A','B','C','D','E','F','G','H','I','J','K','L','M','N',
                'O','P','Q','R','S','T','U','V','W','X','Y','Z'} AlphaList}
             };      

For example if we split a string, transform it to a table  and remove any
blanks we'll just have numbers left. If we order by sequence (Index) it will
show the numbers as they occur from left to right. 

select ToTable('XY40A3'.Split(AlphaTable[].AlphaList),'StrNum','Index' )
                where StrNum>' ' 
                  order by {Index};                  

StrNum Index 
------ ----- 
40     2     
3      3     
           
Not only would it be easy to get the 1st number but we can get the
occurrence of any number easily. The 1st occurrence is just a special
case of the general problem of getting the Nth occurrence in a string.

By using ToTable(ToList(cursor by Index (that follows the order of
numbers from left to right in the string) we can create a consecutive rank
from 1 to N over the table of numbers that will allow direct access to the 
Nth number (if it exists).
       
select
 ToTable
        (  
         ToList
               (
                cursor
                      (
                        ( 
                         ToTable('XY40A3RAD853'.Split(AlphaTable[].AlphaList),'StrNum','Index' )
                         where StrNum>' ' 
                         {ToInteger(StrNum) Num,Index}
                        )
                         order by {Index}
                      )   
               )   
        )
         {Num,Index,sequence+1 NthIndex} ;
         
Num Index NthIndex 
--- ----- -------- 
3   3     2        
40  2     1        
853 6     3               
        
Here is the an operator for the Nth occurrence that takes into account
lower case letters and returns a -1 if the Nth occurrence doesn't exist.
        
create session operator NthNum(AStr:String,N:Integer):Integer
begin
var T1:= 
  ToTable
        (  
         ToList
               (
                cursor
                      (
                        ( 
                         ToTable(Upper(AStr).Split(AlphaTable[].AlphaList),'StrNum','Index' )
                         where StrNum>' ' 
                         {ToInteger(StrNum) Num,Index}
                        )
                         order by {Index}
                      )   
               )   
        )
         {Num,Index,sequence+1 NthIndex};
result:=IfNil((T1 adorn{key{NthIndex}})[N].Num,-1);
end;

select NthNum('SF346fs47sGs759 ',1); //returns 346
select NthNum('SF346fs47sGs759 ',2); //returns 37
select NthNum('SF346fs47sGs759 ',3); //returns 759  
select NthNum('SF346fs47sGs759 ',4); //returns -1

Here a table of strings in stored in an Sql Server database from which
we can extract the 1st occurrence of a number.

create table FooStrings
{
 keycol:Integer,
 datacol:String nil,
 key{keycol}
};
FooStrings:=
 table
  {
 row{1 keycol, 'XYZ40AB' datacol},
 row{2, 'WX32A'},
 row{3, '27 blah'},
 row{4, 'A87BNC30'},
 row{5, 'XY40A3'},
 row{6, 'TWFD'},
 row{7, 'XYA53GH5JGV934'},
 row{8, '7'},
 row{9, nil}
  };
  
select FooStrings add{NthNum(IfNil(datacol,' '),1) MyNumber}
                   with {IgnoreUnsupported = 'true'}
                    order by {keycol};
                      
keycol datacol        MyNumber 
------ -------------- -------- 
1      XYZ40AB        40       
2      WX32A          32       
3      27 blah        27       
4      A87BNC30       87       
5      XY40A3         40       
6      TWFD           -1       
7      XYA53GH5JGV934 53       
8      7              7        
9      <No Value>     -1   


Think how easy the operator can be modified if it's desired to return
the minimun or maximum number etc.

If you would like to see a coherent and easy to follow all t-sql solution
that doesn't cobble together string functions into spaghetti code see:
RAC - Are you Coordinated?

But I think you'll agree the much preferred solution is in Dataphor ☺

Sql server dynamic crosstabs by Jeff Moden

2008-12-06T21:05:00.000-08:00

This is the RAC version of Jeff Modens fine article on dynamic crosstabs
for sql server at:

Cross Tabs and Pivots, Part 2 - Dynamic Cross Tabs 2008/12/03 
www.sqlservercentral.com/articles/cross+tab/65048/

The data is generated by the method outlined in the article. The table was 
populated with 1 millions rows

- Basic crosstab. 2 secs in QA for S2005.
Exec Rac
@transform='Sum(SomeMoney) as SumMony',
@rows='Someletters2',
@pvtcol='(left(datename(mm,DATEADD(mm,DATEDIFF(mm,0,SomeDate),0)),3)+~ ~+
          cast(year(DATEADD(mm,DATEDIFF(mm,0,SomeDate),0)) as char(4))) as mthyr',
@pvtsort='month(DATEADD(mm,DATEDIFF(mm,0,SomeDate),0))', -- Sort pivot expression by an integer. 
@from='##JBMTest',
@WHERE='SomeDate>=~Jan  1 2008 12:00AM~ AND SomeDate<~Jul  1 2008 12:00AM~',
@rowtotalsposition='end',@racheck='y',@shell='n'

Someletters2 Funct   Jan 2008   Feb 2008   Mar 2008   Apr 2008   May 2008   Jun 2008   Totals
------------ ------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
AA           SumMony 685.67     763.64     656.13     575.93     879.13     192.13     3752.63
AB           SumMony 927.06     928.98     280.20     632.43     560.99     785.50     4115.16
AC           SumMony 791.09     555.18     916.71     273.23     187.48     508.31     3232.00
AD           SumMony 250.04     341.58     426.53     645.56     670.13     422.86     2756.70
AE           SumMony 809.14     487.21     295.33     625.92     716.12     527.19     3460.91
.
.
ZY           SumMony 776.32     682.98     326.17     677.69     546.87     926.54     3936.57
ZZ           SumMony 532.75     500.92     277.93     636.40     607.05     553.43     3108.48
Totals       SumMony 433997.25  383211.70  411913.12  411878.29  425431.07  409809.47  2476240.90

Here some additional bells and whistles are thrown in:)
 
-- Executed in 26 secs in QA for S2005.
Exec Rac
-- The same transformed repeated twice for different purposes. 
@transform='Sum(SomeMoney) as SumMony & Sum(SomeMoney) as [% row]',
@rows='Someletters2',
@pvtcol='(left(datename(mm,DATEADD(mm,DATEDIFF(mm,0,SomeDate),0)),3)+~ ~+
          cast(year(DATEADD(mm,DATEDIFF(mm,0,SomeDate),0)) as char(4))) as mthyr',
@pvtsort='month(DATEADD(mm,DATEDIFF(mm,0,SomeDate),0))',
@from='##JBMTest',
@WHERE='SomeDate>=~Jan  1 2008 12:00AM~ AND SomeDate<~Jul  1 2008 12:00AM~',
@rowtotalsposition='end',@racheck='y',@pformat='_pvtcols_',@shell='n',@translabel='Summary',
-- Display min and max sum for each Someletters along with pivot (date) it occurred.
-- The min and max are displayed in separate rows. Default is same row for all rowfunctions.
@rowfunctions='min(SumMony) & max(SumMony)',@rowfunctionslabel='Min/Max',@displayrowfunctions='m',
-- Running sum of pivot columns for each row from left to right. The pivot sum is followed by the 
-- run in each cell.
@colruns='SumMony',
-- The percentage of the pivot sum/[row total] displayed in a separate row.
@cpercents='[% row] %only' -- a different transform alias to force a separate row.
                           -- We could display in same row as sum (and column runs).


Someletters2 Summary Min/Max                 Jan 2008            Feb 2008            Mar 2008             Apr 2008             May 2008             Jun 2008             Totals
------------ ------- ----------------------- ------------------- ------------------- -------------------- -------------------- -------------------- -------------------- ----------
AA           SumMony min(192.13,Jun 2008)    685.67/685.67       763.64/1449.31      656.13/2105.44       575.93/2681.37       879.13/3560.50       192.13/3752.63       3752.63
                     max(879.13,May 2008)                                                                                                                                 
             % row                           18.3%               20.3%               17.5%                15.3%                23.4%                5.1%                 -
AB           SumMony min(280.20,Mar 2008)    927.06/927.06       928.98/1856.04      280.20/2136.24       632.43/2768.67       560.99/3329.66       785.50/4115.16       4115.16
                     max(928.98,Feb 2008)                                                                                                                                 
             % row                           22.5%               22.6%               6.8%                 15.4%                13.6%                19.1%                -
AC           SumMony min(187.48,May 2008)    791.09/791.09       555.18/1346.27      916.71/2262.98       273.23/2536.21       187.48/2723.69       508.31/3232.00       3232.00
                     max(916.71,Mar 2008)                                                                                                                                 
             % row                           24.5%               17.2%               28.4%                8.5%                 5.8%                 15.7%                -
AD           SumMony min(250.04,Jan 2008)    250.04/250.04       341.58/591.62       426.53/1018.15       645.56/1663.71       670.13/2333.84       422.86/2756.70       2756.70
                     max(670.13,May 2008)                                                                                                                                 
             % row                           9.1%                12.4%               15.5%                23.4%                24.3%                15.3%                -
AE           SumMony min(295.33,Mar 2008)    809.14/809.14       487.21/1296.35      295.33/1591.68       625.92/2217.60       716.12/2933.72       527.19/3460.91       3460.91
                     max(809.14,Jan 2008)                                                                                                                                 
             % row                           23.4%               14.1%               8.5%                 18.1%                20.7%                15.2%                -
AF           SumMony min(406.49,May 2008)    788.30/788.30       415.40/1203.70      605.56/1809.26       613.81/2423.07       406.49/2829.56       520.40/3349.96       3349.96
                     max(788.30,Jan 2008)                                                                                                                                 
.
.
ZY           SumMony min(326.17,Mar 2008)    776.32/776.32       682.98/1459.30      326.17/1785.47       677.69/2463.16       546.87/3010.03       926.54/3936.57       3936.57
                     max(926.54,Jun 2008)                                                                                                                                 
             % row                           19.7%               17.3%               8.3%                 17.2%                13.9%                23.5%                -
ZZ           SumMony min(277.93,Mar 2008)    532.75/532.75       500.92/1033.67      277.93/1311.60       636.40/1948.00       607.05/2555.05       553.43/3108.48       3108.48
                     max(636.40,Apr 2008)                                                                                                                                 
             % row                           17.1%               16.1%               8.9%                 20.5%                19.5%                17.8%                -
Totals       SumMony min(383211.70,Feb 2008) 433997.25/433997.25 383211.70/817208.95 411913.12/1229122.07 411878.29/1641000.36 425431.07/2066431.43 409809.47/2476240.90 2476240.90
                     max(433997.25,Jan 2008)                                                                                                                              
             % row                           17.5%               15.5%               16.6%                16.6%                17.2%                16.5%                -

Linq to Sql as an opportunity to sql users

2008-11-25T22:02:00.000-08:00

Instead of a dead end, Linq presents opportunities for sql experts.

My comment orginally appeared in the www.sqlblog.com post:

'Continuing Industry Consolidation and Other Trends.' 1/16/08
by Kevin Kline
http://tinyurl.com/5opxlz

I think the Sun acquisition of MySql was a checkmate move. It was Oracle
that attempted to slow down MySql with their acquisition of the InnoDB
transaction engine. Now they have been paid back in spades. Now it's Sun
vs. MS in a new competition for the minds and hearts of developers. The
broad view is LINQ/net and Sql Server vs. java and MySql. This is not
about databases per se but a new war based on the object/entity model
inspired by LINQ. I don't see IBM and Oracle in this war. They will have
to be content to battle amongst themselves in the old database wars.
(I'll come back to LINQ:)

As for 'Checking out the Competition' of course I too applaud Linchi.
But honestly we're first 'now' recognizing the advantages of eyes wide
open?:) This attitude should be a given for a professional. Perhaps
better late than never, perhaps an airing of cheerleading vs.reality
checking:) For most ansi features that MS has pre-annouced all one has
to do to study them is read about them in Oracle or DB2 documentation
(ie. OVER). And as Linchi pointed out it often goes the opposite
direction for other types of features. This attitude contributes to the
great divides that are common in the industry. 

And now we come to the 'dead-end routes' like LINQ. I take the opposite
view you do. There's a compelling case to be made (I've commented on
this matter on this site) that if there is a deadweight MS sees sql as
it. LINQ is not just a piece of new technology, it's not just a local
variable, it's a global one. LINQ is both an affirmation of the runtime
environment for 'all' application development using an object model and
a rejection of the sql interface. MS can live with the physical file
structure (the idea of relational data, rows and columns) but they don't
want to live with the sql interface for application development. MS
explains this global move in terms of solving the historic impedance
mismatch between two completely different environments. And they have
picked their winner and at the same time the loser. The rows and columns
abstraction now ends at physical storage. The object abstraction and
LINQ will take up everything else. Sql server is now something quite
different than it used to be. Almost all developmental changes in server
will be based on servicing objects and quite probably at the expense of
features associated with a furtherance of the relational model. Look at
all the work on partitioned views in S2008. This lies at the heart of
how LINQ will translate entity updates. LINQ is still in its enfancy. I
would expect it to appear to many just like sql did when it was intially
introduced in the early eighties. It will take time to get the matured
version. What is truely ironic is I see no real argument in the sql
community that LINQ represents a great opportunity for sql developers.
MS is inventing a declarative language in the spirit of sql. Don't
people see an opportunity to jump in and at least influence the design
of the language? Or get involved in the LINQ translation/optimizations
to sql. Over time as MS integrates LINQ deeper into server (returning
entities) I can assure you the current translations will change:) Sql
was most certainly not an implementation of the relational model. So sql
folks shouldn't get hung up over this. The relational model would
require the same strong typed runtime as net but MS is certainly not
going there. But they are going to a place that sql skills can be used.
And now Sun is going to go along with them. It's actually a small world
if your eyes are open:)

Linq to Sql should be important to sql users

2008-11-25T20:52:00.000-08:00

My plea for the sql server community to look closely at Linq.

My comment orginally appeared in the www.sqlblog.com post:

'PASS presentation: What would you like to hear about?' 2/23/08
by James Luetkehoelter
http://tinyurl.com/5lmj4m

Your eccentric. Such people tend to be really bright. So that's kewl.
Your passionate, so your highly motivated. That's very good. If you're
'over the top' that means your not afraid of taking risks! That's best
of all. If all this is true you can take on the burden of tackling LINQ
and the entity framework. Now I'm not talking about the 'how' of it. I'm
sure there will be plenty of people presenting point and click slides.
What I am talking about is the 'why' and 'what' of it. LINQ/EFM dwarfs
everything else in terms of importance in S2008. It's a reflection of a
movement in a different direction from what everyone is used to. It's
also a reflection of a change in attitude. When I look for sql folks
tackling the subject what do I find? Frighteningly little or nothing!
Now let me say if you're willing to make the case that sql folks can
simply ignore the subject, have at it:) If you even want to make the
case that it simply represents an isolated piece of technology go for
it. Some sql folk are waxing nostalgic about past 4GL/xbase languages
when discussing LINQ. So it may be that many sql folks think it's quite
the same idea and wonder why it's structure is different and possibly
inferior to what was. Well LINQ is different, it's comparing apples and
oranges, and it can't possibly be the same. But how can folks really get
a grasp of such a comparison if they don't really understand the 'what'
of LINQ. Trying to explain 'what' it is isn't easy especially to those
sql folks who are not familiar with net or any other contemporary
strongly typed object oriented runtime environment. I think MS is
finding out that it's a challenge. Even trickier is to explain the 'why'
of it. The motivation for it. Surely it didn't come out of thin air:)
And the 'why' involves taking risks. You may frighten or alienate
people:) The 'why' cuts to the heart of what most folks believe in. LINQ
is a departure. It will have significant implications for what
development is and will be like in the future. It will take a very
unique person to put all these ideas together so they'll be really
understood. Interestingly, all the answers about LINQ are right on the
MS website. All that's required is to dig around and back thru the years
to find them. With over eight years of development this shouldn't be
surprising:) But how many sql folks have bothered to do this homework?
From the looks of it very, very few. Presenting concepts is much harder
than presenting code. It takes a very special kind of guy to connect
the dots here:)
I'd be happy to share with you an extensive MS list of articles/notes
thru the years about the subject.

Linq to Sql vs. older 4GL attempts

2008-11-25T19:38:00.000-08:00

Some thoughts on comparing Linq to older reporting languages.

My comment orginally appeared in the www.sqlblog.com post:

'LINQ - Lessons that could have been learned from languages like Progress' 2/25/08
by Greg Low 
http://tinyurl.com/56powf

>it's still very much an add-on rather than being fully baked into the language.

I remain somewhat perplexed by just what you mean by this. By definition
LINQ is 'burned in'. This is the achievement that MS has accomplished.
To imply that it's not fully baked in is like saying a woman is somewhat
pregnant. It is or it isn't, you are or you are not:) Either a
table/entity or projection can be represented as a variable or it
cannot. That all these things 'can' be a variable of some type
referenced in the language is the whole point of it no? Your use of
'add-on' seems to imply that LINQ is something external to the language
much like an external 4GL like Progress. I don't think this can be
further from the truth. In your interview with Pablo Castro he referred
to Progress as an example of a language having 'direct line of sight
from language to database'. Wasn't he struggling here to convey the idea
to sql folks of the fundamentally different natures of the two? To bring
the 4GL idea into contemporary languages one shouldn't expect they are
going to be similar. And you seem to be implying that LINQ is not as
'natural' as Progress. How could it be? If you first have to define a
query variable (table) you certainly can't expect to start your variable
with a Select/Find. You define and then perform an action right? In
t-sql 'Select @MyNumber' only makes sense if your first declare
@MyNumber. Is LINQ any different? And in the sense that 'declare
@MyNumber int' is burned into t-sql, isn't 'var customers = from c in
db.Customers' burned into C#?

I do think sql users should listen to your interview with Pablo. It is
proving difficult for MS folks to convey just what they have done to
those outside of net. What is worse, misunderstanding it or ignoring it?:)
Shouldn't sql folks fully understand why MS thinks it's such an
improvement over sql? So far I think either they don't or simply don't
care.

Linq to Sql: the 'what' and 'why' of it

2008-11-25T18:40:00.000-08:00

The following comments concern 'what' is Linq (to Sql)/EF and the 'why'
of it, what motivated MS to develop it. What does MS mean by 'beyond 
'relational'? I also explore in what ways Linq, sql and the relational
model are related to each other. How these technologies are connected
to each other is quite a fascinating picture:)
My following 5 comments orginally appeared on www.sqlblog.com
in the post:

'Beyond Relational ???' 10/29/07
by Paul Nielsen 
http://tinyurl.com/686z6h

Comment #1

There is the association of relational to mathemetics (set theory). So
people criticize sql based on this point of view. Sql allows duplicates
rows, doesn't require a table to have a key, dependencies based on
ordinal position, is a poorly designed language etc. etc. These things
really are critical but the real problem is the prevailing idea that
relational is just a question of mathemetics. If it's just mathemetics
then allowing duplicate rows is perceived as 'close enough'. All the
objections from the set theory point view are not perceived as
compelling enough to really question the validity of sql. IMO the real
holes of sql have nothing to do with mathemetics. Rather it's the
foundation, the computer science if you will, that set theory and
relational algebra are embedded in. This point of view is unfortunately
not prevalent in IT. What the hell do I mean by the computer science of
the relational model? Well first, the set theory that relational depends
on is not some special kind of set theory. There is only one set theory.
In the same way there is only one computer science, there is no special
kind of computer science. But sql has invented such a special computer
science and this is the biggest flaw. What am I talking about?:)
Consider this, here is a table variable:

DECLARE @MyTableVar table(
   EmpID int NOT NULL primary key,
   OldVacationHours int,
   NewVacationHours int,
   ModifiedDate datetime);

Here is a server table:

create MyTable
      EmpID int NOT NULL primary,
      OldVacationHours int,
      NewVacationHours int,
      ModifiedDate datetime);
 
Here's the key question. If @MyTableVar really is a variable then what
is MyTable? In other words, @MyTableVar is to variable as MyTable is to
?????. If MyTable is persisted in the database what is it persisted as?
What computer science term describes it? Well whatever the hell it is (a
constant?) it certainly isn't a variable. And if it isn't a variable
then end of ballgame, end of relational model. And what of @MyTableVar?
Bol says 'A table variable behaves like a local variable.' and at the
same time says 'Assignment operation between table variables is not
supported.'. When is a door not a door?..when it's ajar:) Who the hell
ever heard of a variable that doesn't support assignment? Who ever heard
of a variable that doesn't support comparison? No one. Whatever
@MyTableVar really is it sure as hell ain't a variable. In a relational
db I should be able to assign the table @MyTableVar, all its rows, to
MyTable: 
      
MyTable=@MyTableVar

And I should be able to compare them.

if MyTable=@MyTableVar
 then print 'All rows in MyTable are in @MyTableVar and all rows in   
  @MyTableVar are in MyTable'
   else print 'Nope they're not equal'      
   
A relational db demands a table be a variable just like an integer
variable. Sql simply does not support basic computer science for tables.
Whatever a table is in sql it doesn't have a 'type' because computer
science is computer science and a variable must be typed. The only way
sql can recognize a table is by its name, not its type. This is why sql
doesn't support relational division and why dynamic sql must be used so
much. A table as a variable is a completely different animal than a
table in sql. This is why the expressive power of a relational db is
orders of magnitude greater than an sql db. Sql views and constraints
are redefined relationally. The 'types' in Dates work:   
Databases, Types and the Relational Model, The Third Manifesto' 2007
is about the central importance of variables of a particular type (a
table as one of many types) in a relational db. What a table as a
variable means and its significance. It is really a basic computer
science book. Ripping out the mathematics of relational theory (at least
trying to copy it), ie. the syntax to join, union tables, without the
computer science of relational has done all the damage. MS can't change
sql server because they are caught in an crazy computer science. The
difference in computer science between sql and net is the impedance
mismatch they're trying address. But I'm afraid they still don't get the
idea of a table as a variable. This is different than a table as a
class. The anonymous type in Linq is distinctly different
than a table type. So MS is doing the same thing IBM did forty years ago
with the sql System-R. The damage is the difference between a pickup
game in a playground and organized sports. You can draw up plays in the
dirt but they don't quite work the same as those run in a stadium. We're
still doing application development in the playground. Sometimes it
works, sometimes it doesn't but we're not basing it on the science of
any model. Sql is not a model of anything, it's an invention all its own. 
Close enough is only for horsehoes:) Maybe my blog will make more sense now:)

Comment #2

Wherever you see the word 'relational' just substitute 'fog'. As in fog of war:) 
> But when you have guys like Don Chamberlin (co-inventor of SQL and  
> co-designer of XQuery) on your staff, I guess you can afford to 
> boast your XML prowess.
He is revered in the sql world and reviled in the relational one. He was
a lead designer of System-R, the prototype of all sql database systems.
Those guys created a query language based on Codds description of basic
relational operators like projection, union and join. But they did NOT
implement the relational model Codd described. They just ripped out
these constructs without regard for their meaningfulness in the entire
relational model. So what you have today is nothing like the relational
model as it was envisioned. (IT successfully marginalizes the huge
difference and those that point it out:) And now comes 'beyond
relational'. What does this phrase really mean to MS? They are more than
willing to tell us. Aside from Jim Grays article/presentation, everyone
should read the articles on this site, the 'Comega language':
http://research.microsoft.com/Comega/
Especially this article:
'Unifying Tables, Objects and Documents'
http://tinyurl.com/yq7c4f

Here you'll find history repeating itself. MS, just like IBM did with
System-R, has extracted relational operators out of the relational model
and put them in an imperative object environment without any regard to
relational theory. The great irony is that the extensions that MS added
to net to realize projections of columns and xml within net is the
foundation for a true relational model! But the compiler generated
anonymous type of Linq while a variable is a different beast than the
explicit variable that is a table type in the relational model. It's the
relational variable that supports assignment and comparison as opposed
to the Linq variable that's no where near as smart:) But each supports a
'variable' which is a major step up from sql. Had MS any idea of the
friggin true relational model they would make a different kind of
history. Talk about dumbing down. Talk about of only academic interest.
Talk about relational fog (I should add that Alphora (Dataphor)
recognized the ability of the object imperative environment to support
the D relational language and implemented it. And it works:) 
Here is what Anders Hejlsberg, MS VS guru, and now the head of database
technology has to say about the disconnect:
InfoWorld
Interview of Microsoft Distinguished Engineer Anders Hejlsberg
'Microsoft's Hejlsberg touts .Net, C-Omega technologies'
June 10, 2005
http://www.infoworld.com/article/05/06/10/HNhejlsberg_1.html

"So what we're looking at is really trying to much more deeply integrate
the capabilities of query languages and data into the C# programming
language. And I don't specifically mean SQL, and I emphatically don't
mean just take SQL and slap it into C# and have SQL in there. But rather
try to understand what is it expressively that you can do in SQL and add
those same capabilities to C#."

Anders Hejlsberg is microsofts version of Don Chamberlin at IBM. So what
they have done is replace one flawed implementation of sql with another.
And this is how they achieve efficiency in application development. Now
that is unfriggin believable:) Well there's no free lunches. And I await
to be enlightened on just how this environment will replace the concept
of the logical relational model in solving business problems. I would
say the real meaning of beyond relational is sideways.

Comment #3

Consider the MS whitepaper:
'Microsoft SQL Server 2008 and Microsoft Data Platform Development'
http://www.microsoft.com/sql/techinfo/whitepapers/sql_2008_dp.mspx

Does anyone find it the least bit odd that an sql server technical article
is all about VS, LINQ and the entity framework? At the expense of the logical
relational model and the sql language.

What MS means by 'beyond relational' is 'forget relational already':)
Looking at sql server as if it was somehow an embodiment of relational
theory is every bit a form of dumbing down as some silly utterance by
some poor nitwit at MS. There never was and never will be any 'intent'
by MS to offer a 'relational' database. Sql servers only intent now is
to be responsive to its biggest customer, visual studio. And that team
is as knowledgeable in relational databases as the server team. Not. Why
does the community still view sql server thru an imaginary lense? Did
you ever hear of somewhat pregnant? If you open the dumbing down door be
prepared to greet all those who come thru:)

Comment #4

There is no longer a great divide, a debate, an impedance mismatch. MS 
has issued their own Emancipation Proclamation. And as a result they no
longer support the relation model as it is know to developers today.  
'A Call to Arms'
by Jim Gray, Microsoft
Mark Compton, Consultant
April 2005
http://www.acmqueue.org/modules.php?name=Content&pa=showpage&pid=293

This paper is an invitation to embrace a new model. It's just as much
'A Farewell to Arms', an emancipation from the relational model which 
they are leaving behind. What does sql server look like in this new model?

'Interview with David Campbell'
General Manager of Strategy, Infrastructure and Architecture of Microsoft SQL Server.
May 14, 2007
http://tinyurl.com/6maseb
Campbell says:
"I believe the next major advance in Information Technology will come 
from addressing the gap between people and information."

That gap is the relational logical model itself.

Campbell continues:
"The focus will move to the data itself rather than on the machinery
used to manipulate it. We'll be less concerned with the plumbing and
more concerned with data quality, data protection, and information
production."
"Most of the data services provided by SQL Server will be driven from
a common data model. Whether you're creating a report, building an
information cube, or integrating data from another system, you will be
able to start from a common model of the key data entities such as
'customer', 'order', or 'prospect'."
"Finally, fewer and fewer people will miss, (or remember), the 'open
databases' sp_configure option..."

The class replaces the table as the basic unit of work.  VS replaces
QA/SSMS as the interface for application development. There is no
concept of relational anything in this object world. Sql constructs are
independent of application development. The language of the relational
model is replaced with the language of entities. There is no concept of
a dba.
MS is no longer in the database wars as we know it. They are trading 3rd
place in that world for 1st place in another. And they now have the
freedom to talk about this new world. It just sounds silly to those who
have not left the old one.
Ironically some were hoping for a new sub-language to further AD. Perhaps
the lesson here is to be careful of what you wish for. I too was hoping
they'd enter a new world but not the one they have chosen.

Comment #5

> should we be concerned staying in the DB world long with the fear 
> that we become obsolete one day?
Although I'm not an expert I can understand where you're coming from. It
would be nice to get a clear and concise answer to where MS is going and
what you should do about it. But there is no Oracle when it comes to MS.
There is no one position paper, no one person that clearly lays out
their five year plan and what it means to you. The experts here have
enormous importance and influence in the db community. But they also
have an enormous investment. How far can they be reasonably expected to
go without putting themselves in an awkward position should they take a
position that is not currently in line with company thinking? In the end
it's a question of connecting the dots. You get a dot here a dot there.
You have to do your homework. Study what they say and write and study
what they offer. Sql server pros shouldn't neglect what's going on in VS
and it's impact. If you study the company and the various technologies
enough you should be able to draw your own picture. Think of it as the
MS X-files:)

Listing Missing Table Item

2008-11-23T17:42:00.000-08:00

The following example is based on the post:

microsoft.public.sqlserver.programming
Friday, November 21, 2008 11:56 AM
"T-SQL Comparing a String of Items to a Table of Items; Listing Missing Table Items"
http://tinyurl.com/5dvc6o

Here's what the future of application development will hopefully be like
using a relational system. To program relationally you not only have to
think in "terms of sets" (the declarative thing:) but you have to think
"in terms of type" (the relation thing). Sql essentially doesn't concern
itself with variables and types. They are essentially viewed as foreign
thingies found in net. But in a relational system they are every bit as 
significant as understanding how to write an outer join. The following
example illustrates the relational approach using the Dataphor system
(which can play nicely with Sql Server ☺)

create session table Products {
  product_key:Integer,
  part_no: String,
  options: list(String), //options is defined as a list type, not a string.
  price: Money,
  key{product_key}
  };

All tables, like numbers, are variables with an explicit 'table' type, the
column names and their datatypes. And like integers they can be 'assigned'
values. Tables are composed of 'rows'. For each row options is input as a
'list' of type string ({'<string>','<string>','<string>'..}).

Products:=
table {
      row{11 product_key, 'ASP-20J' part_no, {'Wiz Bang', 'Hopla Enhancer'} options, $10.00 price},
      row{12, 'ASP-20R', {'Widget Holder','Wiz Bang',  'Hopla Enhancer'}, $12.00}
      }; 

create session table Options {
  option_key: Integer,
  product_key: Integer, 
  option: String,
  key{option_key},
  reference OptiontoProduct {product_key} references Products {product_key}
  };

Options:=
table {
       row{5 option_key, 11 product_key, 'Wiz Bang' option},
       row{6, 11, 'Hopla Enhancer'},
       row{7, 12, 'Wiz Bang'},
       row{8, 12, 'Hopla Enhancer'}
      };
       
Here are easy ways to find the missing product from the Options table.

Using row in table. First a table of numbers (Numbers) with a single
column N (from 0-10K) is used to create a row for each element in
the options list bringing along the other Products columns. The row
is constructed from this table to see if it's not in options. 

select 
   (
    Products 
       times  //Like an sql cross join. 
         Numbers
           where N<options.Count()
              //create a row for each element in the list.
             {product_key,options[N] option} 
                                           
    )     
       where not (
                  row{product_key product_key,option option}
                  in
                 (Options {product_key,option})
                 ) ;
                
Using relational division. Because tables (table expressions) are
variables one table can be tested to see if it's contained in another.

select 
   (
    Products times Numbers
         where N<options.Count()
          {product_key,options[N] option}
    )     
       where not (  
               table{row{product_key product_key,option option}}
                <=
               (Options {product_key,option})
               ) ;                 
               
Using a left join. Test whether the table on the right has a matching row.

select 
   (
    Products times Numbers
         where N<options.Count()
          {product_key,options[N] option}
    )     
      left join Options
                include rowexists  //A special Boolean column for whether there is a match.
                  where not rowexists 
                    {product_key,option} ;
      
Instead of inputting a list directly, a delimited string can be converted to
a list when input.

Products:=
table{
      row{11 product_key, 'ASP-20J' part_no, ('Wiz Bang, Hopla Enhancer '.Split()) options, $10.00 price},
      row{12, 'ASP-20R', (' Widget Holder ,Wiz Bang,  Hopla Enhancer'.Split()), $12.00}
      }; 
Options:=
table{
      row{5 option_key, 11 product_key, 'Wiz Bang' option},
      row{6, 11, 'Hopla Enhancer'},
      row{7, 12, 'Wiz Bang'},
      row{8, 12, 'Hopla Enhancer'}
     };

The queries are the same as above except for trimming each element of the list.

select 
   (
    Products times Numbers
         where N<options.Count()
          {product_key,options[N].Trim() option}
    )     
     where not (
                row{product_key product_key,option option}
                in
               (Options {product_key,option})
                ) ;
                
select 
   (
    Products times Numbers
         where N<options.Count()
          {product_key,options[N].Trim() option}
    )     
     where not (  
               table{row{product_key product_key,option option}}
                <=
               (Options {product_key,option})
               ) ;                 
               
select 
   (
    Products times Numbers
         where N<options.Count()
          {product_key,options[N].Trim() option}
    )     
      left join Options
                include rowexists 
                  where not rowexists 
                    {product_key,option};
                    
All queries produce a result of:

product_key option        
----------- ------------- 
12          Widget Holder 

Types eliminate violations of normal forms much like education eliminates ignorance ☺

Dataphor is a RAD tool, R(elational) A(ccelerator) D(evelopment).
Visit dataphor at:
www.dataphor.org

S(ecure) Q(uery) L(anguage)?

2008-11-14T00:59:00.000-08:00

Concerning the thread:
microsoft.public.sqlserver.programming
Nov 10, 2008
"cascading deletes"
http://tinyurl.com/6nwjmd

For some reason I couldn't get my reply to post thru OE (it got thru
via google though). Perhaps there's an MS filter for metaphors ☺
In any event any mature adult should be able to handle it. So here's
my reply with a touch of creative writing ☺

'Most women believe men think with their tool. And it's just as true in
IT. Users model business problems in terms of the abilities of their
db. The idea that modeling exists independent of ones db is a myth.
It's not a question of seepage but of flooding. Modeling business
problems in terms of the available sql server constructs is messy
precisely because their immature and superficial to the task. The
result is you turn away from the db and towards the do-it-myself
model. You roll around in your own layer because you can't get layered
by the db. It's ridiculous to write a join procedurally but when it
comes to modeling it's perfectly acceptable to roll your own. Because
the model equivalent of the join is so lacking and messy. The genie
isn't going back in the sql server bottle. It's simply to far gone. 
That's why I advocate Dataphor. There the genie is in the join as well
as the modeling. Use Dataphor and put your tool back where your head
and shoulders are. You can still use sql server. But you aren't going
to get tooled by it :)'

www.dataphor.org
www.beyondsql.blogspot.com

Geoff Schaller wrote:
> Andre.

> I vote with Hugo here. We manage everything from code, not from TSQL in
> SSMS or via some other mechanism so we generally have to code everything
> (and that is not as difficult or as expansive as it sounds). Whilst
> cascading referential integrity is "nice" from a simplicity point of
> view, we've found that the act of deleting something (say an invoice) is
> almost never a simple thing. There is reversal of stock levels,
> rebalancing totals and if others are running reports when you thought
> you wanted to do the delete, it gets messy.

> The other thing is that we quite often have to delete the child entries
> individually or prevent the parent from being deleted because a child or
> two cannot be. Writing all that logic into a trigger and enforcing the
> rollback is quite complex. I find code an easier way to manage and
> maintain this.

To add insult to injury my reply to a post on SQLServerCentral was hacked (edited).

SQLServerCentral 
11/10/2008
'Arrays in Stroed Prcoedure'
http://tinyurl.com/5th5n4

My reply, as shown there under the name rog pike, was edited to read:  

'An array is a 'type', something the archaic computer science of sql knows
nothing about. You have to move to a 'real' relational system to find a
list/array type. You'll find such adult computer science in Dataphor.'

My orginal reply was a follows:

'Arrays are in sql server in the same sense as having sex by yourself which may account for
the shortsightedness of so many sql mavens. An array is a 'type', something the archaic computer
science of sql knows nothing about. You have to move to a 'real' relational system to find
a list/array type. You'll find such adult computer science in Dataphor.'

Is the site for mature adults or for the whole family? ☺ Just how much protection
does sql and its users need? Is this a security or, better yet, an insecurity problem? ☺

Finally, I'll repeat here what I posted in the above thread:

'Apparently someone complained/reported something I wrote as being objectionable. They
got their wish as it was magically extracted from the text. What was yanked, besides
my chain, was a metaphor, albeit a vivid one, to drive a salient point home. Now I
write for adults, I don't do child-speak very well. Nor do I have a predilection 
to only write drone-on-speak. So, if I can, I won't hesitate to use an adult metaphor
to amplify a point in an industry that is usually tone deaf. God forbid IT encourage
ability in something other than code or pixels. So if you are an adult, with a surname
other than anonymous, please explain just what you found R or X rated. Mature adults
usually confront conflicts thru the front door not the back one.'

Sorting a delimited string numerically

2008-09-08T02:54:00.000-07:00

Sorting a delimited string by its numerical string parts.

This is a possible solution to the problem presented in the article:   

'T-SQL Sorting Challenge'
By: Itzik Ben-Gan 
http://www.sqlmag.com/Article/ArticleID/100156/100156.html

"You are given a table called t1 with a character string column called val. 
Each string in the val column holds a dot separated list of integers. Your
task is to write a T-SQL solution that sorts the strings based on the integer
values constituting the string segments. Note that the number of integers in
each string may vary, and is only limited by the column type VARCHAR(500).
Extra points if your solution will also support negative integers."

So the problem is how to construct a sort expression that represents the
positive and negative integers of the ids.

This solution uses Dataphor with the data being stored in Sql Server 2005.

Sample data

The data is populated thru dataphor and persisted in the sql server northwind
db. The table t1 in the article is table IZ here. The article uses an sql
identity for the primary key id, here it is explicitly declared.

create table IZ
{
 id:Integer,
 val:String tags {Storage.Length='75'},
 key{id}
};
//Positive value strings.
insert row{1 id,'100' val} into IZ;
insert row{2 id,'7.4.250' val} into IZ;
insert row{3 id,'22.40.5.60.4.100.300.478.19710212' val} into IZ;
insert row{4 id,'22.40.5.60.4.99.300.478.19710212' val} into IZ;
insert row{5 id,'22.40.5.60.4.99.300.478.9999999' val} into IZ;
insert row{6 id,'10.30.40.50.20.30.40' val} into IZ;
insert row{7 id,'7.4.250' val} into IZ;
//Add negative values.
insert row{8 id,'-1' val} into IZ;
insert row{9 id,'-2' val} into IZ;
insert row{10 id,'-11' val} into IZ;
insert row{11 id,'-22' val} into IZ;
insert row{12 id,'-123' val} into IZ;
insert row{13 id,'-321' val} into IZ;
insert row{14 id,'22.40.5.60.4.-100.300.478.19710212' val} into IZ;
insert row{15 id,'22.40.5.60.4.-99.300.478.19710212' val} into IZ;

Go directly to dataphor solution
Go directly to a solution using dataphor and sql server t-sql
Go directly to a solution using the Rac utility on sql server 2000
Go directly to a solution using the Rac utility on sql server 2005
(
 Rac is a system of stored procedures and functions for sql server designed
 to simplify solving various data manipulation problems including dynamic
 crosstabs, complex running sums and ranking, string manipulations etc.
)

Stepping thru the logic of the solution

Use the dataphor Split operator to split the val string for each id into
individual strings starting at Index 1 and going to the number of parts
delimited by the period ('.'). Note that the string is converted to an
integer. So we're dealing with numbers and not strings.

select
 (IZ add{val.Split({'.'}) StrList,val.Split({'.'}).Count() StrListCnt})
    times                //times is equivalent to an sql CROSS JOIN.
    (Numbers where N<10) //A table with a single column N, an integer from 0 to 800.
      where N<StrListCnt
       {id,N+1 Index,ToInteger(StrList[N]) StrNum}
         order by {id,Index} ;      
         
id Index StrNum   
-- ----- -------- 
1  1     100  <- id 1 has only a single value.
2  1     7    <- id 2 has 3 values.
2  2     4        
2  3     250      
3  1     22   <- id 3 has 9 values.
3  2     40       
3  3     5        
3  4     60       
3  5     4        
3  6     100      
3  7     300      
3  8     478      
3  9     19710212     
4  1     22  
.  .     .

Now lets look at the same data but within each Index and within each Index
ordered by the string as a number (StrNum). Remember all ordering is ascending.

select
 (IZ add{val.Split({'.'}) StrList,val.Split({'.'}).Count() StrListCnt})
    times
    (Numbers where N<10)
      where N<StrListCnt
       {id,N+1 Index,ToInteger(StrList[N]) StrNum}
         order by {Index,StrNum};
         
id Index StrNum   
-- ----- -------- 
13 1     -321  <- id 13 has lowest overall number for 1st string part (Index 1).
12 1     -123     
11 1     -22      
10 1     -11      
9  1     -2       
8  1     -1       
2  1     7        
7  1     7        
6  1     10       
3  1     22    <- ids 3,4,5,14,15 have the same value (22) for Index 1.   
4  1     22      
5  1     22       
14 1     22       
15 1     22       
1  1     100   <- id 1 has highest overall number for 1st string part (Index 1).  
.  .     .
3  8     478   <- Index 8 has the same string value for the 5 ids that have an 8th string part.
4  8     478      
5  8     478      
14 8     478      
15 8     478      
5  9     9999999  
3  9     19710212 
4  9     19710212 
14 9     19710212 
15 9     19710212 

The strings as integers are nicely sorted within each Index over the ids. 
How can we represent the same ordering within each Index independent of
the positive and negative numbers (and strings that indicate positive and
negative numbers)? What about with a ranking. So lets rank the numbers
within each Index. The combination of the dataphor ToTable, ToList and
cursor operators will generate a rank (a column named sequence) that 
follows the specified cursor ordering. We order the cursor by Index,StrNum
to get an ascending rank within each Index based on the StrNum values. 
The rank is column RowNum.

select
ToTable(
        ToList(
               cursor(
                       (
                        (IZ add{val.Split({'.'}) StrList,val.Split({'.'}).Count() StrListCnt})
                          times
                           (Numbers where N<10)
                             where N<StrListCnt
                              {id,N+1 Index,ToInteger(StrList[N]) StrNum}
                        )   
                                order by {Index,StrNum}))) 
                                 {Index,id,StrNum,sequence+1 RowNum}
                                   order by {Index,StrNum};

Index id StrNum   RowNum 
----- -- -------- ------ 
1     13 -321     1  <- id 13 has lowest value (-321) so id 1 gets lowest rank (1) within Index 1.    
1     12 -123     2      
1     11 -22      3      
1     10 -11      4      
1     9  -2       5      
1     8  -1       6      
1     2  7        7  <- ids 2 and 7 get a different rank for the duplicate values of 7 :(    
1     7  7        8      
1     6  10       9      
1     3  22       10 <- all 5 ids get a different rank for the duplicate values of 22 :(  
1     4  22       11     
1     5  22       12     
1     14 22       13     
1     15 22       14     
1     1  100      15 <- id 1 has highest value (100) so id 1 gets highest rank (15) within Index 1.
.     .  .        .
8     3  478      56     
8     4  478      57     
8     5  478      58     
8     14 478      59     
8     15 478      60     
9     5  9999999  61     
9     3  19710212 62 <- all 4 ids get a different rank for the duplicate values of 19710212 :(    
9     4  19710212 63     
9     14 19710212 64     
9     15 19710212 65     

(
 Note that this rank is equilvant to the sql ranking function ROW_NUMER().
 The rank could be obtained in sql using:
 ROW_NUMBER()OVER(ORDER BY [Index],StrNum) AS RowNum
 But the rank we want is really based on the sql RANK() function which
 accounts for duplicate/ties (of StrNum) by giving them the same rank. 
 Therefore it's necessary in dataphor to use a join to append the correct
 ranks to the table. In sql the join isn't necessary, RANK() can be used
 directly on the table:ie.
 RANK()OVER(ORDER BY [Index],StrNum) AS Rank
 (See t-sql solution)
 For more on dataphor and sql ranking see:
 'The Sql ranking OVERture' 
 http://beyondsql.blogspot.com/2008/04/sql-ranking-overture.html
)

If you haven't guessed it by now ☺ the idea is to create a string for each
id based on the ranks which will be used to order the ids. But we have a 
problem because for duplicate values of a number we're getting different
ranks. We want the 'same' rank for duplicate values since the same integer
cannot be used to distinguish among the ids. 

We can remedy the different ranks for duplicate values by simply choosing
the minimum rank (RowNum) for the value and assigning this rank to all ids.
Also note that the ranks continue to ascend over the Indexs. This is ok
because any numbers representing the ranks are ok if they correctly maintain
the ordering of integer strings values within the Index.

select
ToTable(
        ToList(
               cursor(
                       (
                        (IZ add{val.Split({'.'}) StrList,val.Split({'.'}).Count() StrListCnt})
                          times
                           (Numbers where N<10)
                             where N<StrListCnt
                              {id,N+1 Index,ToInteger(StrList[N]) StrNum}
                        )   
                                order by {Index,StrNum}))) 
                                 {Index,id,StrNum,sequence+1 RowNum}
                                   group by {Index,StrNum} add{Min(RowNum) Rank} 
                                     order by {Index,StrNum};

Index StrNum   Rank 
----- -------- ---- 
1     -321     1    
1     -123     2    
1     -22      3    
1     -11      4    
1     -2       5    
1     -1       6    
1     7        7  <- a rank of 7 can be assigned to the two ids (2,7) with a value of 7 for Index 1.  
1     10       9    
1     22       10 <- a rank of 10 can be assigned to all ids with a value of 22 for Index 1. 
1     100      15   
.     .        .
8     478      56   
9     9999999  61   
9     19710212 62 <- a rank of 62 can be assigned to all ids with a value of 19710212 for Index 9.

What we now have is a table of unique Index/StrNum combinations with a
unique rank for each combination. It's only necessary to join this table
to the table of split strings (IZ) for all ids by Index and StrNum to properly
assign the correct(ed) ranks. (As mentioned above this is the same rank that
would be obtained using the sql RANK() function and ordering by Index,StrNum.
And note that using the sql RANK() would eliminate the need to do a join in 
dataphor. Imagine dataphor with native sql like ranking operations ☺ ) 

Because the objective is to create a string to sort the ids we can't just
use the numeric rank, we have to modify it for string ordering. Given two
ranks of 7 and 11 if they are strings, '7' and '11', an ascending sort
would have '11' come '7':

'11'
'7'

This is the very problem the article is addressing! So we have to modify the
the strings to have '7' come '11' ☺. We can modify the '7' by left padding it
with '0'. So when we sort ascending we'll have the correct representation of
of the true numeric order of the values:

'07'
'11'

How much to pad a rank, how many '0's to insert, is the string length of 
the maximum rank generated. Because the ranks in dataphor keep ascending
regardless of the cursor ordering, the maximum rank (ignoring duplicates)
is the count of rows in the table. You could even make an educated guess
based on the amount of data and use that ☺. 

Left padding the string rank (RankStr) based on a maximum length of 2 we now
have all the data to finally construct a sorting column for the ids.

var UnpackedStrings:= //This variable holds all the split data and will be used in the select query. 
ToTable(ToList(cursor(
                       (
                        (IZ add{val.Split({'.'}) StrList,val.Split({'.'}).Count() StrListCnt})
                          times
                            (Numbers where N<10)
                               where N<StrListCnt
                             {id,N+1 Index,ToInteger(StrList[N]) StrNum}
                        )   
                           order by {Index,StrNum}))) 
                          {Index,id,StrNum,sequence+1 RowNum};
var LengthofMaxRank:=2;
select
      (UnpackedStrings {id,Index,StrNum})
         join //Join the unique ranks to all split data. This is a natural join (on Index/StrNum).
              //Create a left padded string (RankStr) from the numeric rank.
          (UnpackedStrings group by {Index,StrNum} add{Min(RowNum) Rank})
            {id,Index,StrNum,Rank,PadLeft(ToString(Rank),LengthofMaxRank,'0') RankStr}
              order by {Index,Rank};
            
id Index StrNum   Rank RankStr 
-- ----- -------- ---- ------- 
13 1     -321     1    01     <- ranks 1-9 are left padded in the rank string (RankStr).    
12 1     -123     2    02      
11 1     -22      3    03      
10 1     -11      4    04      
9  1     -2       5    05      
8  1     -1       6    06      
2  1     7        7    07      
7  1     7        7    07      
6  1     10       9    09      
3  1     22       10   10      
4  1     22       10   10      
5  1     22       10   10      
14 1     22       10   10      
15 1     22       10   10      
1  1     100      15   15      
.  .     .        .    .
3  8     478      56   56      
4  8     478      56   56      
5  8     478      56   56      
14 8     478      56   56      
15 8     478      56   56      
5  9     9999999  61   61      
3  9     19710212 62   62      
4  9     19710212 62   62      
14 9     19710212 62   62      
15 9     19710212 62   62    

The id sort column can now be formed by concatenating, using the Concat 
operator, the RankStr within each id in the order of the rank (either Rank,
Index or RankStr). This is easy to see by ordering the above data (table) 
by id,Rank. The ascending order of Index, Rank and RankStr all reflect 
where an id lies in value (RankStr) relative to the other ids. The sort
expression will be column SortStr.

id Index StrNum   Rank RankStr 
-- ----- -------- ---- ------- 
1  1     100      15   15      
2  1     7        7    07      
2  2     4        16   16      
2  3     250      30   30      
3  1     22       10   10      
3  2     40       19   19      
3  3     5        24   24      
3  4     60       33   33      
3  5     4        38   38      
3  6     100      49   49      
3  7     300      51   51      
3  8     478      56   56      
3  9     19710212 62   62    

The complete dataphor solution

var UnpackedStrings:= //Variable that holds all split data and ranks the split strings
                      //numerically within each Index ordered by the numeric string value.
ToTable(ToList(cursor(
                       (
                        (IZ add{val.Split({'.'}) StrList,val.Split({'.'}).Count() StrListCnt})
                          times
                            (Numbers where N<10)
                               where N<StrListCnt
                             {id,N+1 Index,ToInteger(StrList[N]) StrNum}
                        )   
                           order by {Index,StrNum}))) 
                          {Index,id,StrNum,sequence+1 RowNum};
var LengthofMaxRank:=
  Length(Count(UnpackedStrings).ToString()); //A string length used to left pad the rank strings for
                                             //a correct string sort (will be 2 here). 
select
    IZ
     join   //Natural join of input table IZ to sort column expression by id.
         (
            //Join unique ranks to all split strings.
          (
           (UnpackedStrings {id,Index,StrNum})
             join
                 // Adjust ranks to be unique for each Index/StrNum, duplicate values should get same rank.
              (UnpackedStrings group by {Index,StrNum} add{Min(RowNum) Rank})
                 //Left pad rank string with '0' (RankStr) for sorting string correctly. Add empty
                 //string ('') to be used as a delimiter in concatenating the ranks.
               {id,Index,StrNum,Rank,PadLeft(ToString(Rank),LengthofMaxRank,'0') RankStr,'' Del}
          ) adorn {key{id,Rank}} //Sql uses physical hints, in dataphor you use logical ones.
                //Form the sorting expression SortStr to sort ids by concatenating the string ranks
                //in the order of any of the ranking columns or Index.
              group by {id} add{Concat(RankStr,Del order by {id,Rank}) SortStr}
         ) 
          order by {SortStr}; //The object of the exercise, sort the ids by SortStr to get the correct
                              //numerical order of val.

id val                                SortStr            
-- ---------------------------------- ------------------ 
13 -321                               01                 
12 -123                               02                 
11 -22                                03                 
10 -11                                04                 
9  -2                                 05                 
8  -1                                 06                 
2  7.4.250                            071630             
7  7.4.250                            071630             
6  10.30.40.50.20.30.40               09182932434650     
14 22.40.5.60.4.-100.300.478.19710212 101924333844515662 
15 22.40.5.60.4.-99.300.478.19710212  101924333845515662 
5  22.40.5.60.4.99.300.478.9999999    101924333847515661 
4  22.40.5.60.4.99.300.478.19710212   101924333847515662 
3  22.40.5.60.4.100.300.478.19710212  101924333849515662 
1  100                                15              

Solution using the sql server RANK() function with a pass thru query

The solution can be made more compact using the sql RANK() function since
dataphor doesn't have a direct equivalent ranking operation.

Since sql server can only access persisted tables (and views) and not
dataphor expressions (we can't just stick in any dataphor table expression
in a pass thru query) we'll create a persisted table to hold all the 
split data.

create table IZSqlRanks  //The table will be created in sql server.
{
 id:Integer,
 Index:Integer,
 StrNum:Integer,
 key{id,Index}
}; 

Do the splitting in dataphor (because it knows the difference between a
string and a list ☺ ) and then 'assign' the resulting table to the
persisted table IZSqlRanks. Relational databases support this kind of
assignment for all variables including tables. And all tables in dataphor
are variables. (To my sql friends this makes all the difference in the world ☺ )

IZSqlRanks:=
 (IZ add{val.Split({'.'}) StrList,val.Split({'.'}).Count() StrListCnt})
    times
    (Numbers where N<10)
      where N<StrListCnt
       {id,N+1 Index,ToInteger(StrList[N]) StrNum};
//Use a t-sql passthru query to take advantage of the sql RANK() function. The
//resulting table will be treated like any other table (expression) in dataphor.
//Left pad the string rank (RankStr) for sorting purposes. We're using a total
//string length of 2 here so single digit ranks will be padded with a leading '0'. 
select 
 IZ
  join
     (
       SQLQuery('SELECT id,[Index],StrNum,RANK()OVER(ORDER BY [Index],StrNum) AS Rank
                 FROM IZSqlRanks') 
     //Use the sql result as if it was a native dataphor expression.
        {id,Index,StrNum,Rank,PadLeft(ToString(Rank),2,'0') RankStr,'' Del} 
          adorn {key{id,Rank}} //Metadata (a key) pertaining to the table expression.
                               //This key will be efficiently used by the Concat operation.
     //Concatenate the rank strings for each id to be used as the sort order for ids.
          group by {id} add{Concat(RankStr,Del order by {id,Rank}) SortStr}
      )       
        order by {SortStr};     
        
id val                                SortStr            
-- ---------------------------------- ------------------ 
13 -321                               01                 
12 -123                               02                 
11 -22                                03                 
10 -11                                04                 
9  -2                                 05                 
8  -1                                 06                 
2  7.4.250                            071630             
7  7.4.250                            071630             
6  10.30.40.50.20.30.40               09182932434650     
14 22.40.5.60.4.-100.300.478.19710212 101924333844515662 
15 22.40.5.60.4.-99.300.478.19710212  101924333845515662 
5  22.40.5.60.4.99.300.478.9999999    101924333847515661 
4  22.40.5.60.4.99.300.478.19710212   101924333847515662 
3  22.40.5.60.4.100.300.478.19710212  101924333849515662 
1  100                                15                        

Solving the problem on sql server 2000 with the Rac utility

The Rac solution follows the same logic as the dataphor and sql methods.
The 1st Rac execute creates the Index column, from 1 to N, for every string
part (delimited integer) an id has. The 2nd Rac execute creates a rank over
the ids based on the integers (string parts) within each Index. The rank
Rac generates is equivalent to the sql DENSE_RANK() function. This rank,
like RANK(), gives duplicate values the same rank but, unlike RANK(), does
not take the duplicate ranks into account when generating the next rank.
RANK() skips ahead based on ties while DENSE_RANK() consecutively numbers
the ranks. Both types of ranks give the same correct sort result for the ids.
The 3rd Rac execute concatenates the left padded rank strings for each id
and returns the ids sorted by them, correctly :) Note that Rac is called
recursively twice.

Exec Rac 
@split='[position]',    -- Rac splits each val string from left to right based on a period ('.').
@rows='id & [position]',-- Rac keeps the position each new string part starts in.
@pvtcol='val',          -- the target column of the split operation.
@from='IZ',    
-- use a counter to generate the Index column (from 1-N) that indicates the individual string part in a val.  
@separator='.',@rowcounters='id{[Index]}',@counterdatatype='int',
@defaults1='y',@rowbreak='n',@racheck='y',@shell='n',
@select='
        SELECT 1*id AS id,[Index],CAST([1] AS INT) AS StrNum
               INTO #J1      
                 FROM rac
 Exec Rac
 @transform=~_dummy_~,
 @rows=~[Index] & StrNum & id~,
 @pvtcol=~Report Mode~,
 @from=~#J1~,
 @defaults1=~y~,@rowbreak=~n~,@racheck=~y~,@shell=~n~,
  /* use a Rac counter to rank the integer string parts within each Index (column Rank).*/
 @rowindicators=~StrNum{Rank}_overtable_~,@counterdatatype=~int~,
        @select=
          ~if object_id(~~##J2~~) is not null 
           drop table dbo.t1  
           SELECT 1*id AS id,1*StrNum AS StrNum,1*[Index] AS [Index],1*Rank AS Rank,
                  /* left pad single digit string ranks for proper character sorting.*/
                  REPLICATE(~~0~~,2-DATALENGTH(CAST(Rank AS VARCHAR(2)))) + CAST(Rank AS VARCHAR(2)) AS RankStr
                  INTO ##J2
                   FROM rac~
             /* concatenate the string ranks (RankStr) within each id into column SortStr.*/     
      Exec Rac
      @transform=~Max(RankStr) as RankStr~,
      @rows=~id~,
      @pvtcol=~Rank~,
      @from=~##J2~,
      @defaults1=~y~,@racheck=~y~,@shell=~n~,@cutpvt=~y~,
      @concatenate=~RankStr~,@separator=~~,@stringname=~SortStr~,
       /* return the ids sorted by the concatenated string ranks.*/
      @select=~SELECT IZ.id,val,SortStr  
               FROM IZ JOIN rac ON IZ.id=rac.id 
               ORDER BY SortStr
               DROP TABLE ##J2~'

id          val                                  SortStr
----------- ------------------------------------ ------------------
13          -321                                 01
12          -123                                 02
11          -22                                  03
10          -11                                  04
9           -2                                   05
8           -1                                   06
7           7.4.250                              071116
2           7.4.250                              071116
6           10.30.40.50.20.30.40                 08121517202326
14          22.40.5.60.4.-100.300.478.19710212   091314181921272830
15          22.40.5.60.4.-99.300.478.19710212    091314181922272830
5           22.40.5.60.4.99.300.478.9999999      091314181924272829
4           22.40.5.60.4.99.300.478.19710212     091314181924272830
3           22.40.5.60.4.100.300.478.19710212    091314181925272830
1           100                                  10

More on Rac @
www.rac4sql.net

Solving the problem on sql server 2005 with the Rac utility

The Rac solution in S2k5 is similar to the one on S2K (go there for more info).
But it's simpler since the split string parts (integers) can be ranked
directly using a dense rank function not available in S2k. This eliminates
a recursive call to Rac so here it's executed twice instead of three times (in S2k).

Exec Rac 
@split='[position]',   
@rows='id & [position]',
@pvtcol='val',
@from='IZ',
@separator='.',@rowcounters='id{[Index]}',@counterdatatype='int',
@defaults1='y',@rowbreak='n',@racheck='y',@shell='n',
 -- the DENSE_RANK() function, available in S2k5, is used to rank the integer string parts
 -- and is left padded.
@select='
         SELECT id,[Index],StrNum,Rank,
                REPLICATE(~0~,2-DATALENGTH(CAST(Rank AS VARCHAR(2)))) + CAST(Rank AS VARCHAR(2)) AS RankStr
                INTO #J1     
         FROM 
              (SELECT 1*id AS id,[Index],CAST([1] AS INT) AS StrNum,
                      DENSE_RANK()OVER(ORDER BY [Index],CAST([1] AS INT)) AS Rank
               FROM rac ) AS A
     Exec Rac
     @transform=~Max(RankStr) as RankStr~,
     @rows=~id~,
     @pvtcol=~Rank~,
     @from=~#J1~,
     @defaults1=~y~,@racheck=~y~,@shell=~n~,@cutpvt=~y~,
     @concatenate=~RankStr~,@separator=~~,@stringname=~SortStr~,
     @select=~SELECT IZ.id,val,SortStr  
               FROM IZ JOIN rac ON IZ.id=rac.id 
                ORDER BY SortStr~'

id          val                                  SortStr
----------- ------------------------------------ ------------------
13          -321                                 01
12          -123                                 02
11          -22                                  03
10          -11                                  04
9           -2                                   05
8           -1                                   06
7           7.4.250                              071116
2           7.4.250                              071116
6           10.30.40.50.20.30.40                 08121517202326
14          22.40.5.60.4.-100.300.478.19710212   091314181921272830
15          22.40.5.60.4.-99.300.478.19710212    091314181922272830
5           22.40.5.60.4.99.300.478.9999999      091314181924272829
4           22.40.5.60.4.99.300.478.19710212     091314181924272830
3           22.40.5.60.4.100.300.478.19710212    091314181925272830
1           100                                  10

More on Rac @
www.rac4sql.net

Demoralization by trigger

2008-07-16T20:14:00.000-07:00

In an interesting blog by Conor Cunningham, 'The Trouble with Triggers', he says:
"The problem with this area is that there is a great temptation to think
 about databases procedurally, as you would with a programming language
 like C++ or C#.  You can write code that looks like a procedural function
 call and have it get called for each insert into table!  Before you know
 it, non-database programmers are checking in code to your production sysem.
 Suddenly, your application grinds to a halt because a trigger plan has no
 index or is poorly designed.
 Databases are great tools for working on SETS of rows."

In another related blog, 'Triggers...Evil?', there is this comment with an 
insightful Freudian slip:

James Luetkehoelter said:
>Some sorts of demoralization lend themselves to triggers...

I would say this hits the nail on the head. I could understand a developer
getting a case of depression triggered by Conors article. Triggers were
'implemented' to work efficiently on tables (sets) not on rows. The principle
that's operating here is that how something was implemented to be most
effective is the basis for what's best in application development. Are you
kidding me, has everyone gone nuts? ☺ Because triggers don't consider a row
as a primary concept 'functional' programmers, application developers, must
'unlearn' their database contrarian views. This is Celkos 'them' vs. 'us' 
nonsense. Never mind that the real subject is application development and
possibly a theory that would best serve it, the basis for key concepts is 
what a bunch of programmers did for a MS product manager. Talk about the
tail wagging the dog ☺ Not only is the absence of a 'row' type or at least
concept antithetical to a relational dbms but it's central to application
development. Perhaps to even the score MS decided developers should learn
entities and unlearn sql entirely, LINQ. Or perhaps we'll get a hole new
science of application development based on what works fastest ☺

An opposite constraint

2008-07-06T15:46:00.000-07:00

This article discusses an issue raised in the thread:

microsoft.public.sqlserver.programming
Sunday, June 29, 2008 
'Storing a collection of lines'
http://tinyurl.com/56dpd4

Give two columns in a table suppose you want to eliminate the opposite
data where there's no mathematical or logical relationship between
the columns. For example, consider a trip between two cities. It's 
equally likely a trip could start and end in either direction. If the 
table already has:

column A  column B
--------  --------
NEW YORK  CHICAGO

We want to prevent the opposite from being entered:

column A  column B
--------  --------
CHICAGO   NEW YORK  

If the table has:

column A  column B
--------  --------
CHICAGO   NEW YORK  

We want to prevent the opposite from being entered:

column A  column B
--------  --------
NEW YORK  CHICAGO

Because there's no relationship between the columns an sql check
constraint can't be used. But in Dataphor a simple transition
constraint can be used. The Opposite constraint simply checks if 
the opposite data for columns A and B already exists in the table. 
If it does the current row is rejected. The constraint works the
same way for a single insert as it does for inserting multiple rows
(a table). 
For example:

create session table MyTable
{ A:String,B:String,C:String,key{A,B} };

alter table MyTable
{
  create transition constraint Opposite
   on insert //The current (row) values for columns A and B are accessed 
             //by prefixing each with 'new', ie. new.A, new.B . 
    not exists (MyTable {A X,B Y} {Y A,X B} where A=new.A and B=new.B)  
 tags        //A custom error message can be written using the current 
             //row values (new.A, new.B).
 { DAE.Message =
   "'For A: ' + new.A + ' and B: '+ new.B + ' there is a opposite, A: ' + new.B + ' and B: ' + new.A " 
 }   
};

These insert succeed:

insert row{'NEW YORK' A,'CHICAGO' B,'1c' C} into MyTable;
insert row{'CALIFORNIA' B,'TEXAS' A,'1d' C} into MyTable;

Inserting an opposite will fail and the custom error message will be raised:

insert row{'NEW YORK' B,'CHICAGO' A,'1e' C} into MyTable;
"For A: CHICAGO and B: NEW YORK there is a opposite, A: NEW YORK and B: CHICAGO"

insert row{'CALIFORNIA' A,'TEXAS' B,'1d' C} into MyTable;
"For A: CALIFORNIA and B: TEXAS there is a opposite, A: TEXAS and B: CALIFORNIA"

Given that the table contains the data "NEW YORK" (A) and "CHICAGO" (B), 
inserting the following rows as a table will fail:

insert table{
             row{'DENVER' A,'BOSTON' B,'1c' C},
             row{'RENO' B,'MIAMI' A,'1d' C},
             row{'CHICAGO' A,'NEW YORK' B,'1e' C} 
            } into MyTable;
       
"For A: CHICAGO and B: NEW YORK there is a opposite, A: NEW YORK and B: CHICAGO. 

Note that the primary key constraint will eliminate the same A and B cities from
being entered twice but entering the opposite cities does not violate it. That's
what the Opposite constraint is for. The Opposite constraint is much simpler than
an sql solution using triggers.

A one-one requirement constraint

2008-07-06T03:14:00.000-07:00

The issue of a one-one requirement, a new order inserted must be
accompanied by a detail row (insert), was raised in the thread:

microsoft.public.sqlserver.programming
Thursday, June 26, 2008 
'update joined tables'
http://tinyurl.com/69hh5c

While this is a non-trivial exercise in sql it's quite easy in the
relational system Dataphor. For example:

create session table SOrders
{
 order_nbr:Integer,
 A:Integer,
 key{order_nbr}
}; 
 
create session table SOrderDetails
 {
  order_nbr:Integer,
  sku:Integer,
  B:Integer,
  C:String,
  key{order_nbr,sku},
  reference Details_SOrders{ order_nbr } references SOrders { order_nbr }
 };

This constraint makes a detail row a requirement for a new order. The user
defined message will be displayed anytime the constraint is violated.

alter table SOrders
  {
 create transition constraint MustHaveOrderDetails
  on insert 
    exists(SOrderDetails where order_nbr=new.order_nbr)  
   tags
 {
   DAE.Message =
          "'A detail row with order#: ' + ToString(new.order_nbr) + ' is required'" 
        }    
  };   
 
Inserting just a new order will violate the constraint:
insert row{1 order_nbr, 14 A} into SOrders; 
Violation of constraint "MustHaveOrderDetails", "A detail row with order#: 1 is required."

Inserting data can easily be done via a view.

create session view VSOrderDetails
                SOrders join SOrderDetails;//A natural join on order_nbr.

The MustHaveOrderDetails constraint works for the insertion of a single
row or a table.

insert row{1 order_nbr, 1 sku, 14 A, 5 B, 'Joe' C} into VSOrderDetails;     

delete SOrders;

Insert into the tables thru the view.

insert table
            {
             row{1 order_nbr, 1 sku, 14 A, 5 B, 'Joe' C},
             row{1,2,9,23,'Steve'},
             row{2,3,34,2,'Larry'}
            } into VSOrderDetails;   

select SOrders;

order_nbr A  
--------- -- 
1         14 
2         34 

select SOrderDetails;            

order_nbr sku B  C     
--------- --- -- ----- 
1         1   5  Joe   
1         2   23 Steve 
2         3   2  Larry 

This rather simple example shows off the much higher level of abstraction
offered to developers with a relational system like Dataphor over sql. In
sql only one table may be updated with a view. In Dataphor there is no
concept of updating multiple tables. There is only the idea of updating a
view/expression which is a table. It's the system responsibility to resolve
a view to its base tables given the constraints in the schema. The user need
only realize a table is being used for data modification and it's irrelevant
how that table was constructed as well as the number of tables in its definition.

Justifying an Sql Cop Out

2008-07-04T23:33:00.000-07:00

This is my response to Joe Celko in the thread:

microsoft.public.sqlserver.programming
Thursday, June 26, 2008 
'update joined tables'
http://tinyurl.com/69hh5c

Joe invokes the 'NP complete' argument as the basis for why sql can't
handle complicated modeling issues. The specific modeling issue here
concerns a schema where there are multiple paths between tables due to
cascading updates. My view is the NP complete argument is essentially a
cop out. The same schema with multiple cascading updates that is rejected
by sql server is perfectly acceptable in Dataphor (along with associated
views involving multiple tables that can resolve data to the base tables).
(I suggest reading thru the whole thread, it's interesting ☺)

My response in the thread:

What do we call a man who demurs to an obstinate woman on every decision? We 
call him a whimp, a guy who doesn't think it's worth the effort to present 
his counter point because he's convinced it's impossible to change her 
mind. The bottom line is, if it's so hard to change her mind why even 
bother. And an easy out for taking any responsibility for control of the 
outcome. Here you're using the idea of an NP complete problem as a crutch 
for the mess sql has left developers in. Woman may be an NP complete problem 
for man but it doesn't follow that given a relationship all men are whimps. 
Just because there isn't a universal and instantaneous quick fix technique 
to change a woman's mind in any situation doesn't bum out all men. Some guys 
are inventive and creative and come up with techniques that will work at 
least in some situations. Isn't that what lying, pleading and begging are 
for?:) NP complete system problems are brick walls not because there's no 
way to solve them but because there's no good quick fix from a systems point 
of view. But that doesn't mean that dbms should whimp out on them. You're using 
NP to suggest an all or nothing game. Since no efficient computational 
scheme exists to cover all situations where referential integrity involving 
cascading updates comes into play then sql is going take its ball and just 
go home. This is at most nonsense, at the very least no sense. You're setting 
poor sql up as the victim here and whining about Petri Nets! The perfect 
solution doesn't exist, so what. Sql is/was just being damn lazy. All they 
had to do is talk to a bunch of guys who have been married for twenty years 
to get a clue:) Like they couldn't use some heuristics or approximations? 
Nope they just whimped out. Same with views. Sql is the victim of a yet to 
be found super quick universal solution to updatability. This is simply an sql 
crutch for abandoning the idea totally! They couldn't be creative and cover 
some percentage of possible expressions that could be resolved to base tables? 
Apparently not. So sql gets away with only being able to update a single table
and sticks a huge inconsistency between a table and view/expression in the face
of developers.

Here's an sql server example that has multiple cascading update paths:

-- Updating order_nbr in TableA cascades to order_nbr in TableB and TableC.
create table TableA
(
 order_nbr Integer,
 A         Integer,
 primary key(order_nbr)
) ;
 
 --Updating sku in TableB cascades to sku in TableC.
 create table TableB
(
 order_nbr Integer,
 sku       Integer,
 B         Integer
 primary key(order_nbr,sku),
CONSTRAINT  Details_TableB foreign key (order_nbr) REFERENCES TableA (order_nbr)
ON UPDATE CASCADE
) ;
  
 -- TableC has a dependency/path on TableA and TableB.
 create table TableC
(
 order_nbr Integer,
 sku       Integer,
 ID        Integer,
 C         char(1),
primary key (order_nbr,sku,ID),
CONSTRAINT Details_TableC1 FOREIGN KEY (order_nbr)     REFERENCES TableA (order_nbr)
ON UPDATE CASCADE,
CONSTRAINT Details_TableC2 FOREIGN KEY (order_nbr,sku) REFERENCES TableB (order_nbr,sku)
ON UPDATE CASCADE
 ) ;

Now is this a non-trival problem for a dbms. Yes it is. Is it insolvable in 
terms of solution and efficieny? Of course not. But when you try to create 
TableC you're told by sql server that it can't do it:

"Introducing FOREIGN KEY constraint 'Details_TableC2' on table 'TableC' may 
 cause cycles or multiple cascade paths. Specify ON DELETE NO ACTION or 
 ON UPDATE NO ACTION, or modify other FOREIGN KEY constraints."

Game over. Anything other than the most trivial path is outside the scope of 
the 'system' to handle. Users are forever told the sql game is 'what' not 
'how'. But when it's convenient and expedient you turn the idea around and 
say 'how' can this be done when there's no known (NP complete) solution. 
Nice sophistry!:) Now this same DDL, with some modification, is perfectly 
fine in Dataphor using sql server to persist the tables. And using views, 
such as TableA join TableB, is also fine. The views will resolve data to the 
tables and the update cascades work as expected. Do you really think that 
this problem is NP complete for the Sql Server team but not for Dataphor? 
Give me a break:)
Users should understand that because sql doesn't try to solve complex 
modeling issues and doesn't attempt to resolve views/expressions with 
multiple tables that this doesn't mean these things can't be solved. Dbms 
can be built so these things are handled by the 'system' and the user isn't
left twisting in the wind making a mess in the front end. Will something
like Dataphor be able to handle every conceivable combination of cascades
between tables along with other types of constraints? Of course not, but it
will handle a lot of tricky schemes that sql can't. Is Dataphor capable of
resolving (updatability) every view/expression to the base tables? Of course
not, but it will handle a large chunk of them. And in each case these things
are transparent to the user. It would be nice to have a little honesty about
this. Instead of creating straw men and red herrings just tell it like it is. 
The framework/foundation of sql is just not able to handle these issues. This
dye was cast a long time ago (and yes sql was lazy on a lot of fronts). And 
there's little hope of sql addressing these things in the future. This
certainly doesn't make sql irrelevant. It's a 'right tool for the job' world. 
I think sql folks can handle it. After all, not all sql users are whimps:)

Dataphor - Splitting a string into fixed sized word segments

2008-05-14T03:00:00.000-07:00

This is a solution using Dataphor to the problem in the post:

comp.databases.ms-sqlserver
May 9, 2008
'SQL Statement Problem : Separate long varchar into word seqments 35 chars long'
http://tinyurl.com/42vty5

The op states the problem as:
'I have a table of names which can be very long.  These names get printed on envelopes.  
 The problem is the envelope can only hold 35 characters per line.  I have to divide into
 4 lines at most. 
 So I need to separate these long varchars into segments, no longer than 35 characters but 
 preserving whole words.' 

Data is stored in MS Sql Server 2005. Here is some sample data stored in table SMerge:

-- Tsql DDL statements to create the SMerge table in a sql server database.
create table SMerge (ID Int primary key,InStr varchar(250))
go
insert SMerge values
(1,'BIOLOGICAL SURVEY FOUNDATION OF ONTARIO HAPPY IF SOMEONE CAN HELP SOLVE THIS SQL PROBLEM') 
insert SMerge values
(2,'I SURE HOPE I CAN GET THIS THING TO WORK RIGHT') 
insert SMerge values
(3,' What will happen  with a big string full of all kinds of names and addresses including  
     some goobly goop. Will it hold up ?') 

This is a fairly simple iterative solution. It illustrates various dataphor programming
capabilities including set based, cursor and procedural.
I've relaxed the requirement of at most 4 lines. Given a maximum line size the batch
iterates over each input string creating lines not exceeding the maximum size until all 
words in the string are assigned to a line. There is no limit to the number of lines
allowed (although there certainly could be). 

//                    Start of batch

if not ObjectExists("WTable") then                  
Execute(
"create session table WTable         //Temporary table used in batch. 
{
 StrID:Integer,                      //Input string id.
 Str:String,                         //An individual string, ie. a word, from splitting.
 Seq:Integer,                        //Position/sequence of word from split.
 Interval:Integer nil {default nil}, //The line number the word will eventually belongs to.
 StrLen:Integer,                     //Length of word (Str).
 RunningSumStrLens:Integer nil,      //Running sum of word lengths within a line. 
 LenCheck:Boolean {default false},   //Whether or not a word (Str) has been assigned to a line.
 key{StrID,Seq}
};"
      )
  else
   Execute("delete WTable;");
                               
var MaxLineLength:=35; //Max length of line desired.
var LCheck:Boolean;    // Holds boolean (t/f) of adding string to line, 
                       // whether line is within max line size. 
var MaxSeq:Integer;    //Holds max Seq for ID and line (interval).
                       // SMerge is the sql server table. 

var LCursorSqlTable:=cursor(SMerge order by {ID});

// Loop over all rows in the sql table splitting each input string (InStr).

while LCursorSqlTable.Next() do
begin

// Error check. If for any ID the length of any word (Str), via Split, exceeds the max line 
// size then eliminate the ID from the result. This also insures no endless cursor loops.

if exists(
          ToTable(LCursorSqlTable.Select().InStr.Split({' '}),'Str','Seq') 
                   where Length(Str)>MaxLineLength
         ) then
 continue; //Skips below code and gets another input string (InStr).
 
// Insert the split string into work table WTable. Split operator splits using blank as
// the delimiter. A sequence number (Seq) is also generated which is an integer indicating
// the position (from 0 to N) from left to right of each word (Str) in the input string (InStr).
// We also compute the length each word (Str). This is stored in column StrLen.

  insert
     (
      ToTable(LCursorSqlTable.Select().InStr.Split({' '}),'Str','Seq') add{Length(Str) StrLen,
                   LCursorSqlTable.Select().ID StrID,
                   nil as Integer RunningSumStrLens} 
                   where (StrLen>0) //Eliminate extra spaces. We could do all sorts of other
                                    //checks and manipulations here as well.
      ) into WTable;  
var LInterval:=0; //Line(Interval) within an ID (InStr).
var LRunningSumStrLens:Integer;  
MaxSeq:=-1; //Start current ID with Seq>-1, ie. include all words (Seq starts at 0).

// Iterate over WTable. Get a running sum of word lengths. Each iteration represents a new
// line based on comparing the running sum of word lengths to the max line size. Mark words
// in rows of WTable as either true (part of new line) or false (to be assigned in a subsequent
// iteration). As long as there are words to be assigned to a line the iteration (loop)
// continues. The loop stops when all words for an ID have been optimally assigned to a
// line (Interval). In other words loop stops when all values of LenCheck are true because
// each word has been assigned to a line.

 while exists(WTable where (StrID=LCursorSqlTable.Select().ID) and (Seq>MaxSeq)) do
 begin
 LInterval:=LInterval+1 ;  
 LRunningSumStrLens:=0;  //Initialize running sum for word lengths to 0 for a new line.

// Use WTable eliminating prior assigned words. We don't need these rows, we need rows 
// representing words that still need to be assigned to a line. Seq>MaxSeq gets these rows,
// it gets all rows (words) for an ID that haven't yet been assigned to a line (Interval).

 var LCursor:=
 cursor( WTable where ((StrID=LCursorSqlTable.Select().ID) and (Seq>MaxSeq)) order by {StrID,Seq} 
                                        capabilities {Navigable, Searchable, Updateable});
  while LCursor.Next() do   

// For all remaining words that aren't yet assigned to a line(Interval) for the current ID
// get a running sum of word lengths and check that the run value is <= max line size.

  begin

// If it's the start of a new line we only want the running sum to be the length of the word, 
// if it's after the 1st word we add the previous word lengths, which includes a 1 for a space
// between words, and the length of the current word.

  LRunningSumStrLens:= if LRunningSumStrLens=0 then LCursor.Select().StrLen 
                       else LRunningSumStrLens + 1 + LCursor.Select().StrLen;
// Check if the word, as part of the running sum, is within the maximum line size.                       
  
  LCheck:=LRunningSumStrLens<=MaxLineLength ;

// If the current word length, when added to the running sum of lengths, is greater than the  
// max line size there's no more point in staying within the LCursor loop. The line(Interval)
// has been determined using prior words so break out of this loop and start forming a new 
// line(Interval) with as yet unassigned words (where LenCheck=false) with a new running sum. 

  if not LCheck then
               break;

// Update LenCheck to true. The current word fits on the current line as tested by comparing
// the running sum (with the current word length) to the max line size. We can now ignore
// this word (row) in assigning subsequent (unassigned) words to a line. Remember that the
// default value of LenCheck is false. So we don't have to update words (set LenCheck=false) 
// as false since this is the default. And we'll get those words (where LenCheck=false) on
// the next iteration of the loop.  

  LCursor.Update(row{LRunningSumStrLens RunningSumStrLens,LCheck LenCheck,LInterval Interval});
 end; //Lcursor, looping over WTable for an ID until all words are assigned to a line. 

// Find max Seq for the current ID where the word has been assigned to a line (LenCheck=true). 
// The next definition for the cursor for WTable starts with Seq = MaxSeq+1, ie. the rows with
// Seq values greater than the max Seq value of assigned words. Interval is also used here
// to help make the query more efficient (a possible better alternative would be to keep
// track of Seq).

 MaxSeq:=Max(Seq from WTable where (StrID=LCursorSqlTable.Select().ID) 
                                    and (Interval=LInterval) and (LenCheck));
 end; //WTable
end;  //LCursorSqlTable. 

// Here is a simple check that this batch assigned each word to a line for each ID. All
// LenCheck values must be true or else there is a failure somewhere in batch. Of course we
// could more thoroughly check exactly where (ID and line) failure(s) occurred in the query.
// We could also put extensive error handling throughout batch. Just so you know ☺

if not All(LenCheck from WTable) then
               exit;

// Use a query to build the output strings (ALine) by concatenating (using the Concat operator) 
// the words (Str) within IDs and lines (Interval) and ordering the concatenation by Seq 
//(the position determined by Split). We also include the line length for each 
// concatenated line (ALine) of words within an ID.

select WTable add{' ' del} 
         adorn {key{Seq}}  
            group by {StrID,Interval} add{ Concat(Str,del order by {Seq}) ALine} 
              add{Length(ALine) LineLength}
                  order by {StrID,Interval}; 

//                    End of batch                  
                  
Result:

StrID Interval ALine                               LineLength 
----- -------- ----------------------------------- ---------- 
1     1        BIOLOGICAL SURVEY FOUNDATION OF     31         
1     2        ONTARIO HAPPY IF SOMEONE CAN HELP   33         
1     3        SOLVE THIS SQL PROBLEM              22         
2     1        I SURE HOPE I CAN GET THIS THING TO 35         
2     2        WORK RIGHT                          10         
3     1        What will happen with a big string  34         
3     2        full of all kinds of names and      30         
3     3        addresses including some goobly     31         
3     4        goop. Will it hold up?              22     

Here is what WTable looks like before concatenation. Note the running sum of string lengths
(RunningSumStrLens) within each line (Interval) for each ID (StrID).

select WTable order by {StrID,Interval,Seq};    

StrID Str        Seq Interval StrLen RunningSumStrLens LenCheck 
----- ---------- --- -------- ------ ----------------- -------- 
1     BIOLOGICAL 0   1        10     10                True     
1     SURVEY     1   1        6      17                True     
1     FOUNDATION 2   1        10     28                True     
1     OF         3   1        2      31                True     
1     ONTARIO    4   2        7      7                 True     
1     HAPPY      5   2        5      13                True     
1     IF         6   2        2      16                True     
1     SOMEONE    7   2        7      24                True     
1     CAN        8   2        3      28                True     
1     HELP       9   2        4      33                True     
1     SOLVE      10  3        5      5                 True     
1     THIS       11  3        4      10                True     
1     SQL        12  3        3      14                True     
1     PROBLEM    13  3        7      22                True     
2     I          0   1        1      1                 True     
.
.
3     hold       23  4        4      18                True     
3     up?        24  4        3      22                True     


How will this batch work for larger strings? Lets take the whole question of the post
(cut and pasted from the op on google).

Here, through dataphor, we assign the sql server table SMerge just one row of data 
consisting of the post data as the input string. As part of the insert we remove
carrige returns and line feeds using the Replace function.

SMerge:=
table
{
row{1 ID,
"I'm having a really tough time with a SQL statement and I am wondering
if someone is able to help out or point me in the right direction.

I have a table of names which can be very long.  These names get
printed on envelopes.  The problem is the envelope can only hold 35
characters per line.  I have to divide into 4 lines at most.

So I need to separate these long varchars into segments, no longer
than 35 characters but preserving whole words.

So far my approach has been to take a LEFT segment, REVERSE it, find
the first space with CHARINDEX and use it to calculate how many
characters to take in a SUBBSTRING.

Here's an example of what I have been trying.  I can find the first
two segments, but then it starts to get confusing. 
Can anyone suggest a better approach?  Am I going to be able to do
this in SQL?

I appreciate any help." InStr}
} // Replace carrige returns and line feeds with blanks in InStr.
redefine {InStr:=Replace(Replace(InStr,ASCII(list(Byte){13}) ,' '),ASCII(list(Byte){10}) ,' ')}
;

Executing the batch with a maximum line size of 80 (MaxLineLength:=80) we get:

StrID Interval ALine                                                                            LineLength 
----- -------- -------------------------------------------------------------------------------- ---------- 
1     1        I'm having a really tough time with a SQL statement and I am wondering if        73         
1     2        someone is able to help out or point me in the right direction. I have a table   78         
1     3        of names which can be very long. These names get printed on envelopes. The       74         
1     4        problem is the envelope can only hold 35 characters per line. I have to divide   78         
1     5        into 4 lines at most. So I need to separate these long varchars into segments,   78         
1     6        no longer than 35 characters but preserving whole words. So far my approach has  79         
1     7        been to take a LEFT segment, REVERSE it, find the first space with CHARINDEX and 80         
1     8        use it to calculate how many characters to take in a SUBBSTRING. Here's an       74         
1     9        example of what I have been trying. I can find the first two segments, but then  79         
1     10       it starts to get confusing. Can anyone suggest a better approach? Am I going to  79         
1     11       be able to do this in SQL? I appreciate any help.                                49         

Executing the batch with a maximum line size of 70 (MaxLineLength:=70) we get:

StrID Interval ALine                                                                  LineLength 
----- -------- ---------------------------------------------------------------------- ---------- 
1     1        I'm having a really tough time with a SQL statement and I am wondering 70         
1     2        if someone is able to help out or point me in the right direction. I   68         
1     3        have a table of names which can be very long. These names get printed  69         
1     4        on envelopes. The problem is the envelope can only hold 35 characters  69         
1     5        per line. I have to divide into 4 lines at most. So I need to separate 70         
1     6        these long varchars into segments, no longer than 35 characters but    67         
1     7        preserving whole words. So far my approach has been to take a LEFT     66         
1     8        segment, REVERSE it, find the first space with CHARINDEX and use it to 70         
1     9        calculate how many characters to take in a SUBBSTRING. Here's an       64         
1     10       example of what I have been trying. I can find the first two segments, 70         
1     11       but then it starts to get confusing. Can anyone suggest a better       64         
1     12       approach? Am I going to be able to do this in SQL? I appreciate any    67         
1     13       help.                                                                  5   

Seems to work ok ☺

Note that with this simple framework it would be easy to add all kinds of editing niceties. 
Left justification, centering etc. could be easily added. It would also be easy to edit/check
the individual words, possibly eliminating some given specific rules. Note that we could
easily make the batch an operator (stored procedure) returning the work table or a virtual table.

If you have any ideas on further features or what would be kewl for this framework please 
let me know ☺

You can check out dataphor at:
www.dataphor.org

The Sql ranking OVERture

2008-04-19T19:59:00.000-07:00

Just about everyone who uses Sql Server 2005 likes the ranking functions. But like
a lot of things in sql they are a mile long and an inch deep. You sort of know the
syntax but when it comes to really understanding the concepts behind these functions 
things go south. One of the really interesting benefits of a relational system like
Dataphor is that the underlying relational concepts can illuminate what's going in sql. 
This article looks at the sql ranking functions through the prism of relational types. 
Sql ranking functions can be explained and understood in terms of the basic relational
types of cursor, table and list.  
 
I'm going to use a small sample of data from the Orders table from the Sq1 Server 2005
sample Northwind database: 

CREATE TABLE TRANK (OrderID int,CustomerID char(5),ShipCountry varchar(10),Freight money,
                     primary key (OrderID))
GO
INSERT INTO TRANK (OrderID,CustomerID,ShipCountry,Freight)
SELECT OrderID,CustomerID,ShipCountry,Freight
FROM Orders
WHERE CustomerID in ('CENTC','GROSR','LAZYK','NORTS','SPECD')
 
SELECT * FROM TRANK

OrderID     CustomerID ShipCountry Freight
----------- ---------- ----------- -------
10259       CENTC      Mexico      3.25
10268       GROSR      Venezuela   66.29
10482       LAZYK      USA         7.48
10517       NORTS      UK          32.07
10545       LAZYK      USA         11.92
10738       SPECD      France      2.91
10752       NORTS      UK          1.39
10785       GROSR      Venezuela   1.51
10907       SPECD      France      9.19
10964       SPECD      France      87.38
11043       SPECD      France      8.80
11057       NORTS      UK          4.13

Now before electricity, I mean before Sql Server 2005 and ranking functions, you had
to use a not very pretty subquery or join to get any kind of ranks/rownumbers. The
outstanding issues with these queries are they could get complicated and often result
in poor performance since they involve non-equal (>,<) comparisons involving NxN number
of rows. For example, consider getting a row number and a rank for CustomerID. 
(
 For more background see 
 Visualizing a ranking query 
 http://beyondsql.blogspot.com/2007/06/dataphor-sql-visualizing-ranking-query.html
)

Using an sql subquery to get a row number and a rank:

SELECT A.OrderID,A.CustomerID,A.ShipCountry,A.Freight,
-- Klunky subquery with a count and non-equi predicates to get unique row numbers for
-- CustomerID.
-- Duplicate CustomerID values are resolved by using the unique values of OrderID.
(SELECT COUNT(*) FROM TRANK AS B
WHERE (B.CustomerID<A.CustomerID) 
          OR ((B.CustomerID=A.CustomerID) and (B.OrderID<A.OrderID)))+1 as RowNumber,
-- Another klunky subquery to get ranks for CustomerID.
-- Important: note that Rank is the minimum RowNumber for each CustomerID.          
(SELECT COUNT(*) FROM TRANK AS B WHERE B.CustomerID<A.CustomerID)+1 as Rank
FROM TRANK AS A
ORDER BY CustomerID,OrderID

OrderID     CustomerID ShipCountry Freight RowNumber Rank
----------- ---------- ----------- ------- --------- ----
10259       CENTC      Mexico      3.25    1         1   <|-- Min RowNumber
10268       GROSR      Venezuela   66.29   2         2    <|-- Min RowNumber
10785       GROSR      Venezuela   1.51    3         2     |   
10482       LAZYK      USA         7.48    4         4     <|-- Min RowNumber
10545       LAZYK      USA         11.92   5         4      |
10517       NORTS      UK          32.07   6         6      <|-- Min RowNumber
10752       NORTS      UK          1.39    7         6       |
11057       NORTS      UK          4.13    8         6       |
10738       SPECD      France      2.91    9         9       <|-- Min RowNumber
10907       SPECD      France      9.19    10        9        |
10964       SPECD      France      87.38   11        9        |
11043       SPECD      France      8.80    12        9        | 

As the query shows there is a straightforward relationship between RowNumber and Rank. 
So even if you just wanted Rank, if you get the row numbers first, which will always
be unique, then all you have to do is group by CustomerID and take the minimum row
number value to get the Rank. If CustomerID and Rank were in a derived table it could
easily be joined back to TRANK to get all the data. Something like this.

Sql to derive ranks from row numbers:

SELECT G.OrderID,G.CustomerID,G.ShipCountry,G.Freight,Rank
FROM TRANK AS G 
      INNER JOIN
-- Get CustomerID and their rank from row numbers and join back to main table.      
       (SELECT D.CustomerID,MIN(RowNumber) AS Rank
        FROM (SELECT A.CustomerID,
        (SELECT COUNT(*) FROM TRANK AS B
          WHERE (B.CustomerID<A.CustomerID) 
                 OR ((B.CustomerID=A.CustomerID) and (B.OrderID<A.OrderID)))+1
                                                                        AS RowNumber
        FROM TRANK AS A) AS D
        GROUP BY D.CustomerID) AS F
            ON G.CustomerID=F.CustomerID
ORDER BY G.CustomerID,G.OrderID

OrderID     CustomerID ShipCountry Freight Rank
----------- ---------- ----------- ------- ----
10259       CENTC      Mexico      3.25    1
10268       GROSR      Venezuela   66.29   2
10785       GROSR      Venezuela   1.51    2
10482       LAZYK      USA         7.48    4
10545       LAZYK      USA         11.92   4
10517       NORTS      UK          32.07   6
10752       NORTS      UK          1.39    6
11057       NORTS      UK          4.13    6
10738       SPECD      France      2.91    9
10907       SPECD      France      9.19    9
10964       SPECD      France      87.38   9
11043       SPECD      France      8.80    9

But you're probably wondering why do something like this when you can use a subquery
directly for Rank? Yeah, practically speaking it's silly but the idea of how row number
and Rank are related like this is part of the puzzle of what lies behind the ranking
functions in sql. Just because you haven't seen this in a textbook (or anyplace else)
doesn't mean it's not relevant. ☺ Lets continue.

Ok here is the data using the sql ranking functions. It's important to realize that
ROW_NUMBER will always create unique values no matter how you sort the data. On the
other hand RANK is sensitive to how you order the data. You'll get duplicate RANKs
if there are duplicate values of the ORDER BY column(s). This is because RANK counts
prior duplicate values (ie CustomerID) as if they were unique. And this can lead to
RANK values skipping around. ☺ But again, bear in mind that Rank can be derived from
row numbers as shown above.

Example sql ranking functions query:

SELECT OrderID,CustomerID,ShipCountry,Freight,
-- ROW_NUMBER guarantees unique ranks regardless of whether or not the ORDER BY colum(s) 
-- are a primary key of the table. CustomerID,OrderID is a primary key, CustomerID 
-- is not. We assume that when ordering by just CustomerID that within duplicate
-- CustomerIDs the ordering is non deterministic but always results in unique row 
-- number values.
ROW_NUMBER() OVER (ORDER BY CustomerID) AS RowNumber1,
ROW_NUMBER() OVER (ORDER BY CustomerID,OrderID) AS RowNumber2,
-- RANK guarantees unique values only if the ORDER BY column(s) are a primary key, 
-- ie. if the ORDER BY column(s) value(s) target only unique rows then the ranks are
-- unique. CustomerID,OrderID is a primary key of the table so the RANKs are unique.
RANK() OVER (ORDER BY CustomerID,OrderID) AS UniqueRank,
-- RANKing by a non primary key (CustomerID) generates duplicate values.
RANK() OVER (ORDER BY CustomerID) AS Rank,
-- DENSE RANK returns consecutive but repeating ranks for a non primary key (CustomerID).
DENSE_RANK() OVER (ORDER BY CustomerID) AS DenseRank
FROM TRANK
ORDER BY CustomerID,OrderID

OrderID     CustomerID ShipCountry Freight RowNumber1 RowNumber2 UniqueRank Rank DenseRank
----------- ---------- ----------- ------- ---------- ---------- ---------- ---- ---------
10259       CENTC      Mexico      3.25    1          1          1          1    1
10268       GROSR      Venezuela   66.29   2          2          2          2    2
10785       GROSR      Venezuela   1.51    3          3          3          2    2
10482       LAZYK      USA         7.48    4          4          4          4    3
10545       LAZYK      USA         11.92   5          5          5          4    3
10517       NORTS      UK          32.07   8          6          6          6    4
10752       NORTS      UK          1.39    6          7          7          6    4
11057       NORTS      UK          4.13    7          8          8          6    4
10738       SPECD      France      2.91    12         9          9          9    5
10907       SPECD      France      9.19    9          10         10         9    5
10964       SPECD      France      87.38   10         11         11         9    5
11043       SPECD      France      8.80    11         12         12         9    5

The ranking functions overcome the two big obstacles of coding them as a subquery,
they're easier and they perform much better because they don't involve the
potentially terribly high number of comparisons involving the (>,<) predicates.

Now I'm pretty sure you gathered how to use ranking functions from BOL. I'm also
pretty sure you gained minimal insight about them from BOL. The documentation
is a really botched job (perhaps I'll go into detail in another article. ☺ Suffice
it to say, for example, that OVER is not a clause but a keyword indicating that what
follows it is an OLAP (ie. ranking) function. What follows OVER is a clause, a 
WINDOW clause consisting of at least an order (ORDER BY) specification. You can
take a step up from BOL by going to the sql standard from which ranking functions
came to gleam a little more about them:

ISO/ANSI: Introduction to OLAP functions (5/99)
http://tinyurl.com/2taahc

Bear in mind that given the ranking function:

RANK() OVER (ORDER BY CustomerID) AS Rank 

The WINDOW is '(ORDER BY CustomerID)'. Now consider what the standard says about 
the ordering (ORDER BY):

"Ordering in windows is specified with the same <sort specification list> used by
 cursors, and with the same semantics."  
"Whether in a cursor or a window, a <sort specification list> specifies an ordering
 of rows. The difference is that, in a cursor, the ordering determines the sequence
 to present rows during sequential fetches. In a window, the ordering helps to
 determine the value of <OLAP function>s."
"The user can copy the <order specification list> from a cursor ORDER BY to a window
 ORDER BY, or the reverse, with precisely the same semantics in each context."
 
If it quacks like a duck, swims like a duck and walks like a duck there's pretty good
reason to believe it's at least related to the idea of a duck. ☺ The concept of a
cursor is embedded in a WINDOW, it's embedded in the ranking functions! The idea
of OVER implies over a table just like you would declare a cursor 'over' a table.

ROW_NUMBER()
------------
 
Now lets take a ranking function from the sql query above:

ROW_NUMBER() OVER (ORDER BY customerID) AS RowNumber1

and come at it relationally using dataphor. First lets define a cursor using the
TRANK table ordered by CustomerID:

cursor(TRANK order by {CustomerID}) *

This encapsulates a specific order of the rows in a different type than a table, a 
cursor. After all, sql users know that a table by definition has no order so the
order that the table can't have is now in the cursor. Cursor, OVER (WINDOW) the same
thing? ☺ Now how can we get at the order of rows represented by the cursor? How can
we get discrete values of the cursors order? Well we can transform the cursor into a
list type. The order of each row in the cursor will be represented by the ordinal
position (an integer) the row has in the list. You explicitly convert the cursor
using the ToList operator.

ToList(cursor(TRANK order by {CustomerID}))

Kewl, but again we face the decision of just how we get at the list. We could
obviously procedurally (loop) thru it and access each row of the list. But what is
much better, we can let the system do it non procedurally. We can transform each row
of the list, which includes the integer value representing the position in the
list of each row, into a row of a table! Then we can simply use a SELECT query with
the transformed list. So lets transform the list to a table using the ToTable operator.

select ToTable(ToList(cursor(TRANK order by {CustomerID}))) 
                               order by {CustomerID,OrderID};
                               
OrderID CustomerID ShipCountry Freight sequence 
------- ---------- ----------- ------- -------- 
10259   CENTC      Mexico      $3.25   0        
10268   GROSR      Venezuela   $66.29  1        
10785   GROSR      Venezuela   $1.51   2        
10482   LAZYK      USA         $7.48   3        
10545   LAZYK      USA         $11.92  4        
10517   NORTS      UK          $32.07  5        
10752   NORTS      UK          $1.39   6        
11057   NORTS      UK          $4.13   7        
10738   SPECD      France      $2.91   8        
10907   SPECD      France      $9.19   9        
10964   SPECD      France      $87.38  10       
11043   SPECD      France      $8.80   11   

The rows represented in the list are now rows in a table. Additionally we get the
auto generated sequence column, the ordinal position each row had in the list 
defined by the sort order of the cursor. Renaming sequence to Index (to indicate
the column comes from an indexed list) and adding 1 to each Index value we have:

select ToTable(ToList(cursor(TRANK order by {CustomerID})),'A_row','Index') 
                                          redefine {Index:=Index+1}
                                            order by {CustomerID,OrderID};
                                           
OrderID CustomerID ShipCountry Freight Index 
------- ---------- ----------- ------- ----- 
10259   CENTC      Mexico      $3.25   1     
10268   GROSR      Venezuela   $66.29  2     
10785   GROSR      Venezuela   $1.51   3     
10482   LAZYK      USA         $7.48   4     
10545   LAZYK      USA         $11.92  5     
10517   NORTS      UK          $32.07  6     
10752   NORTS      UK          $1.39   7     
11057   NORTS      UK          $4.13   8     
10738   SPECD      France      $2.91   9     
10907   SPECD      France      $9.19   10    
10964   SPECD      France      $87.38  11    
11043   SPECD      France      $8.80   12    

This should look familiar, starting with a cursor over the table (TRANK) ordered by
CustomerID and then transformed to a list then transformed back to a table we arrive  
at the same values for Index created from: 

ROW_NUMBER() OVER (ORDER BY customerID) AS RowNumber1 (back to ranking functions query)

Conceptually the transformation between relational types is the red meat underlying
ROW_NUMBER (and all the other ranking functions). This is but another benefit of
working with and understanding types and their relationships in a relational system. 
(Note that there is no such thing as implicit conversion in dataphor as there is in 
 sql. Explicit vs. implicit conversion is a distinguishing characteristic between
 relational and sql systems.)
 
Note that cursors in Sql Server are viewed exclusively as an access method. The idea
of an sql cursor is synonymous with the idea of a procedural approach to data access as 
opposed to a set (non procedural) approach using a query. The type of action the cursor
implies is the primary focus while the representation of rows of a table in a particular
order is secondary. Relationally, the idea of representation of a table in an order is
a primary concept. It's the primary concept of the cursor type. It's within the context
of the relationship between types that the cursor is elevated in a relational system. 
(It's certainly the case that a relational cursor can be used to fetch rows sequentially
just like in sql (see here). But the importance of a cursor as an order representation
of table rows does not exist in sql as it does in a relational system. This hole in
the cursor concept in sql is based on the fundamental idea that sql is essentially a
typeless system compared to a relational one. What can one do with a cursor in sql 
except fetch from it, use it to retrieve data? On the other hand types are the
foundation of a relational system. And understanding a relational system comes thru
types.) 

RANK()
------

From the sql above featuring the ranking functions (go there) now lets look at the
expression:

RANK() OVER (ORDER BY CustomerID) AS Rank

Now, unlike ROW_NUMBER, there's no direct relational transformations we can make to
produce RANKs. But lets go back to the sql subquery methods for RowNumber and Rank
(go there). Here again is the result:


OrderID     CustomerID ShipCountry Freight RowNumber Rank
----------- ---------- ----------- ------- --------- ----
10259       CENTC      Mexico      3.25    1         1   <|-- Min RowNumber
10268       GROSR      Venezuela   66.29   2         2    <|-- Min RowNumber
10785       GROSR      Venezuela   1.51    3         2     |   
10482       LAZYK      USA         7.48    4         4     <|-- Min RowNumber
10545       LAZYK      USA         11.92   5         4      |
10517       NORTS      UK          32.07   6         6      <|-- Min RowNumber
10752       NORTS      UK          1.39    7         6       |
11057       NORTS      UK          4.13    8         6       |
10738       SPECD      France      2.91    9         9       <|-- Min RowNumber
10907       SPECD      France      9.19    10        9        |
10964       SPECD      France      87.38   11        9        |
11043       SPECD      France      8.80    12        9        | 

I previously introduced the idea that RANKs could easily be derived from ROW_NUMBERs.
The RANKs for CustomerID are the minimum ROWNUMBERs within CustomerID (review query here).
And we can use the same idea in dataphor. Here we show that using a group by with
CustomerID returns the Ranks:

select ToTable(ToList(cursor(TRANK {OrderID,CustomerID} order by {CustomerID})),'Arow','Index')
       group by {CustomerID} add{Min(Index) Rank} redefine {Rank:=Rank+1}

CustomerID Rank 
---------- ---- 
CENTC      1    
GROSR      2    
LAZYK      4    
NORTS      6    
SPECD      9  

Note that the cursor is over the TRANK table with only the columns OrderID and 
CustomerID. We include OrderID to guarantee that all CustomerIDs are in the table. 
Without OrderID (ie. only CustomerID) dataphor would only include unique values of 
CustomerID in the cursor (dataphor will never allow duplicate rows in a table).
In other words, we want all CustomerID values (or perhaps more correctly, all rows)
to be in the list, we don't want to eliminate duplicates.

And like in the sql query we can simply join the Ranks to the TRANK table and return
all the data.

select 
//Join TRANK table to table of CustomerID and Rank.
  TRANK
    join //Using a natural join on CustomerID.
     (
      //Use group by to get minimum Index for each CustomerID. 
      //Add 1 to Index to set minimum rank to 1. 
      ToTable(ToList(cursor(TRANK {OrderID,CustomerID} order by {CustomerID})),'Arow','Index')
       group by {CustomerID} add{Min(Index) Rank} redefine {Rank:=Rank+1}
     )
       order by {CustomerID,OrderID};

OrderID CustomerID ShipCountry Freight Rank 
------- ---------- ----------- ------- ---- 
10259   CENTC      Mexico      $3.25   1    
10268   GROSR      Venezuela   $66.29  2    
10785   GROSR      Venezuela   $1.51   2    
10482   LAZYK      USA         $7.48   4    
10545   LAZYK      USA         $11.92  4    
10517   NORTS      UK          $32.07  6    
10752   NORTS      UK          $1.39   6    
11057   NORTS      UK          $4.13   6    
10738   SPECD      France      $2.91   9    
10907   SPECD      France      $9.19   9    
10964   SPECD      France      $87.38  9    
11043   SPECD      France      $8.80   9

Like the sql RANK, it is an easy query with no row by row comparisons as in an sql
subquery. And it all starts with introducing an order of rows in a table thru a
cursor type.

(For those burning with curiosity go here for a procedural solution for the RANKs
in dataphor.)

DENSE_RANK()
------------

Now lets turn our attention to the dense rank expression in the sql query (go there):

DENSE_RANK() OVER (ORDER BY CustomerID) AS DenseRank

Dense ranks are consecutive integers that increment only for unique values of the
ORDER BY column(s). They remain the same for duplicate values. Therefore they're
based on distinct values. Before ranking functions it was common to write a subquery
using COUNT(DISTINCT column). We could get a dense rank on CustomerID with:

SELECT A.OrderID,A.CustomerID,A.ShipCountry,A.Freight,
(SELECT COUNT(DISTINCT B.CustomerID) FROM TRANK AS B WHERE B.CustomerID<=A.CustomerID) AS DenseRank
FROM TRANK AS A
ORDER BY CustomerID,OrderID

OrderID     CustomerID ShipCountry Freight DenseRank
----------- ---------- ----------- ------- ---------
10259       CENTC      Mexico      3.25    1
10268       GROSR      Venezuela   66.29   2
10785       GROSR      Venezuela   1.51    2
10482       LAZYK      USA         7.48    3
10545       LAZYK      USA         11.92   3
10517       NORTS      UK          32.07   4
10752       NORTS      UK          1.39    4
11057       NORTS      UK          4.13    4
10738       SPECD      France      2.91    5
10907       SPECD      France      9.19    5
10964       SPECD      France      87.38   5
11043       SPECD      France      8.80    5

How do we translate the sql idea of counting distinct values of CustomerID to
the context of relational types to get the dense rank? It's actually quite simple. 
First we create a table with only unique values of CustomerID. Then we transform 
the unique CustomerID rows into a list (with an ascending order). In the list the
ordinal position of each row (a row is just CustomerID column) is the dense rank! ☺ 
Then we can transform the list back to a table which includes the index of the list
as a column (DenseRank) in the table.

select ToTable(ToList(cursor(TRANK {CustomerID} order by {CustomerID})),'CustomerID','Index')
                {CustomerID,Index,Index+1 DenseRank};

CustomerID Index DenseRank 
---------- ----- --------- 
CENTC      0     1         
GROSR      1     2         
LAZYK      2     3         
NORTS      3     4         
SPECD      4     5         

We can join the above table with CustomerID and DenseRank to the full TRANK table
to return all the data.

select 
//Join TRANK table to table of CustomerID and DenseRank.
  TRANK
    join //Using a natural join on CustomerID.
     (
      //Get the dense ranks from the ordinal position that each CustomerID has in the list.
       ToTable(ToList(cursor(TRANK {CustomerID} order by {CustomerID})),'CustomerID','Index')
      //Add 1 to Index to set minimum DenseRank to 1. 
                     {CustomerID,Index+1 DenseRank}
     )
       order by {CustomerID,OrderID};

OrderID CustomerID ShipCountry Freight DenseRank 
------- ---------- ----------- ------- --------- 
10259   CENTC      Mexico      $3.25   1         
10268   GROSR      Venezuela   $66.29  2         
10785   GROSR      Venezuela   $1.51   2         
10482   LAZYK      USA         $7.48   3         
10545   LAZYK      USA         $11.92  3         
10517   NORTS      UK          $32.07  4         
10752   NORTS      UK          $1.39   4         
11057   NORTS      UK          $4.13   4         
10738   SPECD      France      $2.91   5         
10907   SPECD      France      $9.19   5         
10964   SPECD      France      $87.38  5         
11043   SPECD      France      $8.80   5     

Make sense? I bet you had no idea of what was behind the ranking functions. ☺
The ranking functions turn out to be a very special type of function. The sql
ranking functions may give a prelude as far as insight into them. I call this
prelude an overture (giving the keyword OVER in ranking functions its due). 
But relational ideas and specifically types take up where sql leaves off. ☺


Where DENSE_RANK fails
---------------------

Consider the following problem described here:

Identifying Sections 
By:Itzik Ben-Gan 
http://www.sqlmag.com/Article/ArticleID/95912/sql_server_95912.html
(You can download sql solutions.)

Given the following table:

CREATE TABLE dbo.T1
(
 id  INT NOT NULL PRIMARY KEY,
 val VARCHAR(10) NOT NULL
);

INSERT INTO dbo.T1(id, val) VALUES( 1, 'a');
INSERT INTO dbo.T1(id, val) VALUES( 2, 'a');
INSERT INTO dbo.T1(id, val) VALUES( 3, 'a');
INSERT INTO dbo.T1(id, val) VALUES( 5, 'a');
INSERT INTO dbo.T1(id, val) VALUES( 7, 'b');
INSERT INTO dbo.T1(id, val) VALUES( 9, 'b');
INSERT INTO dbo.T1(id, val) VALUES(11, 'a');
INSERT INTO dbo.T1(id, val) VALUES(13, 'a');
INSERT INTO dbo.T1(id, val) VALUES(17, 'b');
INSERT INTO dbo.T1(id, val) VALUES(19, 'b');
INSERT INTO dbo.T1(id, val) VALUES(23, 'b');
INSERT INTO dbo.T1(id, val) VALUES(29, 'a');
INSERT INTO dbo.T1(id, val) VALUES(31, 'b');
INSERT INTO dbo.T1(id, val) VALUES(37, 'b');

SELECT * FROM T1

id   val
---- ----
1    a
2    a
3    a
5    a
7    b
9    b
11   a
13   a
17   b
19   b
23   b
29   a
31   b
37   b

How do you get a dense rank for val in the order of id? The solution would look
like this:

id   val  DenseRank
---- ---- ---------
1    a    1
2    a    1
3    a    1
5    a    1         | change 1
7    b    2         |
9    b    2          | change 2
11   a    3          |
13   a    3           | change 3 
17   b    4           |
19   b    4
23   b    4            | change 4
29   a    5            |
31   b    6             | change 5
37   b    6             |


In this case the dense rank is certainly not based on distinct values of val. It's 
based on the number of changes between the values of val, ie. from 'a->b' or 'b->a'
in the direction of id (which is ascending). Over the ascending values of id there
are 5 changes between val values. The dense rank is incremented on every change
regardless of the prior and current values of val as opposed to the distinct values
of val (of which there are only 2). So the number of changes + 1 = number of distinct
dense rank values. Obviously we cannot use the DENSE_RANK function to encapsulate this
rank. The DENSE_RANK is for the ORDER BY column(s) (and PARTITION BY column(s) if
they exist). For example we can try various combinations of columns to sort by:

SELECT id ,val,
DENSE_RANK()OVER(ORDER BY val)    AS DenseRank1,
DENSE_RANK()OVER(ORDER BY val,id) AS DenseRank2
from t1
order by id

id   val  DenseRank1 DenseRank2
---- ---- ---------- ----------
1    a    1          1
2    a    1          2
3    a    1          3
5    a    1          4
7    b    2          8
9    b    2          9
11   a    1          5
13   a    1          6
17   b    2          10
19   b    2          11
23   b    2          12
29   a    1          7
31   b    2          13
37   b    2          14

From the point of view of a cursor we can't encapsulate this change, which is both
ascending and descending, for the ORDER BY column(s). And this is why a single 
DENSE_RANK function cannot be used to solve this kind of problem. The sql ranking
functions reflect only a single cursor. To capture the change multiple cursors
are required or multiple expressions in the case of sql. (You can download sql
solutions offered by Itzik Ben-Gan for the problem described above.)

For an example of a relational solution using multiple cursors, ToList and ToTable see:
Creating a Super Function
http://beyondsql.blogspot.com/2007/08/dataphor-creating-super-function.html
(This solution also emphasizes the relational idea of passing a table as a
 real parameter just like an integer or string.)

For another relational approach see:
Using a dense rank for identifying sections
http://beyondsql.blogspot.com/2007/06/sql-using-dense-rank-for-identifying.html

For a solution exclusively in sql using the RAC utility see:
RAC - Rank This!
http://beyondsql.blogspot.com/2006/09/rac-rank-this.html

I hope you found this article interesting and I hope you'll be stimulated to explore
the relational world of dataphor. ☺


A procedural solution in dataphor for obtaining RANKs for CustomerID.

//Beginning of batch statements.
//Define a list of type string to hold scalar values of CustomerID.
var LList:=list(String){};
//Create a cursor variable ordering by just CustomerID.
var LCursor:=cursor(TRANK order by {CustomerID});
//Create a virtual table to hold the result, ie. table TRANK and a column for the rank (Rank)
//of each CustomerID.
var NewTable:=table of typeof(TRANK add {nil as Integer Rank}){};
var Dummy:Integer;
//Sequentially fetch the cursor rows by CustomerID.
while LCursor.Next() do
begin  
//LCursor.Select().CustomerID is the current scalar value of CustomerID (in the cursor).
//Add the CustomerID value to the list.
Dummy:=LList.Add(LCursor.Select().CustomerID); 
//LCursor.Select() is the current row in the cursor, all columns and their values.
//Insert the current row plus the rank into NewTable.
//The first index in the list for the current CustomerID is the Rank! 
//In other words, the 1st (minimum) ordinal position for a CustomerID+1=rank.
//The list operator IndexOf returns the minimum ordinal position for a list value. 
//In this case for a value of CustomerID. 
insert (LCursor.Select() add{LList.IndexOf(LCursor.Select().CustomerID)+1 Rank}) into NewTable;
end;
select NewTable order by {CustomerID,OrderID};
//End of batch statements.  
  

OrderID CustomerID ShipCountry Freight Rank 
------- ---------- ----------- ------- ---- 
10259   CENTC      Mexico      $3.25   1    
10268   GROSR      Venezuela   $66.29  2    
10785   GROSR      Venezuela   $1.51   2    
10482   LAZYK      USA         $7.48   4    
10545   LAZYK      USA         $11.92  4    
10517   NORTS      UK          $32.07  6    
10752   NORTS      UK          $1.39   6    
11057   NORTS      UK          $4.13   6    
10738   SPECD      France      $2.91   9    
10907   SPECD      France      $9.19   9    
10964   SPECD      France      $87.38  9    
11043   SPECD      France      $8.80   9 


* This is a valid expression in dataphor. It can be used in a query or as the
  definition of a variable, ie.

  var LCursor:=cursor(TRANK order by {CustomerID})

  where the LCursor variable inherits its type, cursor, from the expression.

Dataphor - SET-ing your query in ORDER

2008-04-10T02:03:00.000-07:00

Wanna have some fun, how would you like to build a list anywhere in a query? When you work with
Dataphor you can think in terms of order in the mist of thinking in terms of sets. ☺

Here's how it works. When you Add an item to a list the operation returns an integer
indicating the ordinal position the item holds in the list. Add is therefore like any
other expression that returns a result (remember that Add adds one item after another in
sequence). Well you can place this expression anywhere you could place any other expression. 
That's it! ☺ For example, using an integer list:

var Item:=10;
var LList1:=list(Integer){}; //An empty list of type integer.
var Index:=Add(LList1,Item); //Index is of type integer, the returned type of the expression
                             //that inserts the value of 10 into the list.
                             //Note we can also use the expression: LList1.Add(Item).
select Index;    //Returns 0, the position of the 1st item in the list.
select LList1[0];//Returns 10, the value of the 1st item in the list.

Using a list with a row type:

var LList1:=list(row{OrderID:Integer,CustomerID:String,Freight:Money}){};
var Item:=row{1 OrderID,'STEVE' CustomerID,$100.50 Freight};
var Index:=Add(LList1,Item); //Index holds the value 0, the position of the 1st row 
                             //added to the list.
select Index; //Returns 0, the position of the 1st item (row) in the list.
select LList1[0];
/* Returns the 1st item in the list, the row:
OrderID CustomerID Freight 
------- ---------- ------- 
1       STEVE      $100.50 
*/

Remember that the Add operation with list adds items sequentially to the list starting at position 0.

OK, here's a contrived example query. It uses Sql Server 2005, the Northwind database and the Orders table.
We're using three lists where a list item is a row of the Orders table limited to only three columns
(one of which, OrderID, is the primary key of the table). In broad strokes we're looking at how relational
expressions access data. The list, after all, reflects how the data was accessed, the sequential order
in which rows from the table were processed depending on where we're at in the query. In this sense
we're introducing two independent ideas in a single operation, thinking in terms of sets and thinking
in terms of order (sequentially) that do not conflict with each other. ☺

The query first gets the 10 highest Freight values for all Freight values over $400. It then gets
the lowest 5 freights from the 10 highest ones.

var LList1:=list(typeof(Orders[]{OrderID,CustomerID,Freight})){};
var LList2:=list(typeof(Orders[]{OrderID,CustomerID,Freight})){};
var LList3:=list(typeof(Orders[]{OrderID,CustomerID,Freight})){};
//Note the "with {IgnoreUnsupported = 'true'}" in the query is used so you don't get a warning from the 
//dataphor compiler about sql server not supporting the list.Add operation. Obviously dataphor can't
//hand off the list operation to sql server so the operation is under dataphors control.

select
     ( 
      (
        Orders where Freight>$400 {OrderID,CustomerID,Freight} 
         add{LList1.Add(Orders[OrderID]{OrderID,CustomerID,Freight} with {IgnoreUnsupported = 'true'}) R1} 
          return 10 by {Freight desc} with {IgnoreUnsupported = 'true'}
          //return N is similar to sql server 'Select Top N..Order By'.
       ) 
         add{LList2.Add(Orders[OrderID]{OrderID,CustomerID,Freight} with {IgnoreUnsupported = 'true'}) R2}
          return 5 by {Freight}
      )       
         add{LList3.Add(Orders[OrderID]{OrderID,CustomerID,Freight} with {IgnoreUnsupported = 'true'}) R3}          
          order by {Freight};
          
/*  Query result ordered by Freight.
 R1, R2 and R3 are the ordinal positions from the three lists created from the query.
 See the individual lists below for details.
OrderID CustomerID Freight R1 R2 R3 
------- ---------- ------- -- -- -- 
11032   WHITC      $606.19 19 9  0  
10983   SAVEA      $657.54 16 8  1  
10479   RATTC      $708.95 2  7  2  
10816   GREAL      $719.78 9  6  3  
11017   ERNSH      $754.26 17 5  4  
*/

Using ToTable we can transform a list to a table where the ordinal position of the item (ie. row) in 
the list is converted to a value. Of course we could access the list with a cursor/loop and use
an indexer to get each individual item (ie. LList1[0]). Using ToTable is more straighforward for our
purposes. And remember list->in terms of position, table->in terms of column(s) value. ☺

ToTable automatically creates a column named sequence indicating the ordinal position (of the row).
We're renaming it to Index.

select ToTable(LList1) rename {sequence Index} order by {Index};      
/* List1 result ordered by the ordinal position (Index) of each row. 
   LList1 indicates rows in the query are added to the list in the order of the primary key
  (OrderID) values.
OrderID CustomerID Freight   Index 
------- ---------- --------- ----- 
10372   QUEEN      $890.78   0     
10430   ERNSH      $458.78   1     
10479   RATTC      $708.95   2     
10514   ERNSH      $789.95   3     
10540   QUICK      $1,007.64 4     
10612   SAVEA      $544.08   5     
10633   ERNSH      $477.90   6     
10634   FOLIG      $487.38   7     
10691   QUICK      $810.05   8     
10816   GREAL      $719.78   9     
10836   ERNSH      $411.88   10    
10841   SUPRD      $424.30   11    
10847   SAVEA      $487.57   12    
10897   HUNGO      $603.54   13    
10912   HUNGO      $580.91   14    
10941   SAVEA      $400.81   15    
10983   SAVEA      $657.54   16    
11017   ERNSH      $754.26   17    
11030   SAVEA      $830.75   18    
11032   WHITC      $606.19   19    
*/

select ToTable(LList2) rename {sequence Index} order by {Index}; 
/* List2 result ordered by the ordinal position (Index) of each row. 
   LList2 indicates rows are added to the list in the order of descending Freight values.
OrderID CustomerID Freight   Index 
------- ---------- --------- ----- 
10540   QUICK      $1,007.64 0     
10372   QUEEN      $890.78   1     
11030   SAVEA      $830.75   2     
10691   QUICK      $810.05   3     
10514   ERNSH      $789.95   4     
11017   ERNSH      $754.26   5     
10816   GREAL      $719.78   6     
10479   RATTC      $708.95   7     
10983   SAVEA      $657.54   8     
11032   WHITC      $606.19   9      
*/
  
select ToTable(LList3) rename {sequence Index} order by {Index};        
/* List3 result ordered by the ordinal position (Index) of each row.
   Rows are added to the list in the order of ascending Freight values. 
   The order of the list agrees with the result of the query.
OrderID CustomerID Freight Index 
------- ---------- ------- ----- 
11032   WHITC      $606.19 0     
10983   SAVEA      $657.54 1     
10479   RATTC      $708.95 2     
10816   GREAL      $719.78 3     
11017   ERNSH      $754.26 4     
*/

You can take this idea and really run with it. ☺ Use it to help explain how a query builds an expression,
how indexes actually influence table access to all kinds of list processing driven by a query. When
you combine the ability to build a list with other list operations like IndexOf, IndexOfAny and LastIndexOf
you can do some kewl things. Actually this simple introduction is just the tip of the iceberg when it
comes to combining these two types.

Oh yeah, note that you can conditionally build a list(s) within a query using a case (or if-then-else) statement.
For example:

var LList1:=list(typeof(Orders[]{OrderID,CustomerID,Freight})){};
select
 Orders where Freight>$700  
  add  //Build a list based on a further restriction on freight. 
     {
      case when Freight>$800 then LList1.Add(Orders[OrderID]{OrderID,CustomerID,Freight} with {IgnoreUnsupported = 'true'}) 
      else -1 end
      R1
     } 
       {OrderID,CustomerID,Freight,R1}
         order by {R1,OrderID};
         
/* Query result.
OrderID CustomerID Freight   R1 
------- ---------- --------- -- 
10479   RATTC      $708.95   -1 
10514   ERNSH      $789.95   -1 
10816   GREAL      $719.78   -1 
11017   ERNSH      $754.26   -1 
10372   QUEEN      $890.78   0  
10540   QUICK      $1,007.64 1  
10691   QUICK      $810.05   2  
11030   SAVEA      $830.75   3  
*/
       
select ToTable(LList1) rename {sequence Index} order by {Index};   
/*
OrderID CustomerID Freight   Index 
------- ---------- --------- ----- 
10372   QUEEN      $890.78   0     
10540   QUICK      $1,007.64 1     
10691   QUICK      $810.05   2     
11030   SAVEA      $830.75   3     
*/

Note that other operations with the list type like insert and remove don't return a value so you can't
use them in a query (yet). ☺

More to come on lists.

Dataphor - Get a copy of Dataphor

2008-04-01T20:09:00.000-07:00

Database Consulting Group has acquired Alphora and Dataphor:
www.dataphor.org

You can get the latest full evaluation (commerical) version of Dataphor here:
http://databaseconsultinggroup.com/downloads/

Questions on Dataphor can be asked here:
http://www.databaseconsultinggroup.com/forums/

For information about the open source version of Dataphor see:
www.dataphor.org

Sql - What does it mean to set a value?

2008-03-31T20:05:00.000-07:00

Sql - What does it mean to set a value?

Using t-sql in Sql Server it's perfectly acceptable to use a SET statement to set a value
for a variable. For example, this batch sets the variable @MyRows to the number of rows in
the Orders table from the Northwind sample database:

DECLARE @MyRows int;
SET @MyRows = (SELECT COUNT(*) FROM Orders);
PRINT LTRIM(Str(@MyRows)); -- Returns 830

Now this operation makes perfect sense. Use an aggregate function (count) to derive a scalar
value and set it to a variable.

In the relational language of Dataphor, D4, the same operation can be expressed as:

var MyRows:Integer:=Count(Orders);

In D4 we can confirm that the expression Count(Orders) returns a scalar value of type
integer by using the Clintonian 'is' construct which tests for a particular type and
returns a boolean (true/false).

select Count(Orders) is Integer;//Returns True

We can also dispense with the Integer declaration and allow the integer type for variable
MyRows to be derived from the expression:

var MyRows:=Count(Orders);

Now lets suppose we want to set a value based on the value of a column in the Orders table
for a particular row. The following batch makes use of OrderID as a primary key and targets
a particular row for its Freight value:

DECLARE @MyFrt MONEY;
SET @MyFrt = (SELECT Freight FROM Orders WHERE OrderID=10249);
PRINT CAST(@MyFrt AS VARCHAR(8)); -- Returns 11.61

So the same operation that sets an aggregate value, a value derived about a table, also
sets a value from within a table. This state of affairs seems to be accepted without
question by sql folks. Well lets take a closer look at this from a relational perspective.
The closest D4 construction to the sql batch would be:

var MyFrt:=(Orders where OrderID=10249 {Freight});

And if the D4 operation is the same as the sql operation the MyFrt variable should be a
scalar value of type Money.

var MyFrt:=(Orders where OrderID=10249 {Freight});
select MyFrt is Money //Returns False.

Of course MyFrt isn't a scalar, it's a table!

var MyFrt:=(Orders where OrderID=10249 {Freight});
select MyFrt;

Freight //A table with a single row and column.
-------
$11.61  

select MyFrt is table{Freight:Money}; //Returns True confirming MyFrt is a table.

Variable MyFrt is a table with a single column named Freight with a type of Money. The table
happens to have only a single row but it is still very much a table! ☺ Now what could lie
between a table and a scalar value from within the table? There's only one possible thing,
a row ☺ What's needed is to extract out a row from the table, to transform the table type
to a row type. We do this with a thingie called a pure row extractor, '[]'.

var MyFrt:=(Orders where OrderID=10249 {Freight})[];
select MyFrt;

Freight //A row with a single column whose name is Freight and type is Money.
-------
$11.61

select MyFrt is row{Freight:Money}; //Returns True confirming MyFrt is a type of row.

Once we have zeroed in on a row we can get to a scalar value, a value of a column in the row.
We do this by using a column extractor which is the dot ('.') followed by the column name.
So to finally set the variable to the Freight value of the row we have:

var MyFrt:=(Orders where OrderID=10249 {Freight})[].Freight;
select MyFrt;

$11.61 //A scalar value of type Money.

select MyFrt is Money; //Returns True confirming the type of MyFrt is a scalar type (Money). 

We can set the value using a more succinct expression by using the primary key value (OrderID)
to directly extract the row (eliminating the WHERE clause) and then extracting the Freight value:

var MyFrt:=Orders[10249].Freight;
select MyFrt; //Return $11.61

Going back to the sql version we can even assign the Freight value to a string:

DECLARE @MyFrt varchar(10);
SET @MyFrt = (SELECT Freight FROM Orders WHERE OrderID=10249);
PRINT @MyFrt; -- Returns 11.61

What we have here is an example of three implicit conversions in one statement! To accept
the idea that sql is even relational like is to accept the implicit conversion of a
table to a row followed by a row to a scalar (money) followed by a money type to a string.
This of course is logically ridiculous. There is no such thing as an implicit conversion in
a relational system. There is no principle in any theory or in contemporary computer science
to derive a scalar value from a column(s) in a table without explicit transformations. There
is no logic to what sql is doing here only pure exigency. Which is what you get when essential
types like table and row are missing. In Sql when there are multiple rows that satisfy a
a WHERE clause the variable is left undefined and an error generated:

DECLARE @MyFrt MONEY;
SET @MyFrt = (SELECT Freight FROM Orders WHERE OrderID<=10249);
-- Subquery returned more than 1 value. This is not permitted when the subquery follows
-- =, !=, <, <= , >, >= or when the subquery is used as an expression.

This is really a valid logical operation, an attempt to set (assign) a table to a variable.
This is precisely what D4 does:

var MyFrt:=Orders where OrderID<=10249 {Freight};

select MyFrt;

Freight //A table.
-------
$11.61
$32.38

select MyFrt is table{Freight:Money};//Returns True confirming MyFrt is a table type.

The sql situation is nothing but logical confusion. The operation has nothing to do with
setting a scalar value. It is a table assignment. But sql doesn't know from such a thing
and makes up a spurious error. The so called aggregate concatenation query
(http://support.microsoft.com/default.aspx/kb/287515) is also illogical. That a query, in
the absence of a dedicated aggregate function (ie. concatenate), that returns a table is
supposed to behave like looping over a cursor and concatenating values to a variable is silly.

Perhaps the best that can be said of sql in situations like this is that results, even a
correct result, is based on whimsy instead of a logical foundation. But it's not hard to
understand the attraction of the whimsical nature of sql ☺ The prevalence of sql based on
the susceptibility of users to science fiction, as opposed to computer science, and the
spreading of the infection is why sql is the unchallenged viral language of databases.
Hopefully logic and common sense will halt its spread. But its arrest remains a challenge ☺

Is there really a table variable in sql server?

2008-03-04T20:55:00.000-08:00

Is there really a table variable in Sql Server 2005?
I say no! ☺

The following is a response to the authors comment in his blog post:

Louis Davidson (drsql)
2008: Declaring and instantiating a value
http://sqlblog.com/blogs/louis_davidson/archive/2008/03/01/2008-declaring-and-instantiating-a-value.aspx 

concerning his assertion of a table variable. As of now the author has not yet accepted my comment
for his blog ☺ You can read the full blog article for another interesting comment I made
that was accepted ☺

The authors (drsql) comment:

drsql said:

Steve (and yes it was repeated, but I just kept the last one),

I think you missed the point of the comment (I know I did when I first read it.)  It isn't :

DECLARE @CashMoney TYPE_OF(<table>);

it is

DECLARE @CashMoney TYPE_OF(<table>.<column>);

In other words, you have a table:

CREATE TABLE fred

(

   value varchar(10)

)

DECLARE @value  TYPE_OF(fred.value)

So what you would end up with would be a variable of type varchar(10).  Now do:

ALTER TABLE fred ALTER COLUMN value varchar(100)

When the code gets recompiled, @value would be varchar(100).

>>And all the connect feedback in the world won't change sql to be able to support a type of table variable.<<

We already have the concept of a table variable in 2005.  You can declare:

declare @fred table(value char(1))

Taking the type_of ide a bit further along these same lines though, you would might get to do:

DECLARE @fred  TYPE_OF(fred)

Which would declare a table variable with the structure of the fred table, rather than having to type it out again., 
but either way, you wouldn't be talking about just the type.  

Taking it one step further, you would be able to do:

DECLARE @fred  TYPE_OF(fred) = (select * from fred)

and get a copy of the table into the variable, without having to code the tedious bits of SQL.  

You cannot use a query like this on a table instantiation, but you can use a query on a variable 
instantiation, which I have updated the post to note.
March 3, 2008 10:18 AM 

Here is my response:

Hello Louis,
>I think you missed the point of the comment 
Correct, but my comment is still very relevant. The whole thrust of your comment was right on! 
And proves my point of the existence in your head of a relational programmer trying to pop out from sql☺
Now the idea of  'DECLARE @value  TYPE_OF(fred.value)' is really just a special case of the idea
of ' DECLARE @fred  TYPE_OF(fred) = (select * from fred)'. But the idea behind the ability to
have these type of constructs rests on the truth of this:
>We already have the concept of a table variable in 2005. 
But do we really have such a concept? Given two t-sql 'table variables':

declare @Fred table(value char(1))
insert @fred values ('A')
declare @Ethel table(value char(1))
insert @Ethel values ('C')

Now neither assignment nor comparison is possible:

set @Fred=@Ethel
if @Fred=@Ethel print 'Yes'

And the error message in both cases:
--Msg 137, Level 15, State 2, Line 11
--Must declare the scalar variable "@Fred".

underscores the idea that in the context of assignment and comparison only scalar values are
recognized. So now you could live with the idea that a table variable is a different beast 
from an integer or string which both support assignment and comparison. Two different kinds
of variables? But that's kind of crazy ☺ Now the only way for a compiler to perform assignment 
and do comparisons is to recognize the 'type' of variables involved. A variable can only be a 
variable of a specific type. And therein lies the answer. In declare @x int, we know int is the type.
In 
declare @fred table(value char(1))
the assumption is 'table (value char(1))' is the type just like integer is the type. Wrong! 
This is where we've been had, a big gotcha! ☺ It is not a type, if it was we could do 
assignments and comparisons. And even better @fred isn't even a variable! How could a variable
exist without a type? It may be some sort of reference/pointer but it's not a variable in any 
way recognized in cs. There is no concept of a table variable in t-sql and no such concept of 
table type in sql. So what you rightfully wish for 'DECLARE @value  TYPE_OF(fred.value)', 
'DECLARE @fred  TYPE_OF(fred) = (select * from fred)' makes no sense if there's is no type/variable
for a table. This is why there's no 'TYPE_OF' of in t-sql/sql. Type of what? ☺ What your after
are relational ideas like:

var Fred:typeof(Orders {OrderID,CustomerID,EmployeeID}):=
         Orders where ShipVia=3 {OrderID,CustomerID,EmployeeID}; 

which defines a variable of type table with a specific heading (columns/datatypes) and populates
it with rows of data (of the same type). The idea of setting a variable based on a column in a
table is based on recognizing a table type:

var Freds:typeof(Orders[].OrderID):=2;

Variable Fred is set to the type of scalar of OrderID, integer, and then set to a value. 
Welcome to my world ☺ I hope this all makes sense to you! 
  
best,
steve

Q & A about tables, types and procedures

2007-12-09T18:26:00.000-08:00

Here are some questions from an sql developer I'm trying to 
answer. I'm trying to explain that a relational database realizes 
a stored procedure in a fundamentally different way than sql. 
Along the way I'm talking about relational ideas as they relate
to tables and types. The conversation shows the vise-like grip
sql seems to have on the minds of man ☻ Microsofts functional
programming offering LINQ also comes up. Overall I think I did
a good job trying to explain the big differences between sql
and a relational database. There's some simple but good examples
that drive home the significant differences.
Black is relational(me)
Purple is sql

Here we go:

> My objection is not so much to your general idea of variables of
> type table-with-given-columns (I've recently worked with some systems
> that could be cleaner if such a thing were available; currently they
> work around it using temp tables); more to your specific use of D4 in
> all your examples, as opposed to a pseudo-code extension of SQL.

Ok, let me directly address you dislike of D4 and your preference for
a pseudo-code extension of SQL. I'll refer to your pseudo-syntax in
the thread:
comp.databases.ms-sqlserver
'Basic Anatomy of Sql Server'
http://tinyurl.com/2olako

>
Then you might want to write examples in a pseudo-syntax that
/looks/ like SQL.  I know this is a matter of taste, but your
examples look ugly to me.  Consider:

-- Your example of a stored procedure that returns a result set, the
-- format of which can only be deduced by reading through the code.

CREATE PROCEDURE dbo.GroupByShipCountry
       @Employee Integer
AS
SELECT ShipCountry,Count(*) Cnt,Min(Freight) MinFrt,Max(Freight)
MaxFrt
FROM Orders
WHERE EmployeeID=@Employee
GROUP BY ShipCountry

-- Your example of the same stored procedure rewritten in D4.

create operator GroupByShipCountry (Employee:Integer):
table{ShipCountry:String,Cnt:Integer,MinFrt:Money,MaxFrt:Money}
begin
result:=
       Orders
        where EmployeeID=Employee
         group by {ShipCountry}
           add{Count() Cnt,Min(Freight) MinFrt,Max(Freight) MaxFrt} ;
end;

-- My example of the same stored procedure rewritten in a
-- pseudo-extension of T-SQL.

CREATE PROCEDURE dbo.GroupByShipCountry
       @Employee Integer,
       @ResultSet Table (
             ShipCountry varchar(15),
             Cnt         int,
             MinFrt      money,
             MaxFrt      money
       ) output
AS
SELECT ShipCountry,
        Count(*) Cnt,
        Min(Freight) MinFrt,
        Max(Freight) MaxFrt
INTO @ResultSet
FROM Orders
WHERE EmployeeID=@Employee
GROUP BY ShipCountry
>

To begin with, the idea of a stored procedure returning a 'result' is
an sql concept. This concept does not exist in a relational (D4)
system. Relationally, a stored procedure only exists when it is
created. The  execution of a sp, its runtime realization, does not
involve the definition of the procedure nor the idea of 'returning'
something from it. Relationally at runtime what sql see's as a
procedure and a result 'is' a variable of the type of the result. This
is the huge difference between the two systems. Relationally the
'@ResultSet' and the idea of inserting a query result into it is
contradictory and meaningless. The 'name' of the procedure 'is' the
variable (table), there is no result from a sp (ie. sql). Syntactically
an sql tabled value function is closer in spirit to the D4 procedure
with the big difference that the name of the table valued function is
'not' a typed variable like in D4. Finally, the sql sp makes the
distinction between identifiers as variables and non-variables using
the '@'.  In D4 there is no such distinction as 'all' identitifiers
are by definition variables and the '@' is superfluous. The 'output'
declaration in the sql sp is based on the general sql idea of
'returning' something. Such a declaration is superfluous relationally
as, again. there is no concept of 'returning a something' from a 'this sp'.
Note that LINQ realizes an sql stored procedure exactly as sql intends
it and nothing like the relational D4. The 'functional' part of
integrated query is simply how the sp is accessed within a net
language. There is no concept of a typed variable with the name of the
sp. In other words, the OR map is mapping to the same sql as if the
mapping didn't exist. MS has added a 'functional language' within net
when what it should have done is added a functional language to the
database itself! ☺ D4, after all, represents the functional language
of a relational system and how easy it is to use such a language
within a present day (net) programming language. The D4 answer to
overcoming the object-relational mismatch is of a totally different
nature to the one offered thru LINQ. My objection to LINQ lies in the
idea that no one at MS seems to have considered an alternative. 

Let me try to cut thru the semantics with a few examples that I
hope will clarify some of the points I'm trying to make.

In t-sql this should be perfectly clear:
DECLARE @X INT
SET @X=5

The variable @X can only take one value at any specific time.
In a relational system a procedure that returns some value at runtime
must behave exactly like @X. At runtime the procedure is a variable of
a particular type and has a specific value based on input arguments.
An sql sp has no such nature and behaves in an entirely different way.

create procedure SqlOne
@Y int
AS
SELECT COUNT(*) AS CNT,SUM(FREIGHT) AS SUMFRT
FROM ORDERS
WHERE EMPLOYEEID=@Y
SELECT *
FROM ORDERS
WHERE EMPLOYEEID=@Y

Therefore the idea that an sql procedure can return multiple results
is meaningless if the sp is realized as a variable where only a single
result makes sense. Add to this the idea of type where each result is
a different type and the difference between sql and relational should
be even clearer. Again the relational procedure is realized exactly
like the int @X. No programming language chooses among possibe
multiple definitions of the value of a variable. It would be
equivalent to:
DECLARE @X INT
SET @X=5 or @X=10
which makes no sense. It is because the sql sp is not realized as a
variable that multiple results 'can' be returned.

This sql sp:
CREATE PROCEDURE SqlTwo
@A INT OUTPUT,
@B INT OUTPUT
AS
SET @A=5
SET @B=10

DECLARE @C INT,@D INT
EXEC SqlTwo @C OUTPUT,@D OUTPUT
SELECT @C
SELECT @D

makes no sense relationally because, again, there are multiple
results. Now there are two scalar types (int) returned instead of sql
'resultsets'. Relationally there is no such thing as more than 1 thing
(think a variable of a type) at a time. Two scalar results are
realized as a 'row' type relationally, ie. 'one' thing.
create operator D4Two():row(A:Integer,B:Integer)
begin
result:=row(5 A,10 B);
end;

In this case at runtime D4Two is a variable of type row with 2 scalar
columns.

From the relational perspective a table/row/list is a variable that
behaves exactly like a variable in a programming language. Its value
can be assigned to other values just like a t-sql integer variable
can. It can be compared to other variables (for equality) just like a t-sql
integer variable. It can be passed as an argument to a procedure just
like a t-sql integer variable. For these reasons why MS decided to
call something a 'table variable' remains a mystery. It behaves like no
other variable in any programming language on the face of the
planet ☺ If it doesn't quack like a duck, doesn't behave like a duck,
doesn't waddle like a duck it sure as hell isn't a duck ☺ What MS
calls a table variable is surely not a table variable as the idea
exists in any programming language or relationally. Whatever one wants
to call an sql table the table variable is the same thing. Its
phyiscally implementation may be different but that does not change
the fact it is not a variable of a specific table type.

Sql distinguishes between user defined functions and procedures. But
sql user defined functions are on the same exact level of procedures
when looked at from the point of view of 'variables'. Neither one
has anything to do with the idea of a relational variable. All this
artificial distinction does is serve to make it harder for users to
understand the relational model ☺ (Why sql choose to create a user
define function/procedure dichotomy is another topic. But think of
where and having).

Rather than dwell on particular syntax or pseudo-syntax I think it
is the ideas that the relational model is based on that is important.
And what we're talking about here is just a slice of the relational
model. The relational model is not rocket science ☺ It's actually
quiet straightforward. Ironically it's sql that is out in left field.
The relational model is in line with all current programming
languages. Unfortunately that's never been the case with sql ☺ This is 
one of the reasons I find LINQ so unnecessary. Once you get the idea
that a big part of the relational model is all about the basic concepts of
variables and types I think (I at least hope) that what I've been
trying to explain will make perfect sense ☺ 

>Add to this the idea of type where each result is a different type
>In other words, TABLE (X INT, Y INT) is a different variable type
>from TABLE (M VARCHAR(15), N VARCHAR(15))?

Exactly. Think of sql strings. This table, TABLE(TABLE (M VARCHAR(15),
N VARCHAR(15)),
is a differnt type than TABLE (N VARCHAR(16), N VARCHAR(16))! This
means that we couldn't compare the two and undermines real relational
division. To declare how many characters in a string is clearly the
opposite of what the relational idea of data independence is all
about. Relationally there can only be a 'string' type having
absolutely nothing to do with its storage characteristics.  And this
is the same idea in any programming language. This is just one
manifestation of how sqls design ignores the concept of a strong type.
LINQ is an attempt to hide the fact that:
DECLARE @N VARCHAR(5),@M VARCHAR(6),@P VARCHAR(5),@Q VARCHAR(6)
represents 4 different types. This is but one simple form of the idea
that sql guarantees impedence mismatch! And having the choice of
changing the database or the access to it, MS chose access to the
database (LINQ). The sql community seems to not consider how bizarre
and confounding things like this look to developers coming to sql for
the first time. It must be force of habit blinding a more critical
look at how things are. There are so many that sql defeats the idea of
strong types that it would be better and easier to build a new
database system for application development. And have sql available
for everything else ☺

>What if the data you want to return is not multiple scalars, but
>rather multiple tables?  Upon reflection, I suppose tables could
>be nested in this model, i.e. you can return
>   TABLE(T1 TABLE(X INT, Y INT), T2 TABLE(M VARCHAR(15), N VARCHAR(15)))

The relational database emphasizes types to define structures that can
be used to model processes, ie.

create table T1
{
  A:Integer,
  B:String,
  T:row{X:Integer,Y:String},
  S:row{B:Integer,C:list(String)}
  key{A}

};

This is possible thru system provided types and user defined types.
It's also supported thru explicit conversion processes between one
particular type and another. The foundation to support these
constructs is unique to a relational system and does not exist in sql.
Whether a specific struture makes logical sense and whether it's
supported is another matter. Tables within tables is open to question.
Even if the system supports it would it make sense or would another
type of structure be more appropriate? This question goes to the edges
of a relational system and I'm afraid I can't do it justice here ☺

>This would allow bad developers to commit the common 'a,b,c' 1NF
>violation in a whole new way, but then bad developers can screw
>things up in any language.

The view that strings like 'a,b,c' violate the idea of the atomicity
of a column in an sql table is a direct result of sql's lack of types
and lack of relationships between types. There is no violation of any
kind in a relational system because the string can be stored as
a single value of a column retaining the concept that there individual
elements involved. It would simply be stored as a 'list' type.
For example column B of table TableList takes a comma delimited string
and splits it into a list type.

create table TableList
from
table
     {
      row{1 A,'A,B,C'.Split({','}) B},
      row{2,'D, E , F, G'.Split({','})},
      row{3,'H,I,J,K'.Split({','})}
     };

The table definition of TableList is:
create table TableList
A:Integer,
B:list(String),
key{A}

For each row of the table column A can be retrieved and the individual
items of column B, the list, are availiable.
Select the value of A and the value of the 1st item in the
list(B).
select TableList {A,B[0] ListItem1};
A ListItem1
- ---------
1 A
2 D
3 H

Directly address the 1st item in the list of A=2.
select TableList[2].B[0];
D

Directly address the last item in the list of A=3.
select TableList[3].B[TableList[3].B.Count()-1];
K

Get column B for the row where A=1 and convert the list into a table.
select ToTable(TableList[1].B,'Str','Index');
or
select ToTable( (TableList where A=1 over{B})[].B, 'Str','Index');
Str Index
--- -----
A   0
B   1
C   2

It's types that a relational system guarantees integrity for and high
level operators that allow the explicit conversions between that
developers should have for application development. And this is the
same idea the MS net team calls 'functional programming' which is what
they developed LINQ for. But a relational system is functional
programming! ☺

>bad developers can screw things up in any language.

Sure but application development with sql has a tendency to make
anyone a nitwit at some time or other. I'm for less nitwits ☺

>> 'where' and 'having').
>I'd guess these are both for efficiency.

GROUP BY was added after the original SELECT but instead of
redesigning the language they saw no problem with leaving in two
constructs that do the same thing! Every time I see an MS paper on
'best practices' I have got to grin ☺

On table variables:

>But you agree that (1) it has some features of variables, and (2) it
>could reasonably be extended to have more features of variables?

I don't think MS could lock its developer army in a hotel and tell
them to make sql a little more relational ☺ They have two choices.
Either buy a relational system (like D4) or start from the ground up
to develop one. The gulf between a relational system and sql is too great
to try to simply make changes in sql server. 

>A lot of people find SQL pretty straightforward, especially in this
>newsgroup.  Your choice of (pseudo-)syntax will make a difference to
>them.  (You might get different responses from a newsgroup focusing
>on front-end programming languages, especially if they already
>resemble Pascal as D4 seems to do.)

Sure at least a dozen people who write books and articles find sql
straightforward ☺ For the rest I'd like to at least see a choice for
them. Again, I hope some can get beyond just syntax to grasp what
a relational system would offer.

Basic Anatomy of Sql Server, part III

2007-11-06T00:54:00.000-08:00

This is the third in a series of articles examining some basic
concepts in Sql Server.

Basic Anatomy of Sql Server, part I - What is a stored procedure?
Basic Anatomy of Sql Server, part II - The unit test as part of the database

What does deferred name resolution really mean?     

Essentially part II answered the often asked question, "why can't I
SELECT from an sql stored procedure?". Because there is nothing to
select from. There is an absence of the concept of a table type in sql
which is necessary to realize a table. 

Now we ask another simple question. How is it D4 can understand the assignment
of a table and a test of equality between two tables just as t-sql understands
the same things between numbers:

var aNewTable:= GroupByShipCountry(1) ;     //Assign the table for employee 1 based
                                            //on the D4 operator from Part I to 
                                            //aNewTable.
var Test1:= aNewTable=GroupByShipCountry(1);//Are the two tables the same? Yes.
                                            //Now change aNewTable with an update.
update (aNewTable adorn{key{Cnt,MinFrt}}) 
         set {MinFrt:=$0.00}
          where ((Cnt<11) and (MinFrt<$2.50));
var Test2:= aNewTable=GroupByShipCountry(1); //Now are the two tables the same? No.

select row{Test1 TestofEquality_before_Update, Test2 TestofEquality_after_Update};

TestofEquality_before_Update TestofEquality_after_Update 
---------------------------- --------------------------- 
True                         False                       

And how is it that t-sql can do with numbers what D4 does with tables:

DECLARE @X Integer, @Y Integer; 
SET @X=5; SET @Y=10;  
if @X=@Y
  print '@X=@Y'
   else
    print '@X!=@Y'
    
@X!=@Y    

The answer is obvious and simple. We can only assign values and make comparisons
with variables. And we cannot do anything with a variable unless it is typed.
The unit in 'unit test' is the awareness of the database of a table as a variable.
To say the idea of a unit test is part of the database is to recognize that
a table as a variable and its type is the only way to distinguish one table
from another. The unit is the same for distinguishing among numbers and strings
and tables. And it is this basic unit that sql server lacks for tables. 

Sql server bol describes deferred name resolution as:

'This process is called deferred name resolution because table objects referenced
 by the stored procedure need not exist when the stored procedure is created, 
 but only when it is executed.'
 
What is this really a statement of? 

When t-sql parses these statements:

IF @X=5 
  PRINT '@X=5'
   ELSE
    PRINT '@X!=5'

it returns the error: Must declare the scalar variable "@X".

In D4 when this statement is parsed:

select aNonExistentTable;

it returns the same error as t-sql: Unknown identifier "aNonExistentTable".

These errors are the same. Each system is aware that a variable has not been
defined with a type. There is the intent at parse in each system to check
that an identifier (@X, aNonExistentTable) has been appropriately defined. 
And where the identifier is recognized as a variable and nothing else. This
intent has nothing to do with the context that surrounds the identifier. And
it is this intent that lies at the heart of a relational database. A statement
of just the variable results in the same error.

t-sql   D4
-----   -------
@X      aNonExistentTable;

Must declare the scalar variable "@X"., Unknown identifier "aNonExistentTable".

Now these are all examples of so called t-sql deferred name resolution, there
is no error raised on parsing only execution:

SELECT * FROM aNonExistentTable

SELECT aNonExistentTable

IF aNonExistentTable=anotherNonExistentTable
       print 'YES'

CREATE PROCEDURE theTableDoesNotExist
AS
SELECT * FROM aNonExistentTable

Given that there is computer science and not one computer science for t-sql
and one for everything else, how is it that these statements can be parsed
without error? Because at parse-time if there is no concept of an
identitifier as a variable there is only context to check, syntax. Absence
the idea of a variable there is nothing to resolve a database object to, 
hence it is as if these objects/identifiers do not exist at parse-time. 
How bol describes deferred name resolution is a consequence of working with
tables in a variable-less system. It is a price paid, in terms of sacrifice
of integrity and sacrifice in management of table objects, by the developer
for working in a type-less and variable-less and therefore a non-relational
database. Explanations like deferred name resolution are quite common in
sql but they are all just different sides of the same central issue. For
example there is the issue of impedance mismatch. This is just an expression
of the inherent difficulty of communication between a client, which 
understands what a variable is, and an sql server which does not. And finally
there is the idea of relational division. Since comparisons require variables
sql forces an inference to be made about a comparison of table(s) it cannot
do. It forces the use of constructs that are obtuse and complex compared
to simple and direct comparisons that can be made in a relational database
where the table exists as a variable.
(And now you know why you can't pass a table as a parameter. There is no 
table variable to pass ☺ .

The history of sql is a history of a language trying to exist outside the
mainstream of the rest of IT. It's long overdue that application development
use a database that has the same foundation as other programming languages.
And that kind of database is relational.

Basic Anatomy of Sql Server, part I 
What is a stored procedure?

Basic Anatomy of Sql Server, part II 
The unit test as part of the database.

Basic Anatomy of Sql Server, part III 
What does deferred name resolution really mean?

Some related articles of interest:
All tables are typed variables 
A table as a parameter
Reusable procedures

Basic Anatomy of Sql Server, part II

2007-11-06T00:38:00.000-08:00

This is the second in a series of articles examining some basic
concepts in Sql Server.

Basic Anatomy of Sql Server, part I - What is a stored procedure?

The unit test as part of the database.

If Sql Server is operating with the genie in the bottle what would a system
look like with the genie out of the bottle. Here we take the sql procedure
dbo.GroupByShipCountry from part I and use it in a system that takes the
definition beyond the statements that define it

What does such a system look like and what exactly can we expect such a
system to be aware off. Here we use a system called D4, a system with a
kinship to sql but fundamentally different.

For reference we repeat the sql procedure dbo.GroupByShipCountry from part I:

CREATE PROCEDURE dbo.GroupByShipCountry 
       @Employee Integer
AS
SELECT ShipCountry,Count(*) Cnt,Min(Freight) MinFrt,Max(Freight) MaxFrt
FROM Orders
WHERE EmployeeID=@Employee
GROUP BY ShipCountry

Here is the sql procedure as it would be defined in D4:

create operator GroupByShipCountry (Employee:Integer):
           table{ShipCountry:String,Cnt:Integer,MinFrt:Money,MaxFrt:Money}  
begin
result:=
      Orders 
       where EmployeeID=Employee
        group by {ShipCountry}
          add{Count() Cnt,Min(Freight) MinFrt,Max(Freight) MaxFrt} ;
end;

While the differences in syntax between the two languages are beyond the
scope of this article it should be obvious that the definition of result in 
D4 is similar to the SELECT statement in the sql procedure. Semantically they
are identical. The sql procedure and D4 operator each have a parameter to 
define an employee. The obvious difference between the two is the construct:

table{ShipCountry:String,Cnt:Integer,MinFrt:Money,MaxFrt:Money}  

in the beginning of the D4 operator. This says the result of the operator
is a table with the specific type of:

ShipCountry:String,Cnt:Integer,MinFrt:Money,MaxFrt:Money

Lets execute the operator with syntax as close as possible to the sql sp:

Execute('GroupByShipCountry(8)');   

What is returned? Absolutely nothing. A D4 operator can also be specified
with just its name and input parameter(s) values. So lets try:

GroupByShipCountry(8);

What we get with this is the following system message:

Possibly incorrect use of expression as a statement.

What is going on here? Why doesn't the operator display something like the
sql procedure? Lets go back to the sql procedure. At run-time the procedure
is still only a bunch of t-sql statements so 'execute' is appropriate as 
in 'execute these statements'. But if the D4 operator is a table at run-time
it makes no sense to 'execute' a table! So lets try to use the 'table' in
its appropriate context. In sql we would simply say: 

SELECT * FROM <table>

Which corresponds to the D4 statement:

select <table>;

We now substite the D4 operator with an input parameter for the '<table>':

select GroupByShipCountry(8);

And now we get a similar display as the sql procedure:

ShipCountry Cnt MinFrt MaxFrt  
----------- --- ------ ------- 
Argentina   3   $0.33  $217.86 
Austria     5   $25.22 $353.07 
Brazil      9   $5.32  $110.87 
.
Venezuela   9   $4.41  $141.06 

Clearly the D4 system is aware of two distinct things. The definition of
the operator, its statements, and then at run-time a table. And now it
should also be clear that sql is only aware of statements.

How does D4 guarantee that the GroupByShipCountry(8) table will be
of the type:

table{ShipCountry:String,Cnt:Integer,MinFrt:Money,MaxFrt:Money} 

Because that is the only type of table possible. If the object (table)
exists at run-time it can only be of this type. And if the table is
derived it must have come from compatible datatypes. Another way of
saying this is that a system can distinguish between tables or it can
support implicit conversion. If the system can be aware of a table type
it cannot support implicit conversion. Only explicit conversion, an
intervention on the part of the user, can be consistent with a table type.
Sql supports implicit conversion and therefore cannot tell one table
from another. Sql therefore cannot guarantee the integrity of a table. 
It is object (table) types that offers integrity. For sql to support
implicit conversion means it has no concept of a table of a specific
type as an object. For example, this sql illustrates conversion, from
a money type to an integer type:

CREATE TABLE TestConv
                    (ShipCountry varchar(15),Cnt int,MinFrt int,MaxFrt int,
                     PRIMARY KEY (ShipCountry))
GO
INSERT TestConv (ShipCountry,Cnt,MinFrt,MaxFrt)
EXEC GroupByShipCountry 8 

It also undermines the integrity of each table and therefore the database.
This is what would happen in a D4 operator if a table with incompatible
datatypes with the source was defined:

create operator GroupByShipCountryI (Employee:Integer):
          table{ShipCountry:String,Cnt:Integer,MinFrt:Integer,MaxFrt:Integer}  
begin
result:=
      Orders 
       where EmployeeID=Employee
        group by {ShipCountry}
          add{Count() Cnt,Min(Freight) MinFrt,Max(Freight) MaxFrt} ;
end;

The error returned is a compile error (as opposed to run-time):

Cannot convert a value of type "System.Money" to a value of type "System.Integer".

Not only does the database offer integrity through type compatibility but
the entire framework is exposed to the developer for use as part of the
application itself. A value can be checked for its type with an operator
that accepts any type (a generic). We can make a comparison of the value to
a specific type which returns a boolean value of True/False.

create operator WhatIsIt(aValue:generic):String
begin
result:=
  if aValue is String then
   'String'
     else 
      if aValue is Integer then
       'Integer'
        else 
         if aValue is Money then
            'money'
           else
             'Some other type';
end;          

select 
 WhatIsIt((GroupByShipCountry(8) adorn{key{ShipCountry}})['Canada'].MaxFrt); 

money

We have created the GroupByShipCountry(8) table with a key on ShipCountry
and directly accessed the row where ShipCountry is 'Canada' and extracted
the value of MaxFrt to use as the argument to operator WhatIsIt. The type
of that value is money. And we can extend the same type recognition logic
to a table.

var TryTable:Boolean:=false;
if WhatIsIt(GroupByShipCountry(8))='Some other type' then
  TryTable:=GroupByShipCountry(8) is generic table;
select TryTable; 

True

We have used the GroupByShipCountry(8) table as the argument to the WhatIsIt
procedure. The procedure returned the string 'Some other type' and
in the if statement we asked if the GroupByShipCountry(8) table is a
generic table. In other words we are testing if GroupByShipCountry(8) 
represents some specific type of table. And we can go from a generic type
of table to a specific type of table:

var TryTable:Boolean:=false;
if WhatIsIt(GroupByShipCountry(8))='Some other type' then
 if (GroupByShipCountry(8) is generic table) then
   TryTable:=
      (table of typeof(GroupByShipCountry(8)){})
       is
        table{ShipCountry:String,Cnt:Integer,MinFrt:Money,MaxFrt:Money};
select TableDee add{TryTable Is_GroupByShipCountry8_this_table_type};   

Is_GroupByShipCountry8_this_table_type 
-------------------------------------- 
True        

After determining that GroupByShipCountry(8) is some type of table we have
asked the question, is the type of the GroupByShipCountry(8) table the
specific type represented by the pairs of columns and their datatypes of:

ShipCountry:String, Cnt:Integer, MinFrt:Money, MaxFrt:Money

The statement: table of typeof(GroupByShipCountry(8)){}
extracts just the type information from GroupByShipCountry(8) as information
independent from the values in the rows of the table.

var TryTable:Boolean:=false;
if WhatIsIt(Orders)='Some other type' then
 if (Orders is generic table) then
   TryTable:=
      (table of typeof(Orders){})
       is
       table{ShipCountry:String,Cnt:Integer,MinFrt:Money,MaxFrt:Money};
select TableDee add{TryTable Is_GroupByShipCountry8_this_table_type};    

Is_GroupByShipCountry8_this_table_type 
-------------------------------------- 
False                                  

We have confirmed that the Orders table is not the same particular type of
table as GroupByShipCountry(8). 

And finally we show the ability to make a functional test rests on the
ability to make a unit test:

var C:Boolean:=True;          
if
   (
     table of typeof(GroupByShipCountry(1)){}
      is
       table{ShipCountry:String,Cnt:Integer,MinFrt:Money,MaxFrt:Money}
    ) 
     and
   (
     table of typeof(GroupByShipCountry(8)){}
      is
       table{ShipCountry:String,Cnt:Integer,MinFrt:Money,MaxFrt:Money}
    )   
      then 
       if GroupByShipCountry(1)<>GroupByShipCountry(8) then
         C:=False;
          select TableDee add{C Are_Tables_of_Same_Type_the_Same};
    
Are_Tables_of_Same_Type_the_Same 
-------------------------------- 
False                     

We have first verified that the two tables are the same type of table and then
asked if the two tables are the same. In other words, we can only test the
equality of two tables (they have the same set of rows) if they are the same
table type just as we can only compare two numbers or two dates. It is
obviously nonsensical and illogical to compare two objects of different types.
(I note in passing that the fact that an sql database is aware of individual
 columns and their datatypes is fundamentally different from being aware that 
 such pairs can be encapsulated to define a type of table. It is the notion of 
 encapsulation that defines a type (table) independent of any column(s) as a 
 scalar type that sql is aware of. Any place in t-sql that a table definition
 appears ie. CREATE TABLE statement, a table valued function or a table variable
 (a confusing and misleading construct) whatever this definition is the important
  point is this definition does not signify a table type).

Can we make a test of equality between the sql procedure dbo.GroupByShipCountry
and the D4 operator GroupByShipCountry? As just previously shown we can easily
make such a functional test if both are recognized as table objects.
We can elevate the mere t-sql statements of dbo.GroupByShipCountry by bringing
in the statements to D4 with a pass-thru query. The following D4 query displays
the same result as executing the procedure in Sql Server.

select SQLQuery('Execute dbo.GroupByShipCountry 1');

ShipCountry Cnt MinFrt  MaxFrt   
----------- --- ------- -------- 
Argentina   1   63.7700 63.7700  
Austria     5   74.6000 351.5300 
Belgium     1   29.5900 29.5900  
.
Venezuela   8   0.9300  148.6100 

Is the display a 'result set', as in Sql Server, or a table in D4? Let us
apply a unit test to confirm or reject whether it is a table in D4:

select 
 table of typeof(SQLQuery('Execute dbo.GroupByShipCountry 1')){} is generic table;    
 
True

which confirms that D4 is aware of the dbo.GroupByShipCountry t-sql result 
set as an object of some type of table at run-time. We can now try to test
the equality of the two tables assuming they are of the same specific type. 
The following query should return a boolean, either True or False, on the
equality (sameness) of the tables:

select GroupByShipCountry(1) = SQLQuery('Execute dbo.GroupByShipCountry 1') ;

But on checking the query at 'compile', two errors are returned:

Cannot convert a value of type "System.Decimal" to a value of type "System.Money".
No signature for operator "iCompare" matches the call signature
"(System.Money, System.Decimal)".

Both errors indicate a type mismatch. It is clear we cannot test for equality
because the two tables are of different types. We know the specific type of
table for the D4 operator. Let us go back to the unit test to find out what
type of table is the sql procedure:

var TypeofTable:=table of typeof(SQLQuery('Execute dbo.GroupByShipCountry 1')){}; 
var TypeString:String;
if TypeofTable 
 is table{ShipCountry:String,Cnt:Integer,MinFrt:Decimal,MaxFrt:Decimal} then
  TypeString:='table{ShipCountry:String,Cnt:Integer,MinFrt:Decimal,MaxFrt:Decimal}'
 else
  if TypeofTable 
   is table{ShipCountry:String,Cnt:Integer,MinFrt:Money,MaxFrt:Money} then
    TypeString:='table{ShipCountry:String,Cnt:Integer,MinFrt:Money,MaxFrt:Money}'
    else
     TypeString:='A type where MinFrt and MaxFrt are differnt than Decimal and Money';
select TypeString;     

table{ShipCountry:String,Cnt:Integer,MinFrt:Decimal,MaxFrt:Decimal}  

Now we understand what's happening. The money datatype in Sql Server is
really a decimal type significant to 4 digits. The money type in D4 is
a totally different type. When D4 interacts with a server table it reads
money for datatype. When the sql procedure is brought into D4 as a table
the MinFrt and MinFrt are interpreted as decimal (and this also accounts
for the different displays of the server procedure and the D4 operator).
To compare the two tables we have to make an explicit conversion. We can
convert the server decimal types to D4 money types.

select 
        GroupByShipCountry(1) 
        = 
        (SQLQuery('Execute dbo.GroupByShipCountry 1')
           {ShipCountry,Cnt,ToMoney(MinFrt) MinFrt,ToMoney(MaxFrt) MaxFrt});

True

Clearly, the idea of a unit test in D4 can be used by a developer to provide a 
programmable form of integrity within an application. 

A database that recognizes, supports and exposes a unit test for a table
type is an objects database. An objects database that has an algebra for
tables is a relational database. D4 is a relational database.

Basic Anatomy of Sql Server, part I 
What is a stored procedure?

Basic Anatomy of Sql Server, part II 
The unit test as part of the database.

Basic Anatomy of Sql Server, part III 
What does deferred name resolution really mean?

Some related articles of interest:
All tables are typed variables 
A table as a parameter
Reusable procedures

Basic Anatomy of Sql Server, part I

2007-11-05T23:50:00.000-08:00

This is the first in a series of articles examining some basic
concepts in Sql Server.

What is a stored procedure?
                  
This example uses Sql Server 2005 and the Northwind sample database.

The following is a very simple stored procedure that returns some
aggregates from the Orders table for a given EmployeeID value supplied
as an input parameter:
                  
CREATE PROCEDURE dbo.GroupByShipCountry
      @Employee Integer
AS
SELECT ShipCountry,Count(*) Cnt,Min(Freight) MinFrt,Max(Freight) MaxFrt
FROM Orders
WHERE EmployeeID=@Employee
GROUP BY ShipCountry

After executing the above script to create the procedure we ask a most
simple question: what is it? What exactly is Sql Server aware of
regarding this procedure? We can take as the starting point the definition
of a generic procedure. Bol states:

'A stored procedure is a saved collection of Transact-SQL statements ...
that can take and return user-supplied parameters.'

We can simply ask is there any indication that the server is aware of
anything that rises above the level of the t-sql statements defining
the procedure. Should we expect to find such thing? We do know that
the server recognizes the object Orders in sys.objects as a USER_TABLE.
And if we execute the query:

SELECT *
FROM Northwind.INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME='Orders'

we know the server recognizes the columns and their datatypes for the
Orders table. We know the server recognizes that the procedure depends
on three columns from the Orders table:

SELECT object_name(referenced_major_id) Dependent,COLUMN_NAME
FROM sys.sql_dependencies A
CROSS APPLY (SELECT B.COLUMN_NAME
            FROM Northwind.INFORMATION_SCHEMA.COLUMNS B
            WHERE B.TABLE_NAME=object_name(A.referenced_major_id)
            and A.referenced_minor_id=B.ORDINAL_POSITION) C
WHERE object_id=object_id('dbo.GroupByShipCountry')

Dependent COLUMN_NAME
--------- ---------------
Orders    EmployeeID
Orders    Freight
Orders    ShipCountry

We know the procedure works if we execute it:

Execute dbo.GroupByShipCountry 8       

It retrieves, in the precise wording of bol, 'rows and columns from the
Orders table' which are collectively termed a 'result set'. (We will
return to the server Orders table and result set term in later articles).

ShipCountry     Cnt         MinFrt                MaxFrt               
--------------- ----------- --------------------- ---------------------
Argentina       3           .3300                 217.8600
Austria         5           25.2200               353.0700
Brazil          9           5.3200                110.8700
.
Venezuela       9           4.4100                141.0600

When we look for specific information about the procedure itself with
the sp_help system procedure we see that @Employee is listed as a
parameter.

The information from sys.objects confirms that the procedure is
recognized and it indeed is a definition of a stored procedure:

SELECT name,object_id,type,type_desc
FROM sys.objects
WHERE object_id = OBJECT_ID('dbo.GroupByShipCountry') AND type='P'

name                 object_id   type type_desc             
-------------------- ----------- ---- ----------------------
GroupByShipCountry   1551396646  P    SQL_STORED_PROCEDURE                   

In summary there does not exist any evidence that the server is
aware of any information not contained in the collection of t-sql
statements that define it. In fact the server does not exhibit
any information about three of the four columns (Cnt, MinFrt, MaxFrt)
that are displayed when it is executed. This leaves us in a very unsteady
state. At the very most the server is performing an operation which is
not exposed to us to interact with, at the very least it is performing an
operation that it is itself unaware of. This situation makes it impossible
to look to the server for any characterization of the procedure towards
answering the question of what this procedure really is. It is up to us
to infer what the procedure is. With the consequence that we can not
rely on the server to either characterize what the procedure really is
nor depend on the server to guarantee the integrity of the operation.
This void in the server must be filled.

The unit test

Is it fair to say that the failure of the server necessitates a unit test?
We can more clearly see the answer to this question if we can clearly
see just what is the nature of a unit test that clarifies just what this
procedure is. An insightful perspective on the unit test is offered in
the excellent article:

'Using unit and functional tests in the development process'
by Jeff Canna
www.ibm.com/developerworks/library/j-test.html

The author writes:

'Many times the development of a system is likened to the building of a
house. Unit testing is analogous to a building inspector visiting a
house's construction site. He is focused on the various internal systems
of the house, the foundation, framing, electrical, plumbing, and so on.
He ensures (tests) that the parts of the house will work correctly and
safely, that is, meet the building code.'

He goes on to say:

'Unit tests tell a developer that the code is doing things right;
functional tests tell a developer that the code is doing the right things.'

The author is clearly expressing the idea that a unit test is something
the goes toward confirming something that is of a very basic nature.
Something that is the basis on which any other functional tests are
based on. So what sort of unit can we use to confirm the 'the code is doing
things right'? And will it answer the question of what the procedure really
is?
The sort of unit test we are looking for is approached by the insight offered
in another excellent article:

'Close those Loopholes - Testing Stored Procedures'
by Alex Kuznetsov and Alex Styler
www.simple-talk.com/content/print.aspx?article=426

With reference to a procedure very similar to dbo.GroupByShipCountry the
authors write:

'Suppose you need to write a unit test for that stored procedure.
You might want to verify that your result set has correct structure
(correct column names and types)...'

And further:

'When you set up your unit test, your result set (or result sets, if your
stored procedure returns several ones) is saved into an XML file. All the
information necessary for further comparison is saved column names, column
types, and the data...When you run your unit test, the result set is
compared against the saved XML file. Everything is compared - column names,
column types, and all the data.'

The idea of verifying that the 'result set has correct structure
(correct column names and types).' What the authors refer to as
'correct structure' is what we are really searching for. A test for the
correct structure is exactly what the server lacks. And what is this
structure that is based on pairs of columns and their datatypes? For it
is the columns of this structure that are displayed by executing the
procedure:

ShipCountry Cnt MinFrt MaxFrt 
----------- --- ------ ------

which are based on the structure of column/datatype:

ShipCountry:varchar, Cnt:Integer, MinFrt:Money, MaxFrt:Money

It is these pairs that define a specific type, a specific type of table.
And this simple concept allows us to finally answer the question of
what the stored procedure really is: a table. The unit test is a test
confirming the specific type of the table returned. What the server calls
a 'result set' (and not a 'table') is based on the fact that there does
not exist the concept of a type for a table. There does not exist a way
for the server to distinguish one table from another. It is the non-existence
of this concept in the server that necessitates this most basic unit (test)
being made the responsibility of the user. This is the huge hole in sql
that must be filled by the user, a role that the database should perform
is lost and with it the integrity that it should offer.

In part II we show and describe a database system that takes the
definition of a stored procedure beyond the statements that define it.

Basic Anatomy of Sql Server, part I 
What is a stored procedure?

Basic Anatomy of Sql Server, part II 
The unit test as part of the database.

Basic Anatomy of Sql Server, part III 
What does deferred name resolution really mean?

Some related articles of interest:
All tables are typed variables 
A table as a parameter
Reusable procedures

Sql - Really simple division

2007-10-22T21:29:00.000-07:00

This article is based on the thread:

microsoft.public.sqlserver.programming
Monday, October 22, 2007
"select rows that contain both arg1 = 'x' and arg1 = 'y'"
http://tinyurl.com/386lbx

>I need to select only rows where the name contains x and y.  I do not want
>to get rows that only contain x or contain x, z, or contain y, z.  I only
>want rows where the name has an x an a y (they can have a z but they must
>have an x and a y).

Forget QA or SSMS. Just draw a picture.
You use the term row. So draw a row.

row{aName as sName, acode as code}

Now you want people who have code x and y. These are two
rows. So draw them. Use 'burns' for the person.

row{'burns' as sName, 'x' as code}
row{'burns' as sName, 'y' as code}

Now what do you call two rows together? How about calling it
a table ☺ .
Draw it.

table
   {
    row{'burns' as sName, 'x' as code}
    row{'burns' as sName, 'y' as code}
   }
 
Now if this table, made just for 'burns', is in your
#tmp1 table then 'burns' is a guy you want. In other
words, if both rows (one for x, one for y) are in
#tmp1 then you have a hit. Super simple ☺ .

Now in QA or SSMS do this.

declare @x int, @y int
set @x=2
set @y=3
if @x<=@y
print 'Yes'
else
print 'No'

No explanation necessary. Super simple. Now take the same
idea of comparing two integers and extend it to comparing
two tables.

if
table
     {
      row{'burns' as sName, 'x' as code}
      row{'burns' as sName, 'y' as code}
     }
       <=
         #tmp1
           print 'Yes'
                else 'No'
              
In other words, does each row for 'burns' occur in #tmp1?  
'Burns' can have codes in #tmp1 in addition to x and y (ie. 2<=3)
So 'burns' has to have at least a row for x and a row for y
in #tmp1. If in #tmp1 'burns' has only an 'x' or only a 'y'
no matter what other codes he has that's no good (2<=1).
In the case above you will see 'Yes' printed since the comparison
is true.

This whole scenario is referred to as relational division in
database terminology. But these simple ideas are obscured
by sql because you can't draw a picture of a row, nor a
table nor does sql understand comparing tables like integers.
So instead you're left with grouping and counting, joins,
intersects, existential queries and whatnot all trying to
express a simple idea yet at the same time obscuring it.

Now in a query you want to substitute all the unique names
from #tmp1 into our little table so for each person we can
test the comparison with #tmp1. What would such a query look
like?

select
 select distinct sName as aPerson from #tmp1 
  where      -- draw a table with two rows for each aPerson
             -- the 1st column has a value aPerson and
             -- the column is named 'sName'. The 2nd column is
             -- called 'code'. The column names and datatypes
             -- are the same as in #tmp1.
        table
             {
              row{aPerson as sName, 'x' as code},
              row{aPerson as sName, 'y' as code}
             }
             --  Compare the tables.
              <=
             --  Form a table from #tmp1 of rows belonging to the
             --  aPerson above.
              (Tmp1 where sName=aPerson);

Now this won't quite work in sql no matter where you execute it ☺ .
But what would a query really look like that will work with
#tmp1. Here it is. And it really is almost self-explanatory.
And it works in the D4 language! :)
(Tmp1 is a table same as #tmp1 stored in an Sql Server 2005 database).

select
  Tmp1 {sName aPerson}
    where
         table
              {
               row{aPerson sName, 'x' code},
               row{aPerson sName, 'y' code}
              }
               <=
              (Tmp1 where sName=aPerson);
   
aPerson
-------
burns 
jones 
smith 

Now this is what MS should be doing. Sql is a language of choice
for some things. But it certainly is not the choice language
for others.

Sql Server - The Undefined Trigger

2007-10-21T19:12:00.000-07:00

Ask the question: does Sql Server have a row level trigger.
Answer: it depends. From the point of view of application
development how in the world can the answer possibly be
ambiguous? Well lets back up and look at the Update statement.
From Sql Server 2005 bol:
Using UPDATE with the FROM Clause
'The results of an UPDATE statement are undefined if the
 statement includes a FROM clause that is not specified in
 such a way that only one value is available for each column
 occurrence that is updated, that is if the UPDATE statement
 is not deterministic'.
Ok what is really the rational for even keeping this
'proprietary' syntax? Since it is inherently non-deterministic
why even offer it to developers most of whom do not understand
but the simplest of sql. After all Sql Server is explicitly
relinquishing responsibility for the integrity of the Update and
making the user responsible. Shouldn't the idea of a database be
the other way around? Ok now lets move to the trigger. From bol
under the topic: Multirow Considerations for DML Triggers:
'When you write the code for a DML trigger, consider that the
 statement that causes the trigger to fire can be a single statement
 that affects multiple rows of data, instead of a single row.'
Does this sound familiar? Of course it does. The same non-determinism
of the FROM clause in Update now reappears in the insert table
of the trigger. But of course it is not presented as a 'trigger
is inherently undefined'. No, now the undefined nature of the
trigger is called 'a Multirow consideration'. And again the server
reliquishes responsibility of integrity and puts it in the
hands of the user. Well lets get real. There is no such thing as
a multirow trigger. From an applications standpoint the largest
scope of a trigger is a row. The so called multirow trigger is an 
example of what happens when users don't scrutinize what nonsense
is thrown their way. Instead of getting rid of a stupid idea
in Update using a FROM it was instead extended to a trigger.
It was simply an easy and expedient thing to do. But it was
shameful and lazy too. Lack of integrity is based on lack of
character. But it is equally disappointing that more users do
not complain and demand the integrity that a database should offer.
Sql Server can do much, much better.  But, like E.T., users must
phone home.

Sql - Constantly struggling

2007-10-17T02:14:00.000-07:00

The subject of this article is called (by others) database constants.
So lets be sure we're on the same page with what a constant is.
It is just a value.

X = Substring(Y,1,5);

X and Y are variables, 1 and 5 are values.

X = Y * 5;

X and Y are variables, 5 is a constant. It is a number whose value is 5.
A table in D4 is just as much a variable as X and Y. A table in 
sql is like the value 5, a constant. The difference between a constant
and a variable is also the difference between D4, a relational
database, and sql!

The following is part of an exchange I had with someone on the
issue of database constants. The idea of database constants
was raised in the article:

A New (and Hopefully Better) Approach to Constants
http://www.sqlservercentral.com/articles/T-SQL/61244/

Here I go ☺ :

Sql Server lets users define a 'table variable'. Now you really
don't believe it's a table 'variable' do you?:)  Of course not.
They're just playing with words. There is no such thing in sql
server as a table variable. But because they call it that most
users really believe it. It's like believing in the tooth fairy:)
Now we're going to play the same game with database 'constants':)
There is no such thing as a database constant. There are only
values (which can't be updated or assigned other values) and
variables (which can be updated and assigned other values).
But in sql there is no real talk of these basic things. And
that's why I talk about Dataphor ☺. There is no difference between
what this guy is calling a 'constant' and a table or 'resultset' in
sql. They are all (constant) values and not variables. I'm trying
to show what sql would be like if there really were table 'variables'.
One way was with the CTE article, 'Do you know what the Sql CTE really is?'
But people seem to be having a hard time wrapping their head
around it:) This sql stuff just gets crazier. Now people are
going to believe that you can pass a table as a parameter to a
stored procedure. That is nonsense. One reason people believe it
is they have nothing to compare/constrast it with. So I show what
it really means to pass a table to a sp:
http://beyondsql.blogspot.com/2007/10/sql-whats-really-with-with.html
http://beyondsql.blogspot.com/2007/06/dataphor-13-passing-table-as-parameter.html
http://beyondsql.blogspot.com/2007/08/dataphor-creating-super-function.html
http://beyondsql.blogspot.com/2007/09/dataphor-all-tables-are-typed-variables.html
etc.etc.

But to do these kind of things requires a foundation different than
sql. Hence the name of my site ☺ .
The so called 'constants' are basically a table without a key, ie a row.
So I do:
create table Y
{
 A:Integer,
 B:String,
 key{}
};

insert row{1 A,'Micheal' B} into Y;

select Y[].B
'Michael'
select Y[].A
1
where the '[]' is called a row extracter.
Or I can define a 'list':
var LList:=list(String){'Michael','Steve','Jimmy'};
select LList[0];
Michael

Or I can define a table (ie a row) with lists.
create table Z
{
 Strings:list(String),  //The column holds a list of strings.
 Integers:list(Integer),//The column holds a list of integers.
 key{}
};

insert row{{'Micheal','Steve','Jimmy'} Strings,{2,10,22,40} Integers}
into Z;

select Z[].Strings[2];
Jimmy
select Z[].Integers[1];
10
where the [2] and [1] refer to the ordinal position of the item (value)
in the list.
But sql doesn't have a table 'type' or a list type so it couldn't
understand these things. The only thing sql can do is try to simulate
these types. And it winds up in a mess of gooblygook that few understand
and robs the user of the integrity that the database should provide
by supporting these things directly. All this stuff is based on values,
variables and types. The only types that exist is sql are numbers and
strings. Again there is no type for tables, row or list. And therefore
no variables for these things because without a type you can't define
a variable. Make sense ☺ .
var X:= Orders //X is a variable of type table, specifically the type
               //of the Orders table (the column names and there data types).
This is the huge step forward from sql. In sql the result of a query
is no different than the 'Constant' that guy was trying to talk about ☺ .
It is also the difference between a CTE and a variable that holds the
result of the 'CTE'. You can't reuse a CTE in sql because it's a 'constant',
a value. You can only reuse a variable.

Dataphor - Merging strings together

2007-10-17T01:23:00.000-07:00

This problem was raised in the thread:
microsoft.public.sqlserver.programming
Monday, October 15, 2007 
'Parse and Merge two fields'
http://tinyurl.com/2eosa3

The idea is to take two pipe ('|') delimited strings and created one
string where each item in the first string is matched with the item 
in the second string having the same ordinal position.
The two columns are People and Position. So given:

People:=' Sam | Jane | Gene'
Position:= 'Accounting | Finance | Marketing'

we want:

People_Position:= 'Sam|Accounting|Jane|Finance|Gene|Accounting'

Of course in sql this is usually a mess since sql does not
support a list type and any of the operations that can transform
a delimited string to a list and visa versa.
In D4 we simply split the delimited string to a list of items and
then create a table from the list. The table will contain the
ordinal position as an integer value of each item in the list. 
So a row has the item (string) value and sequence value of that item
in the string as columns. So we just concatenate the columns of each
string together (separated by a pipe) and then concatenate these 
strings over the rows with the Concat operator ordering the 
concatenated string by the sequence value (over rows). 

Note that there is no concept of just unique item in a list. A list
can have any number of duplicate items. When the list is transformed
to a table the sequence number guarantees that same value items will
be in the table. In other words the sequence is a key of the table.

select ToTable('A|A|B|B|C'.Split({'|'}));

value sequence 
----- -------- 
A     0        
A     1        
B     2        
B     3        
C     4      

And with column names of your choice:

select ToTable('A|A|B|B|C'.Split({'|'}),'Str','Seq');

Str Seq 
--- --- 
A   0   
A   1   
B   2   
B   3   
C   4   

(Browse this site for other articles on string operations for more info).

Here is some sample data. The data is stored in a Sql Server 2005 database.
All queries are written with the D4 relational language of Dataphor.

create table JR
{
 id:Integer,
 People:String tags{Storage.Length='100'},
 Position:String tags{Storage.Length='100'},
 key{id}
};
 
insert
table
{
row{1 id,'joe|sam|pete|mike' People,'Manager|Team Lead|Worker Ant|Worker Ant' Position},
row{2,'A| C|  T ',' JobA|JobC |JobT '},
row{3,'A| C ',' JobA | JobB |JobC '}, //Ummatched strings (lists)
row{4,'Bush | Cheney | Rice ' ,' Worser|Worst|Worse '},
row{5,'Z |Y |X ' ,' Backwards Z|Backwards Y |Backwards X '}
} into JR;

select JR;

id People                Position                                
-- --------------------- --------------------------------------- 
1  joe|sam|pete|mike     Manager|Team Lead|Worker Ant|Worker Ant 
2  A| C|  T               JobA|JobC |JobT                        
3  A| C                   JobA | JobB |JobC                      
4  Bush | Cheney | Rice   Worser|Worst|Worse                     
5  Z |Y |X                Backwards Z|Backwards Y |Backwards X

Also note that we can access the table with a pass-thru query using t-sql:

select SQLQuery("SELECT id,People,Position FROM JR"); 

id People                Position                                
-- --------------------- --------------------------------------- 
1  joe|sam|pete|mike     Manager|Team Lead|Worker Ant|Worker Ant 
2  A| C|  T               JobA|JobC |JobCT                       
3  A| C                   JobA | JobB |JobC                      
4  Bush | Cheney | Rice   Worser|Worst|Worse                     
5  Z |Y |X                Backwards Z|Backwards Y |Backwards X   

Note that the only restriction used here is that both strings have
the same number of items. 

select 
  JR //The table JR in Sql Server 2005. 
  //SQLQuery("SELECT id,People,Position FROM JR") Or we could use this t-sql query . 
     where //The same 'where' statement as in sql. 
           //Check that each string has the same number of items.      
          People.Split({'|'}).Count()=Position.Split({'|'}).Count()
                                with {IgnoreUnsupported = 'true'}
      //For each row in the table we 'add' a column (People_Position).
      //The People_Position column is the '|' delimited string of
      //People and Position item by item.
      add
      {
       //From a table whose rows represents each item in People and Position,
       //for each row form a string of the person and position separated by
       //a '|' and concatenate that string over the rows into a single string.
       Concat({People_Position,Del} 
        from
         (
          //Use Split to create a 'list' of items from the delimited string.
          //Then create a table from list with ToTable. Column 'Str1'
          //is the string value of the item from the list. Column 'Seq'
          //is the item number from the list. So now a table represents
          //the original input column People. In other words, the rows of
          //the table are the delimited items in the string from left to right.
          (ToTable(People.Split({'|'}),'Str1','Seq'){Str1.Trim() Str1,Seq} )       
           join  //A natural join on Seq which is an integer from 0-># of items
                 //in the string. Seq preserves the sequence from left to
                 //to right of the items in the string.
          (ToTable(Position.Split({'|'}),'Str2','Seq'){Str2.Trim() Str2,Seq} )
            {Seq,Str1,Str2,'|' Del,(Str1+'|'+Str2) People_Position}  
         )
                order by {Seq}) //The concatenation over rows is ordered by the
                                //Seq column. This is the same order as the items
                                //in the original input columns, People/Position.
                                //Therefore the original order is preserved in
                                //the result (People_Position).
        People_Position //This is name of the new column added to each row of JR.
       }
        {id,People_Position}  //These are the only two columns we've chosen
                              //to display from the query.
            order by {id};    //Order by the values if the id column.
 
id People_Position                                           
-- --------------------------------------------------------- 
1  joe|Manager|sam|Team Lead|pete|Worker Ant|mike|Worker Ant 
2  A|JobA|C|JobC|T|JobT                                      
4  Bush|Worser|Cheney|Worst|Rice|Worse                       
5  Z|Backwards Z|Y|Backwards Y|X|Backwards X                


Instead of eliminating rows where the item counts don't agree
we can insert the string 'Unmatched' for those id values.

select 
 //Instead of table JR we access the table in sql server with a t-sql query.
 //In D4 the query does not return an sql 'resultset' but instead is treated
 //as a table 'variable' which is of a nature fundamentally different than sql.
 SQLQuery("SELECT id,People,Position FROM JR") 
 //Store the restriction on count as a boolean value (T/F) in column TestCnt.
  add{People.Split({'|'}).Count()=Position.Split({'|'}).Count() TestCnt}
                                with {IgnoreUnsupported = 'true'}
   add
      {
 //If the TestCnt is true concatenate the rows, else return a nil (null) value.
 //A case statement just like the sql case statement could also be used.
  if TestCnt
      then
       Concat({People_Position,Del}
        from
         (
          (ToTable(People.Split({'|'}),'Str1','Seq'){Str1.Trim() Str1,Seq} )       
           join
          (ToTable(Position.Split({'|'}),'Str2','Seq'){Str2.Trim() Str2,Seq} )
            {Seq,Str1,Str2,'|' Del,(Str1+'|'+Str2) People_Position}  
         )
                order by {Seq})
                else
                  nil
                    as String
        People_Position
      }
 //Insert the unmatched string for a nil value of the People_Position column.     
        {id,IfNil(People_Position,'*** Unmatched Strings ***') Field} 
          order by {id};
          
id Field                                                     
-- --------------------------------------------------------- 
1  joe|Manager|sam|Team Lead|pete|Worker Ant|mike|Worker Ant 
2  A|JobA|C|JobC|T|JobT                                      
3  *** Unmatched Strings ***                                 
4  Bush|Worser|Cheney|Worst|Rice|Worse                       
5  Z|Backwards Z|Y|Backwards Y|X|Backwards X                 


D4 allows all the expressive power you need to easily test for
any criteria. Here we add some additional rows to table JR that
should be eliminated.

insert
table
{
//Digits in Postion.
row{6 id,'joe|sam|pete|mike' People,'Manager|Team Lead|Worker Ant1|Worker Ant2' Position},
//Digits in People and Position
row{7 id,'joe|sam|pete1|mike' People,'Manager|Team Lead|Worker Ant1|Worker Ant' Position}
} into JR;

select JR;

id People                Position                                  
-- --------------------- ----------------------------------------- 
1  joe|sam|pete|mike     Manager|Team Lead|Worker Ant|Worker Ant   
2  A| C|  T               JobA|JobC |JobT                          
3  A| C                   JobA | JobB |JobC                        
4  Bush | Cheney | Rice   Worser|Worst|Worse                       
5  Z |Y |X                Backwards Z|Backwards Y |Backwards X     
6  joe|sam|pete|mike     Manager|Team Lead|Worker Ant1|Worker Ant2 
7  joe|sam|pete1|mike    Manager|Team Lead|Worker Ant1|Worker Ant

We add some additional restrictions to the where statement to eliminate
rows with strings that contain any non letters.

select 
 JR
  where 
    (
     ( People.Split({'|'}).Count()=Position.Split({'|'}).Count() )
     //We also test that each item is only letters (no digits or other junk).
     //This can be done any numbers of ways. Here we just compare the count
     //of all items to the count of items eliminated by characters other than 
     //letters.
       and
       (
        People.Split({'|'}).Count()
        =
        Count(ToTable(People.Split({'|'}),'F1Str','F1Seq') 
        //Test that items are made up of letters only. To eliminate
        //blanks contaiminating the test we remove blank character before
        //testing.
          where IsLetter(Replace(Trim(F1Str),' ','')))
       )
       and
       (
        Position.Split({'|'}).Count()
        =
        Count(ToTable(Position.Split({'|'}),'F2Str','F2Seq') 
          where IsLetter(Replace(Trim(F2Str),' ','')))
       )   
     )  with {IgnoreUnsupported = 'true'}        
      add
      {
       Concat({People_Position,Del}
        from
         (
          (ToTable(People.Split({'|'}),'Str1','Seq'){Str1.Trim() Str1,Seq} )       
           join
          (ToTable(Position.Split({'|'}),'Str2','Seq'){Str2.Trim() Str2,Seq} )
            {Seq,Str1,Str2,'|' Del,(Str1+'|'+Str2) People_Position}  
         )
                order by {Seq})
        People_Position
       }
        {id,People_Position}
          order by {id};

id People_Position                                           
-- --------------------------------------------------------- 
1  joe|Manager|sam|Team Lead|pete|Worker Ant|mike|Worker Ant 
2  A|JobA|C|JobC|T|JobT                                      
4  Bush|Worser|Cheney|Worst|Rice|Worse                       
5  Z|Backwards Z|Y|Backwards Y|X|Backwards X                

Thanks for stopping by ☺ .

Dataphor - Exploding hierarchical data

2007-10-09T20:49:00.000-07:00

This article show examples of using the D4 explode operator.
This operator is used for expressing hierarchical data. It is
not a 'recursive' operator like the recursive sql CTE but more
like the Oracle Connect By construct.

The examples follow the ones used by Itzik Ben-Gan to illustrate
the recursive CTE query in Sql Server 2005, specifically the
'Single-Parent Environment: Employees Organizational Chart'
which can be found at:
http://msdn2.microsoft.com/en-us/library/ms345144.aspx#docum_topic4

The explode examples are based on the Employees table used in the
sql example. The table is stored in an Sql Server 2005 database.
Several of the examples use the concept of a dense rank.

As with most of my articles, the code is not necessarily 'clever' but
straightforward, not necessarily the best 'performant' but intended
to express the many different concepts and constructs in D4.

select Employees

empid mgrid empname  salary    
----- ----- -------- ----------
1     0     Nancy    $10,000.00
2     1     Andrew   $5,000.00 
3     1     Janet    $5,000.00 
4     1     Margaret $5,000.00 
5     2     Steven   $2,500.00 
6     2     Michael  $2,500.00 
7     3     Robert   $2,500.00 
8     3     Laura    $2,500.00 
9     3     Ann      $2,500.00 
10    4     Ina      $2,500.00 
11    7     David    $2,000.00 
12    7     Ron      $2,000.00 
13    7     Dan      $2,000.00 
14    11    James    $1,500.00 


Get a tree for a specific manager.

select
 (
  Employees
    explode
   by mgrid = parent empid
   where  mgrid=0
     order by {empid}
       include level
   with {IgnoreUnsupported = 'true'}
  )
// Do a little string insert to format the tree.  
    add {'' Temp, level-1 Totalspace}
     add {
          (
           empname.Insert(0,Temp.PadLeft(Totalspace,'|'))
          ).Replace('|',' | ')
          Tree
         }
        {Tree,mgrid,empid,empname,sequence,level};
       
Tree              mgrid empid empname  sequence level
----------------- ----- ----- -------- -------- -----
Nancy             0     1     Nancy    1        1    
 | Andrew         1     2     Andrew   2        2    
 |  | Steven      2     5     Steven   3        3    
 |  | Michael     2     6     Michael  4        3    
 | Janet          1     3     Janet    5        2    
 |  | Robert      3     7     Robert   6        3    
 |  |  | David    7     11    David    7        4    
 |  |  |  | James 11    14    James    8        5    
 |  |  | Ron      7     12    Ron      9        4    
 |  |  | Dan      7     13    Dan      10       4    
 |  | Laura       3     8     Laura    11       3    
 |  | Ann         3     9     Ann      12       3    
 | Margaret       1     4     Margaret 13       2    
 |  | Ina         4     10    Ina      14       3    

Treat each employee as if they are a manager. This will return the
subordinates of each employee regardless of whether or not
they are a manager.

select
 (
  Employees
    explode
   by mgrid = parent empid
   where  mgrid>=0         //Changing where to include all employees.
     order by {empid}
       include level
   with {IgnoreUnsupported = 'true'}
  )
     add {'' Temp, level-1 Totalspace}
     add {
          (
           empname.Insert(0,Temp.PadLeft(Totalspace,'|'))
          ).Replace('|',' | ')
          Tree
         }
        {Tree,mgrid,empid,empname,sequence,level} ;
       
Tree              mgrid empid empname  sequence level
----------------- ----- ----- -------- -------- -----
Nancy             0     1     Nancy    1        1    
 | Andrew         1     2     Andrew   2        2    
 |  | Steven      2     5     Steven   3        3    
 |  | Michael     2     6     Michael  4        3    
 | Janet          1     3     Janet    5        2    
 |  | Robert      3     7     Robert   6        3    
 |  |  | David    7     11    David    7        4    
 |  |  |  | James 11    14    James    8        5    
 |  |  | Ron      7     12    Ron      9        4    
 |  |  | Dan      7     13    Dan      10       4    
 |  | Laura       3     8     Laura    11       3    
 |  | Ann         3     9     Ann      12       3    
 | Margaret       1     4     Margaret 13       2    
 |  | Ina         4     10    Ina      14       3    
Andrew            1     2     Andrew   15       1    
 | Steven         2     5     Steven   16       2    
 | Michael        2     6     Michael  17       2    
Janet             1     3     Janet    18       1    
 | Robert         3     7     Robert   19       2    
 |  | David       7     11    David    20       3    
 |  |  | James    11    14    James    21       4    
 |  | Ron         7     12    Ron      22       3    
 |  | Dan         7     13    Dan      23       3    
 | Laura          3     8     Laura    24       2    
 | Ann            3     9     Ann      25       2    
Margaret          1     4     Margaret 26       1    
 | Ina            4     10    Ina      27       2    
Steven            2     5     Steven   28       1    
Michael           2     6     Michael  29       1    
Robert            3     7     Robert   30       1    
 | David          7     11    David    31       2    
 |  | James       11    14    James    32       3    
 | Ron            7     12    Ron      33       2    
 | Dan            7     13    Dan      34       2    
Laura             3     8     Laura    35       1    
Ann               3     9     Ann      36       1    
Ina               4     10    Ina      37       1    
David             7     11    David    38       1    
 | James          11    14    James    39       2    
Ron               7     12    Ron      40       1    
Dan               7     13    Dan      41       1    
James             11    14    James    42       1            

If we reverse 'by mgrid = parent empid' to 'by empid = parent mgrid' we
get the tree of a particular employee to their top level manager.

select
 (
  Employees
    explode
      by empid = parent mgrid
         where  empid=13
   order by {empid}
       include level
   with {IgnoreUnsupported = 'true'}
  )
     add {'' Temp, level-1 Totalspace}
     add {
          (
           empname.Insert(0,Temp.PadLeft(Totalspace,'|'))
          ).Replace('|',' | ')
          Tree
         }
        {Tree,mgrid,empid,empname,sequence,level};
       
Tree           mgrid empid empname sequence level
-------------- ----- ----- ------- -------- -----
Dan            7     13    Dan     1        1    
 | Robert      3     7     Robert  2        2    
 |  | Janet    1     3     Janet   3        3    
 |  |  | Nancy 0     1     Nancy   4        4    
       
By changing the where predicate to >=1 we can get a report on all
employees (note where empid>=3 eliminates the graph of the 1st 2
employees (Nancy and Andrew) but doesn't eliminate them from graphs
of other employees who report to them).


Here we create an operator that will give a graph in either direction
of a specific employee and their top level manager. In other words,
we can either start with the employee and go down to their top level
manager or start with the employees top level manager and go down
to the employee. The concept of the dense rank is used for binding
all rows of each employee together. By getting the empname and empid
for each dense rank we can target any employee by name or number.   
(Note we could, of course, use a view or any number of other constructs.
I just felt in the mood to use an operator ☺ ).

The operator takes two arguments. The first, aTreeTable, is a table
of type Employees. The second, Start_At, is a string and indicates
the direction of the graph. Using 'M' for manager starts with the
manager. Using 'E' starts with the employee.

create operator EmpTree(aTreeTable:typeof(Employees),Start_At:String):
//The operator returns a virtual table with columns and their data types
//defined by the typeof expression.
typeof(
       Employees
       add{1 level,'S' empnamerank,1 empidrank, 'T' Tree_Graph, 1 Rank, 1 TreeOrder}
       )
begin
result:=table of typeof(result){};
//Starting at top level mgr for each employee is a desc sort (default).
//Starting at each employee to the top level mgr. is an asc sort.    
//Start_At=M(gr) is desc sort (default)
//Start_At='E(mp) is asc sort.
var LSort:='D'; //Default.
if ( ((Start_At.Trim())[0]).Upper() ) = 'E'
   then LSort:='A' ;
var T:=
aTreeTable
explode  
by empid = parent mgrid
    where  empid>=1
      order by {empid}
       include level
   with {IgnoreUnsupported = 'true'};
//Get a dense rank. This rank binds all rows for each employee
//together. The idea is to increment a count for every level 1 since
//a level 1 indicates the start of a new employee.
var TR:=
       T add{case when level=1 then empname else nil end NameEmp}
        add
           {
            Count(
                  T rename {sequence sequenceX}
                   where  (sequenceX<=sequence) and (level=1)
                  ) 
             Rank
            };
var SR:=
     TR               
      join
       (
        TR group by {Rank}
        //We want the emp name, number (empid) and reverse level for each
        //dense rank (empname). We want the reverse level so we can get
        //the tree representative from the employee to top level manager
        //AND the top level manager to the employee.
                add{Max(empid) NumEmpid,Max(NameEmp) Emp_Name,Max(level) Maxlevel}
        )                        
 add {
    (empname.Insert(0,''.PadLeft( (if LSort='A' then (level-1) else (Maxlevel-level)),
                                                         '|'))).Replace('|',' | ')
        Tree_Graph
     }
     rename {sequence seq};
result:=
 ToTable(
  ToList(
 cursor(SR
 order by
     {
      Rank,
      seq
      sort ((1 - (2*ToInteger((LSort = 'D'))))*(.left.value ?= .right.value)) asc
     }
    
       )//cursor
       )//ToList
      )  //ToTable
        {empid,mgrid,empname,salary,level,Emp_Name empnamerank,
          NumEmpid empidrank,Tree_Graph,Rank,sequence+1 TreeOrder};
end;         


Here we show the first five employees by using Rank in a where statement.
The value of the Rank corresponds to the ascending order of empid.
Because empid starts at 1 Rank happens to be equal to the empid.

select EmpTree(Employees,'Mgr') //We use the operator as if it were a table.
  where Rank<=5
      order by {TreeOrder};     

empid mgrid empname  salary     level empnamerank empidrank Tree_Graph   Rank TreeOrder
----- ----- -------- ---------- ----- ----------- --------- ------------ ---- ---------
1     0     Nancy    $10,000.00 1     Nancy       1         Nancy        1    1        
1     0     Nancy    $10,000.00 2     Andrew      2         Nancy        2    2        
2     1     Andrew   $5,000.00  1     Andrew      2          | Andrew    2    3        
1     0     Nancy    $10,000.00 2     Janet       3         Nancy        3    4        
3     1     Janet    $5,000.00  1     Janet       3          | Janet     3    5        
1     0     Nancy    $10,000.00 2     Margaret    4         Nancy        4    6        
4     1     Margaret $5,000.00  1     Margaret    4          | Margaret  4    7        
1     0     Nancy    $10,000.00 3     Steven      5         Nancy        5    8        
2     1     Andrew   $5,000.00  2     Steven      5          | Andrew    5    9        
5     2     Steven   $2,500.00  1     Steven      5          |  | Steven 5    10     


We can overload the EmpTree operator so as to provide a default value
for the Start_At parameter, ie. the direction of the graph. We make
the default 'M' so the graph starts with the top level manager of the
employee.

We simply supply the literal 'Mgr' for the sort direction for the
same operator whose signiture includes the Start_At parameter.

create operator EmpTree(aTreeTable:typeof(Employees)):
typeof(
       Employees
       add{1 level,'S' empnamerank,1 empidrank, 'T' Tree_Graph, 1 Rank, 1 TreeOrder}
       )
begin
result:= EmpTree(aTreeTable,'Mgr');
end;

     
Here get the tree of employee 'James' starting at the highest level manager
by using the overload signature of the EmpTree operator.

select EmpTree(Employees)
  where empnamerank='James'
  {Tree_Graph,TreeOrder}
      order by {TreeOrder};
     
Tree_Graph        TreeOrder
----------------- ---------
Nancy             38       
 | Janet          39       
 |  | Robert      40       
 |  |  | David    41       
 |  |  |  | James 42       


Here we start with employee 'James' up to his highest level manager.

select EmpTree(Employees,' Emp ')
  where empnamerank='James'
  {Tree_Graph,TreeOrder}
      order by {TreeOrder};
     
Tree_Graph        TreeOrder
----------------- ---------
James             38       
 | David          39       
 |  | Robert      40       
 |  |  | Janet    41       
 |  |  |  | Nancy 42        


Here we use the EmpTree operator to get the enumerated paths in both
directions using the Concat (concatenation) operator. (More info here).

select
 (EmpTree(Employees) add{'.' Del} adorn{key{empidrank,TreeOrder}})
   group by {empidrank}
    add 
          {
            Max(empnamerank) empname,
            Concat(empname,Del order by {empidrank,TreeOrder}) PathMgrtoEmp,
            Concat(empname,Del order by {empidrank,TreeOrder desc}) PathEmptoMgr
          }
     rename {empidrank empid}
      order by {empid};
     
empid empname  PathMgrtoEmp                   PathEmptoMgr                  
----- -------- ------------------------------ ------------------------------
1     Nancy    Nancy                          Nancy                         
2     Andrew   Nancy.Andrew                   Andrew.Nancy                  
3     Janet    Nancy.Janet                    Janet.Nancy                   
4     Margaret Nancy.Margaret                 Margaret.Nancy                
5     Steven   Nancy.Andrew.Steven            Steven.Andrew.Nancy           
6     Michael  Nancy.Andrew.Michael           Michael.Andrew.Nancy          
7     Robert   Nancy.Janet.Robert             Robert.Janet.Nancy            
8     Laura    Nancy.Janet.Laura              Laura.Janet.Nancy             
9     Ann      Nancy.Janet.Ann                Ann.Janet.Nancy               
10    Ina      Nancy.Margaret.Ina             Ina.Margaret.Nancy            
11    David    Nancy.Janet.Robert.David       David.Robert.Janet.Nancy      
12    Ron      Nancy.Janet.Robert.Ron         Ron.Robert.Janet.Nancy        
13    Dan      Nancy.Janet.Robert.Dan         Dan.Robert.Janet.Nancy        
14    James    Nancy.Janet.Robert.David.James James.David.Robert.Janet.Nancy
    

Here we get the counts of employees directly or indirectly reporting to managers.

select
  Employees
    explode
      by empid = parent mgrid
         where  empid>=1
   order by {empid}
       include level
   with {IgnoreUnsupported = 'true'}
     group by {mgrid} add{Count() MgrCnt}
      where mgrid>0

 mgrid MgrCnt
 ----- ------
 1     13    
 2     2     
 3     7     
 4     1     
 7     4     
11     1    


Here are the employees who are the managers. For example there are 13 people
who report to Nancy. This is represented indirectly by people who report
to the three managers directly below her (Andrew, Janet, Margaret) and
directly by the same managers reporting to her. Dan, David and Ron report
to Robert. But there is an additional employee reporting to David (James)
so Robert (mgrid 7) has 4 employees reporting to him.

select
 (
  Employees
    explode
      by empid = parent mgrid
         where  empid>=1
   order by {empid}
       include level
   with {IgnoreUnsupported = 'true'}
     group by {mgrid} add{Count() MgrCnt}
  )
   join Employees
     where mgrid>0
      join  ( Employees {empid MgrEmpid,empname MgrName} )
        by mgrid=MgrEmpid
      {mgrid,MgrName,empname,MgrCnt};
     
mgrid MgrName  empname  MgrCnt
----- -------- -------- ------
1     Nancy    Andrew   13    
1     Nancy    Janet    13    
1     Nancy    Margaret 13    
2     Andrew   Michael  2     
2     Andrew   Steven   2     
3     Janet    Ann      7     
3     Janet    Laura    7     
3     Janet    Robert   7     
4     Margaret Ina      1     
7     Robert   Dan      4     
7     Robert   David    4     
7     Robert   Ron      4     
11    David    James    1           


Here we get salaries of subordinates under managers. Those employees
who are not managers are omitted.

var T:=
Employees
explode  
by mgrid = parent empid
     where  mgrid>=0
      order by {empid}
       include level
   with {IgnoreUnsupported = 'true'};
//Get a dense rank. This rank binds all rows for each employee
//together. The idea is to increment a count for every level 1 since
//a level 1 indicates the start of a new employee.
var TR:=
       T add{case when level=1 then empname else nil end NameEmp}
        add
           {
            Count(
                  T rename {sequence sequenceX}
                   where  (sequenceX<=sequence) and (level=1)
                  ) 
             Rank
            };
var SR:=
   (
     TR               
      join
       (
        TR group by {Rank}
        //We want the empid and name for each dense rank.
          add{Min(empid) NumEmpid,Max(NameEmp) Emp_Name}
        )                        
    )
    //We only want managers, those that have subordinates.
     where
     NumEmpid
              in
                ( Employees {mgrid} ) with {IgnoreUnsupported = 'true'}
      add {
         ( (empname+ case when level>1 then ' ('+ToString(salary)+')'
             else '' end).Insert(0,''.PadLeft((level-1),'|'))).Replace('|',' | ')
              Mgr_Sal_Tree
          };

select SR {Mgr_Sal_Tree,sequence} order by {sequence};   

Mgr_Sal_Tree                  sequence
----------------------------- --------
Nancy                         1       
 | Andrew ($5,000.00)         2       
 |  | Steven ($2,500.00)      3       
 |  | Michael ($2,500.00)     4       
 | Janet ($5,000.00)          5       
 |  | Robert ($2,500.00)      6       
 |  |  | David ($2,000.00)    7       
 |  |  |  | James ($1,500.00) 8       
 |  |  | Ron ($2,000.00)      9       
 |  |  | Dan ($2,000.00)      10      
 |  | Laura ($2,500.00)       11      
 |  | Ann ($2,500.00)         12      
 | Margaret ($5,000.00)       13      
 |  | Ina ($2,500.00)         14      
Andrew                        15      
 | Steven ($2,500.00)         16      
 | Michael ($2,500.00)        17      
Janet                         18      
 | Robert ($2,500.00)         19      
 |  | David ($2,000.00)       20      
 |  |  | James ($1,500.00)    21      
 |  | Ron ($2,000.00)         22      
 |  | Dan ($2,000.00)         23      
 | Laura ($2,500.00)          24      
 | Ann ($2,500.00)            25      
Margaret                      26      
 | Ina ($2,500.00)            27      
Robert                        30      
 | David ($2,000.00)          31      
 |  | James ($1,500.00)       32      
 | Ron ($2,000.00)            33      
 | Dan ($2,000.00)            34      
David                         38      
 | James ($1,500.00)          39    

Here is the same tree as above (of salaries for subordinates) using a
table and a view created with a pass-thru query.

We can create a table based on the result of explode.

create table Emp_T
from
     (
       Employees
       explode  
       by mgrid = parent empid
       where  mgrid>=0
       order by {empid}
       include level
       with {IgnoreUnsupported = 'true'}
       add{case when level=1 then empname else nil end NameEmp}
        adorn  //We can include various meta-data about the columns of the table.
             {
              NameEmp nil static tags {Storage.Length = "10"},
              empname static tags {Storage.Length = "10"}
             }//end adorn.
     );  

Now we get the dense rank using a pass-thru query to Sql Server. The result
of the sql query could be set to var TR which means that TR is a table
variable in D4. The TR variable is therefore of the same nature
no matter how it (a table variable) was derived. Or we could create
a view based on the pass-thru query and use that.
var TR:= but we're using a view instead of setting the pass-thru to var TR.

create view TR
  SQLQuery("select A.*, 
      (select Count(*)
              from Emp_T as B
                 where (B.sequence<=A.sequence) and (B.level=1)) as Rank
          from Emp_T as A");

Use the TR view in a batch to get the salary tree.

var SR:=
   (
     TR               
      join
       (
        TR group by {Rank}
        //We want the empid and name for each dense rank.
                add{Min(empid) NumEmpid,Max(NameEmp) Emp_Name}
        )                        
    )
    //We only want managers, those that have subordinates.
     where
     NumEmpid
              in
                ( Employees {mgrid} ) with {IgnoreUnsupported = 'true'}
      add {
         ( (empname+ case when level>1 then ' ('+ToString(salary)+')'
             else '' end).Insert(0,''.PadLeft((level-1),'|'))).Replace('|',' | ')
              Mgr_Sal_Tree
          };
select SR {Mgr_Sal_Tree,sequence} order by {sequence};

Mgr_Sal_Tree             sequence
------------------------ --------
Nancy                    1       
 | Andrew (5000)         2       
 |  | Steven (2500)      3       
 |  | Michael (2500)     4       
 | Janet (5000)          5       
 |  | Robert (2500)      6       
 |  |  | David (2000)    7       
 |  |  |  | James (1500) 8       
 |  |  | Ron (2000)      9       
 |  |  | Dan (2000)      10      
 |  | Laura (2500)       11      
 |  | Ann (2500)         12      
 | Margaret (5000)       13      
 |  | Ina (2500)         14      
Andrew                   15      
 | Steven (2500)         16      
 | Michael (2500)        17      
Janet                    18      
 | Robert (2500)         19      
 |  | David (2000)       20      
 |  |  | James (1500)    21      
 |  | Ron (2000)         22      
 |  | Dan (2000)         23      
 | Laura (2500)          24      
 | Ann (2500)            25      
Margaret                 26      
 | Ina (2500)            27      
Robert                   30      
 | David (2000)          31      
 |  | James (1500)       32      
 | Ron (2000)            33      
 | Dan (2000)            34      
David                    38      
 | James (1500)          39


Here we get the sum of salaries for subordinates under managers using
the TR view. Eliminating level 1 in the query excludes the managers
salary in the sum. Again the dense rank idea makes this an easy query.

select
   (
    TR               
      join //This is a natural join based on Rank.
       (
        TR group by {Rank}
        //We want the empid and name for each dense rank.
            add{Min(empid) NumEmpid,Max(NameEmp) Emp_Name}
        )//->The relation of view TR being group by Rank.
    )//->The relation from TR joined to (TR grouped by Rank) 
          where level>1 //A 'where' applied to above relation. This relation
                        //is now grouped by NumEmpid to get subordinate salaries.
            group by{NumEmpid} add{Max(Emp_Name) Emp_Name,Sum(salary) SumSalary}
              order by {NumEmpid};

NumEmpid Emp_Name SumSalary 
-------- -------- ----------
1        Nancy    37500.0000
2        Andrew   5000.0000 
3        Janet    15000.0000
4        Margaret 2500.0000 
7        Robert   7500.0000 
11       David    1500.0000

Sql - The Two Standards of the Sql CTE

2007-10-01T16:46:00.000-07:00

Sql really has two standards. There is the ANSI sql standard and then
there is the DOUBLE standard.

This example uses Sql Server 2005 and specifically the CTE (common table expression).

Sql Server sees nothing wrong with this query that duplicates a column name.
Something that no database should allow.

SELECT employeeid,customerid as employeeid
FROM orders

From Bol on the CTE column_name
'Duplicate names within a single CTE definition are not allowed.The list of
 column names is optional only if distinct names for all resulting  columns
 are supplied in the query definition.'

But this query produces an error.

WITH Emps  (Emp, Emp) AS
(
    SELECT employeeid,customerid as employeeid
    FROM orders
)
SELECT *
FROM Emps
Error: 'The column 'Emp' was specified multiple times for 'Emps'.

But try this. And it works!

WITH Emps  (Emp, Emp1) AS
(
    SELECT employeeid,customerid as employeeid
    FROM orders
)
SELECT *
FROM Emps

So what does Bol really mean in the column_name description. It means
that the stupid and error prone idea of creating duplicate column names
is perfectly ok in the query definition. It's only the outer reference
names (the  ( column_name [ ,...n ] ) that are going to be checked for
duplicates. So the MS statement:
'The list of column names is optional only if distinct names for all
 resulting columns are supplied in the query definition.'

is a recognition that its okay to screw up the query_definition as
as long as you come away with distinct names in the CTE. In other words,
give distinct names for the 'same' columns created in the query_definition
with the same name. If you don't duplicate names in the quey_definition
you can dispense with the CTE column name list totally. So not only is the
MS statement a statement of guilt (don't expect us to correct duplicate column
names when it first occurs, we'll catch it the 2nd time it comes around),
it's also a statement of a double standard. One standard for an independent
SELECT statement and another standard for a CTE! Double standards allow
the initial error to propagate. Not what you want in a database.

This is what a database should do when duplicate columns are declared.
This is what Dataphor will do when you try this.

select Orders {EmployeeID,CustomerID EmployeeID};
Error: Duplicate object name "EmployeeID".

Only if you 'love' sql could you tolerate its nonsense ☺ .

For more on the sql CTE see:
'Do you know what the Sql CTE is?'
http://beyondsql.blogspot.com/2007/10/sql-whats-really-with-with.html

Sql - Do you know what the Sql CTE is?

2007-10-01T16:10:00.000-07:00

From Sql Server 2005 Bol for WITH CTE (common_table_expression):
'Specifies a temporary named result set, known as a common table expression (CTE).
 This is derived from a simple query and defined within the execution scope of
 a single SELECT, INSERT, UPDATE,
 or DELETE statement.'

Now this works fine (using the NorthWind database in Sql Server 2005):

WITH Emps (Emp, Cnt) AS
(
    SELECT employeeid,count(*) AS Cnt
    FROM orders
    GROUP BY employeeid
)
SELECT Max(Cnt)
FROM Emps

But try to define a CTE independently. You can't.
So this CTE craps out:

WITH Emps (Emp, Cnt) AS
(
    SELECT employeeid,count(*) AS Cnt
    FROM orders
    GROUP BY employeeid
)

Try to use the CTE in a batch with a 2nd SELECT. You can't 'reuse' the CTE.
So this batch craps out with an error thrown on the use of the CTE
on the 2nd SELECT.

WITH Emps (Emp, Cnt) AS
(
    SELECT employeeid,count(*) AS Cnt
    FROM orders
    GROUP BY employeeid
)
SELECT Max(Cnt)
FROM Emps

SELECT Min(Cnt)
FROM Emps       -- Invalid object name 'Emps'.

Now what if a CTE in a batch could be defined independently and
be used with multiple SELECT statements. What would the batch look
like. Well it could look something like this using the D4 language of Dataphor
(see http://beyondsql.blogspot.com/2007/09/dataphor-all-tables-are-typed-variables.html
 for some more background):

//Define a CTE independently. And to define a CTE independently it must
//be stored in a variable. We are using a pass-thru query from D4 to Sql Server
//in t-sql to the NorthWind database.
var EMPS.CTE:=
              SQLQuery("SELECT employeeid,count(*) AS Cnt
                          FROM orders
                            GROUP BY employeeid");
//Define a row using the CTE.
var LRow:=row{Min(Cnt from EMPS.CTE) MinCnt,Max(Cnt from EMPS.CTE) MaxCnt};//A row.  
//SELECT the row.
select LRow; 
/*
MinCnt MaxCnt
------ ------
42     156
*/
//Use the CTE in a 2nd SELECT, SELECT a table
select EMPS.CTE return 2 by {Cnt desc};
/*
employeeid Cnt
---------- ---
4          156
3          127
*/

What is really going on here. Now Sql has managed to immunize itself
against computer science forever. But if we were to apply a compute
science term to the sql CTE what term what we call it. We would call
it a value. Because a value cannot exist independently (by itself :))
and cannot be reused. To reuse a value we have to declare every time
we want to use it. With a variable we don't have to worry about
any of these value limitations. We can simply reuse the variable
as much as we want. The 'var' in 'var EMPS.CTE:=..' means define
a variable that contains a value and the value is the CTE SELECT
statement. So now you know the sql term 'result set' is really just
a value :) And now you know the benefit of working with variables
over values.

Of course the situation with the sql CTE is not the only case where
you have to repeat the damn thing to use it.

This works fine in sql.

SELECT orderid,customerid,employeeid
FROM orders
WHERE customerid in ('COMMI','TORTU','HUNGO')

But can you store the list of ('COMMI','TORTU','HUNGO') in a variable? No.
You have to repeat it every time you want to use it. It's because there is
no variable available to assign the list to. Of course in D4 we can
assign the list to a variable and use the variable just like the CTE.

var LList:=list(String){'COMMI','TORTU','HUNGO'};
select Orders
where CustomerID in LList
{OrderID,CustomerID,EmployeeID};

Doesn't this make more sense than having to use xml in sql to shred the
string or other crazy sql techniques to store the list items in a table?

Dataphor - Super Function II

2007-09-20T01:31:00.000-07:00

This article extends the functionality of the Dense Rank function
described in:
http://beyondsql.blogspot.com/2007/08/dataphor-creating-super-function.html

For background on the concepts involved see:
http://beyondsql.blogspot.com/2007/06/dataphor-13-passing-table-as-parameter.html
http://beyondsql.blogspot.com/2007/09/dataphor-all-tables-are-typed-variables.html
http://beyondsql.blogspot.com/2007/06/sql-using-dense-rank-for-identifying.html 
Sql Server 2005 Bol, Ranking Functions:
ms-help://MS.SQLCC.v9/MS.SQLSVR.v9.en/tsqlref9/html/e7f917ba-bf4a-4fe0-b342-a91bcf88a71b.htm

The original DenseRank operator was based on a table with a single
primary key. We now extend the usability of the operator by overloading
it to return a dense rank for a compound primary key, ie. a primary key 
consisting of two columns.

As an example consider the problem posed in the thread:

microsoft.public.sqlserver.programming
Thursday, September 13, 2007 2:27 PM
SQL Query Help: Need to remove consecutive items and just keep the earliest

Here is some sample data (courtesy of Tom Cooper) created in an Sql Server
2005 database (Column names are a bit different and additional data has
been added):

CREATE TABLE AssignmentLog (
    TicketID [varchar] (6) NOT NULL,
    EventTime [smalldatetime] NOT NULL,
    GroupMember [varchar] (15) NOT NULL,
    DeptInNeed varchar(15) NOT NULL,
    aNumber int NOT NULL, 
  PRIMARY KEY CLUSTERED (TicketID, EventTime)
)
go
INSERT INTO AssignmentLog (TicketID, EventTime,
GroupMember,DeptInNeed,aNumber)
VALUES ('ABC001', '9/10/2007 15:30:00', 'Helpdesk','Mkt',14)
INSERT INTO AssignmentLog (TicketID, EventTime,
GroupMember,DeptInNeed,aNumber)
VALUES ('ABC001', '9/10/2007 15:42:00', 'Server Admin','Mkt',43)
INSERT INTO AssignmentLog (TicketID, EventTime,
GroupMember,DeptInNeed,aNumber)
VALUES ('ABC001', '9/10/2007 15:50:00', 'Helpdesk','ACT',134)
INSERT INTO AssignmentLog (TicketID, EventTime,
GroupMember,DeptInNeed,aNumber)
VALUES ('ABC001', '9/10/2007 15:52:00', 'Helpdesk','Sales',60)
INSERT INTO AssignmentLog (TicketID, EventTime,
GroupMember,DeptInNeed,aNumber)
VALUES ('ABC001', '9/10/2007 16:01:00', 'Helpdesk','Mkt',88)
INSERT INTO AssignmentLog (TicketID, EventTime,
GroupMember,DeptInNeed,aNumber)
VALUES ('ABC001', '9/10/2007 16:22:00', 'Server Admin','Fin',245)
INSERT INTO AssignmentLog (TicketID, EventTime,
GroupMember,DeptInNeed,aNumber)
VALUES ('ABC001', '9/11/2007 8:30:00', 'Server Admin','Per',15)
INSERT INTO AssignmentLog (TicketID, EventTime,
GroupMember,DeptInNeed,aNumber)
VALUES ('ABC001', '9/11/2007 9:05:00', 'Field Services','Maint',75)
INSERT INTO AssignmentLog (TicketID, EventTime,
GroupMember,DeptInNeed,aNumber)
VALUES ('ABC002', '9/10/2007 11:30:00', 'Helpdesk','HR',0)
INSERT INTO AssignmentLog (TicketID, EventTime,
GroupMember,DeptInNeed,aNumber)
VALUES ('ABC002', '9/10/2007 15:30:00', 'Helpdesk','Fin',24)
INSERT INTO AssignmentLog (TicketID, EventTime,
GroupMember,DeptInNeed,aNumber)
VALUES ('ABC002', '9/11/2007 15:42:00', 'Field Services','Maint',42)
INSERT INTO AssignmentLog (TicketID, EventTime,
GroupMember,DeptInNeed,aNumber)
VALUES ('ABC002', '9/11/2007 15:50:00', 'Field Services','HR',63)
INSERT INTO AssignmentLog (TicketID, EventTime,
GroupMember,DeptInNeed,aNumber)
VALUES ('ABC002', '9/11/2007 15:52:00', 'Helpdesk','Sales',1)
INSERT INTO AssignmentLog (TicketID, EventTime,
GroupMember,DeptInNeed,aNumber)
VALUES ('ABC002', '9/11/2007 16:01:00', 'Helpdesk','HR',20)
INSERT INTO AssignmentLog (TicketID, EventTime,
GroupMember,DeptInNeed,aNumber)
VALUES ('ABC002', '9/11/2007 16:22:00', 'Helpdesk','HR',25)
INSERT INTO AssignmentLog (TicketID, EventTime,
GroupMember,DeptInNeed,aNumber)
VALUES ('ABC002', '9/12/2007 8:30:00', 'Field Services','MKT',75)
INSERT INTO AssignmentLog (TicketID, EventTime,
GroupMember,DeptInNeed,aNumber)
VALUES ('ABC002', '9/12/2007 9:05:00', 'Field Services','MKT',90)

We now work with the AssignmentLog table in Dataphor where it is
treated as a 'variable'.

select AssignmentLog;    

TicketID EventTime             GroupMember    DeptInNeed aNumber 
-------- --------------------- -------------- ---------- ------- 
ABC001   9/10/2007 3:30:00 PM  Helpdesk       Mkt        14      
ABC001   9/10/2007 3:42:00 PM  Server Admin   Mkt        43      
ABC001   9/10/2007 3:50:00 PM  Helpdesk       ACT        134     
ABC001   9/10/2007 3:52:00 PM  Helpdesk       Sales      60      
ABC001   9/10/2007 4:01:00 PM  Helpdesk       Mkt        88      
ABC001   9/10/2007 4:22:00 PM  Server Admin   Fin        245     
ABC001   9/11/2007 8:30:00 AM  Server Admin   Per        15      
ABC001   9/11/2007 9:05:00 AM  Field Services Maint      75      
ABC002   9/10/2007 11:30:00 AM Helpdesk       HR         0       
ABC002   9/10/2007 3:30:00 PM  Helpdesk       Fin        24      
ABC002   9/11/2007 3:42:00 PM  Field Services Maint      42      
ABC002   9/11/2007 3:50:00 PM  Field Services HR         63      
ABC002   9/11/2007 3:52:00 PM  Helpdesk       Sales      1       
ABC002   9/11/2007 4:01:00 PM  Helpdesk       HR         20      
ABC002   9/11/2007 4:22:00 PM  Helpdesk       HR         25      
ABC002   9/12/2007 8:30:00 AM  Field Services MKT        75      
ABC002   9/12/2007 9:05:00 AM  Field Services MKT        90      

Here's how the OP described his problem:
Quote
I just want to know when the ticket left the helpdesk to go to an 
outside group like Server Admins, Field Services, etc.  So when there
are consecutive entries within the same group, I just need the earliest
one of the series.  Here is the output I'm trying to generate:

ticket_id event_timestamp     group_assigned
--------- ------------------- ---------------
ABC001    09/10/2007 15:30:00 Helpdesk
ABC001    09/10/2007 15:42:00 Server Admin
ABC001    09/10/2007 15:50:00 Helpdesk
ABC001    09/10/2007 16:22:00 Server Admin
ABC001    09/11/2007 09:05:00 Field Services

But how can I do that in a query?  I can't simply use a GROUP BY
clause, something like this:

SELECT ticket_id, MIN(event_timestamp) AS event_timestamp,
group_assigned
FROM #AssignmentLog
GROUP BY ticket_id, group_assigned
ORDER BY ticket_id, MIN(event_timestamp)

Because that will lump EVERYTHING to the earliest occurance of each
group_assigned entry.  In other words that doesn't account for the
items in a consecutive series.  The output looks like:

ticket_id event_timestamp     group_assigned
--------- ------------------- ---------------
ABC001    09/10/2007 15:30:00 Helpdesk
ABC001    09/10/2007 15:42:00 Server Admin
ABC001    09/11/2007 09:05:00 Field Services

This shows the first time the helpdesk had the ticket, but doesn't
show when the ticket came BACK to the helpdesk.

I've tried multiple variations of sub-queries with unequal joins (<,
>, etc) but keep running into similar results... can't get the query
to notice consecutive entries.

I would really appreciate any direction or ideas someone may have
about how best to accomplish this.  For reference I'm using MS SQL
2000.
Close Quote

This is a classic problem of constructing a dense rank for group_assigned
(GroupMember) given the compound primary key (ticket_id/TicketID and
event_timestamp/EventTime). When the value of GroupMember stays the same
within a TicketID and EventTime (where we look at EventTime ascending
within TicketID) the rank stays the same. When GroupMember changes
the rank is incremented by 1, we get a new value for the rank. This
behavior is that of a dense rank. Once such a rank is obtained it
can be used as a grouping column and almost all problems (such as the
minimum EventTime for each GroupMember within a TicketID) reduce to 
a simple and straightforward query.
Unfortunately all to often developers are not able to simply state
what 'type' of problem they're trying to solve (the fact that the OP
is on S2000 and does not have ranking functions available is no
excuse. Should vendors be the ultimate authority on types of problems?).
The point is if there was a function library with something like
the DenseRank operator available a developer should know when it is 
appropriate.

The IDenseRank and DenseRank operators for a compound key

Here are the same operators for a single primary key overloaded for a
compound key where one column of the PK is a string (column PKcol2)
and the other (column PKcol2) is a datetime value. If you compare
the single PK operators to these you'll see there are no real drastic
changes. The table parameter (aTable) is changed to accommodate the
compound key and the result now includes both columns of the PK.
The central idea is that only the PK, regardless of whether it's
singular or compound, and the target of the rank (Grp) are the only
columns that are necessary. Note the conceptually similarity to
the S2008 dense_rank() function:

DENSE_RANK ( ) OVER ( [ < partition_by_clause > ] < order_by_clause > )

While the partition column is clearly TicketID we cannot use the
dense_rank() function where the target of the rank (GroupMember) repeats.
This being due to the colossal shortsightedness of sql of not separating
the target of the rank and the ordering of it. We should be able to order
the target of the rank by any column(s) we choose. All sql ranking functions
treat the target of the rank and the order of the target of the rank
as the same column(s). See:
http://beyondsql.blogspot.com/2006/09/rac-rank-this.html
for more details.

create operator IDenseRank
 (aTable:table{PKcol1:String,PKcol2:DateTime,Grp:String}):
     //This is the virtual table returned by IDenseRank.
        table{PK.col1:String,PK2Min:DateTime,PK2Max:DateTime,DenseRank:Integer}
begin
result:=table of typeof (result){};            
var D1:=
ToTable(ToList(cursor(
    (
 ToTable(ToList(cursor(
//We want to get PKs consecutively numbered (PK12Seq).
//
//The ToTable(ToList(cursor table order by { }))) construct is fundamentally
//important both functionally and conceptually. I will cover it in detail
//in future article(s).
//
 ToTable(ToList(cursor(aTable order by {PKcol1,PKcol2} )))
                            {PKcol1,PKcol2,Grp,sequence+1 PK12Seq}
     order by {Grp,PK12Seq}  )))  
      add{PK12Seq-(sequence+1) Drank}
       group by {PKcol1,Grp,Drank} add{Min(PKcol2) PK2Min,Max(PKcol2) PK2Max}
      )      
                 order by {PKcol1,PK2Min}  )))  
                   {PKcol1,PK2Min,PK2Max,sequence+1 DenseRank};
//                   
//We want to renumber the dense rank from 1 to N for each (ie,PKcol1/groupID)             
//
result:= 
    D1
     join
      (
        D1 group by {PKcol1} add {Min(DenseRank) OffSet} {PKcol1,OffSet-1 OffSet}
      )                          
        {PKcol1 PK.col1,PK2Min,PK2Max,DenseRank-OffSet DenseRank} ;
end;

create operator DenseRank(aTable:table{PKcol1:String,PKcol2:DateTime,Grp:String}):
            //This is the virtual table that the operator returns.
                         table{PKcol1:String,PKcol2:DateTime,DenseRank:Integer}
begin
result:=
        table of typeof (result){};                        
result:=
   aTable {PKcol1,PKcol2}
    times               //times is like an sql cross join.
     IDenseRank(aTable) //The virtual result of the IDenseRank operator is used.
//Note the where statement is changed from the single PK operator. We include
//the condition PKcol1=PK.col1 so we get the right PKcol2 values for
//the correct PKcol1 values (groups).
      where (PKcol1=PK.col1) and (PKcol2 between PK2Min and PK2Max)
            {PKcol1,PKcol2,DenseRank};       
end;

We now follow the same methodology for a single PK. The AssignmentLog table
simply must be the same 'type' as the aTable parameter. That means the
column names and their data type must match those of the aTable parameter. 
All we have to do is rename the appropriate AssignmentLog columns, 
ie. TicketID to PKcol1, EventTime to PKcol2 and GroupMember to Grp.

select 
 DenseRank(AssignmentLog {TicketID PKcol1,EventTime PKcol2,GroupMember Grp});     

PKcol1 PKcol2                DenseRank 
------ --------------------- --------- 
ABC001 9/10/2007 3:30:00 PM  1         
ABC001 9/10/2007 3:42:00 PM  2         
ABC001 9/10/2007 3:50:00 PM  3         
ABC001 9/10/2007 3:52:00 PM  3         
ABC001 9/10/2007 4:01:00 PM  3         
ABC001 9/10/2007 4:22:00 PM  4         
ABC001 9/11/2007 8:30:00 AM  4         
ABC001 9/11/2007 9:05:00 AM  5         
ABC002 9/10/2007 11:30:00 AM 1         
ABC002 9/10/2007 3:30:00 PM  1         
ABC002 9/11/2007 3:42:00 PM  2         
ABC002 9/11/2007 3:50:00 PM  2         
ABC002 9/11/2007 3:52:00 PM  3         
ABC002 9/11/2007 4:01:00 PM  3         
ABC002 9/11/2007 4:22:00 PM  3         
ABC002 9/12/2007 8:30:00 AM  4         
ABC002 9/12/2007 9:05:00 AM  4         

We get all the data from the AssignmentLog table and the DenseRank operator
with a simple join. We rename the columns of the result table from the 
DenseRank operator to the orginal column names (of the AssignmentLog table).
(We can rename columns of 'any' table since all tables are 'variables' ☺ .
As you can see the ranks are consecutively numbered starting from 1 within
each TicketID.

select 
(
 DenseRank(AssignmentLog {TicketID PKcol1,EventTime PKcol2,GroupMember Grp}) 
     {PKcol1 TicketID,PKcol2 EventTime,DenseRank}
)   
 join //This is a natural join (match on common columns, ie. TicketID and EventTime.  
    AssignmentLog 
      {TicketID,EventTime,GroupMember,DeptInNeed,aNumber,DenseRank}
        order by {TicketID,EventTime} ;


TicketID EventTime             GroupMember    DeptInNeed aNumber DenseRank 
-------- --------------------- -------------- ---------- ------- --------- 
ABC001   9/10/2007 3:30:00 PM  Helpdesk       Mkt        14      1         
ABC001   9/10/2007 3:42:00 PM  Server Admin   Mkt        43      2         
ABC001   9/10/2007 3:50:00 PM  Helpdesk       ACT        134     3         
ABC001   9/10/2007 3:52:00 PM  Helpdesk       Sales      60      3         
ABC001   9/10/2007 4:01:00 PM  Helpdesk       Mkt        88      3         
ABC001   9/10/2007 4:22:00 PM  Server Admin   Fin        245     4         
ABC001   9/11/2007 8:30:00 AM  Server Admin   Per        15      4         
ABC001   9/11/2007 9:05:00 AM  Field Services Maint      75      5         
ABC002   9/10/2007 11:30:00 AM Helpdesk       HR         0       1         
ABC002   9/10/2007 3:30:00 PM  Helpdesk       Fin        24      1         
ABC002   9/11/2007 3:42:00 PM  Field Services Maint      42      2         
ABC002   9/11/2007 3:50:00 PM  Field Services HR         63      2         
ABC002   9/11/2007 3:52:00 PM  Helpdesk       Sales      1       3         
ABC002   9/11/2007 4:01:00 PM  Helpdesk       HR         20      3         
ABC002   9/11/2007 4:22:00 PM  Helpdesk       HR         25      3         
ABC002   9/12/2007 8:30:00 AM  Field Services MKT        75      4         
ABC002   9/12/2007 9:05:00 AM  Field Services MKT        90      4         

With the inclusion of the DenseRank column we can write relatively simple
queries to answer questions concerning the grouping of TicketID and GroupMember.

We can easily answer the OP original question:
'So when there are consecutive entries within the same group, I just need
 the earliest one of the series.'

This is simply grouping by TicketID,GroupMember and DenseRank and getting
the minimum EventTime.

select
 ( 
  (
    DenseRank(AssignmentLog {TicketID PKcol1,EventTime PKcol2,GroupMember Grp}) 
     {PKcol1 TicketID,PKcol2 EventTime,DenseRank}
  )   
  join AssignmentLog 
      {TicketID,EventTime,GroupMember,DeptInNeed,aNumber,DenseRank}
  )
    group by {TicketID,GroupMember,DenseRank} add{Min(EventTime) MinTime}
                 order by {TicketID,DenseRank}; 

TicketID GroupMember    DenseRank MinTime               
-------- -------------- --------- --------------------- 
ABC001   Helpdesk       1         9/10/2007 3:30:00 PM  
ABC001   Server Admin   2         9/10/2007 3:42:00 PM  
ABC001   Helpdesk       3         9/10/2007 3:50:00 PM  
ABC001   Server Admin   4         9/10/2007 4:22:00 PM  
ABC001   Field Services 5         9/11/2007 9:05:00 AM  
ABC002   Helpdesk       1         9/10/2007 11:30:00 AM 
ABC002   Field Services 2         9/11/2007 3:42:00 PM  
ABC002   Helpdesk       3         9/11/2007 3:52:00 PM  
ABC002   Field Services 4         9/12/2007 8:30:00 AM  

Of course we can remove DenseRank from the result.

select
 ( 
  (
    DenseRank(AssignmentLog {TicketID PKcol1,EventTime PKcol2,GroupMember Grp}) 
     {PKcol1 TicketID,PKcol2 EventTime,DenseRank}
  )   
  join AssignmentLog 
      {TicketID,EventTime,GroupMember,DeptInNeed,aNumber,DenseRank}
  )
    group by {TicketID,GroupMember,DenseRank} add{Min(EventTime) MinTime}
             remove{DenseRank}
                 order by {TicketID,MinTime}; 
                 
TicketID GroupMember    MinTime               
-------- -------------- --------------------- 
ABC001   Helpdesk       9/10/2007 3:30:00 PM  
ABC001   Server Admin   9/10/2007 3:42:00 PM  
ABC001   Helpdesk       9/10/2007 3:50:00 PM  
ABC001   Server Admin   9/10/2007 4:22:00 PM  
ABC001   Field Services 9/11/2007 9:05:00 AM  
ABC002   Helpdesk       9/10/2007 11:30:00 AM 
ABC002   Field Services 9/11/2007 3:42:00 PM  
ABC002   Helpdesk       9/11/2007 3:52:00 PM  
ABC002   Field Services 9/12/2007 8:30:00 AM  

And we can do pretty much anything we want quite easily. Here we
show a concatenated list of Depts (DeptInNeed) for each grouping.

select
 ( 
  (
    DenseRank(AssignmentLog {TicketID PKcol1,EventTime PKcol2,GroupMember Grp}) 
     {PKcol1 TicketID,PKcol2 EventTime,DenseRank}
  )   
  join AssignmentLog 
      {TicketID,EventTime,GroupMember,DeptInNeed,aNumber,DenseRank,',' Del}
   adorn {key{TicketID,EventTime}}
 )  
    group by {TicketID,GroupMember,DenseRank} add{Min(EventTime) MinTime,
             Concat(DeptInNeed,Del order by {TicketID,EventTime}) AllDepts}
                 order by {TicketID,DenseRank};  
                 
TicketID GroupMember    DenseRank MinTime               AllDepts      
-------- -------------- --------- --------------------- ------------- 
ABC001   Helpdesk       1         9/10/2007 3:30:00 PM  Mkt           
ABC001   Server Admin   2         9/10/2007 3:42:00 PM  Mkt           
ABC001   Helpdesk       3         9/10/2007 3:50:00 PM  ACT,Sales,Mkt 
ABC001   Server Admin   4         9/10/2007 4:22:00 PM  Fin,Per       
ABC001   Field Services 5         9/11/2007 9:05:00 AM  Maint         
ABC002   Helpdesk       1         9/10/2007 11:30:00 AM HR,Fin        
ABC002   Field Services 2         9/11/2007 3:42:00 PM  Maint,HR      
ABC002   Helpdesk       3         9/11/2007 3:52:00 PM  Sales,HR,HR   
ABC002   Field Services 4         9/12/2007 8:30:00 AM  MKT,MKT      

Or show distinct Depts for each grouping.
(If you don't understood some of this stuff ask ☺ Everything in due time ☺ . 

var D1:= //D1 is somewhat analogous to an sql CTE. But it's a 'variable'! 
  (
    DenseRank(AssignmentLog {TicketID PKcol1,EventTime PKcol2,GroupMember Grp}) 
     {PKcol1 TicketID,PKcol2 EventTime,DenseRank}
  )   
  join AssignmentLog 
      {TicketID,EventTime,GroupMember,DeptInNeed,aNumber,DenseRank,',' Del};
 
select 
  (
    D1 group by {TicketID,GroupMember,DenseRank}
    add{Min(EventTime) MinTime}
  )  
   join
    (
      (D1 {TicketID,GroupMember,DenseRank,DeptInNeed,Del} adorn{key{DeptInNeed}})
        group by {TicketID,GroupMember,DenseRank}
          add{ Concat( distinct DeptInNeed,Del order by {DeptInNeed}) DistinctDepts}
     ) 
                 order by {TicketID,DenseRank};   
                 
TicketID GroupMember    DenseRank MinTime               DistinctDepts 
-------- -------------- --------- --------------------- ------------- 
ABC001   Helpdesk       1         9/10/2007 3:30:00 PM  Mkt           
ABC001   Server Admin   2         9/10/2007 3:42:00 PM  Mkt           
ABC001   Helpdesk       3         9/10/2007 3:50:00 PM  ACT,Mkt,Sales 
ABC001   Server Admin   4         9/10/2007 4:22:00 PM  Fin,Per       
ABC001   Field Services 5         9/11/2007 9:05:00 AM  Maint         
ABC002   Helpdesk       1         9/10/2007 11:30:00 AM Fin,HR        
ABC002   Field Services 2         9/11/2007 3:42:00 PM  HR,Maint      
ABC002   Helpdesk       3         9/11/2007 3:52:00 PM  HR,Sales      
ABC002   Field Services 4         9/12/2007 8:30:00 AM  MKT           


Another example of a dense rank with a compound primary key

This example is based on the post:

comp.databases.ms-sqlserver
Sep 5, 7:59 am
Grouping similar rows
http://tinyurl.com/3378lk

Here is the create table statement and some rows entered into
an Sql Server 2005 database (courtesy of MVP Hugo Kornelis).

CREATE TABLE TheInput
     (Letter char(1) NOT NULL,
      RowNum int NOT NULL,
      OtherValue int NOT NULL,
      PRIMARY KEY (Letter, RowNum));
go
INSERT INTO TheInput (Letter, RowNum, OtherValue)
SELECT 'A', 1, 50
UNION ALL
SELECT 'A', 2, 50
UNION ALL
SELECT 'A', 3, 100
UNION ALL
SELECT 'A', 4, 50
UNION ALL
SELECT 'A', 5, 100
UNION ALL
SELECT 'A', 6, 100;
INSERT INTO TheInput (Letter, RowNum, OtherValue)
SELECT 'B', 1, 50
UNION ALL
SELECT 'B', 2, 100
UNION ALL
SELECT 'B', 3, 50
UNION ALL
SELECT 'B', 4, 50
UNION ALL
SELECT 'B', 5, 100
UNION ALL
SELECT 'B', 6, 100; 

We now work with the table in Dataphor.

select TheInput; 

Letter RowNum OtherValue 
------ ------ ---------- 
A      1      50         
A      2      50         
A      3      100        
A      4      50         
A      5      100        
A      6      100        
B      1      50         
B      2      100        
B      3      50         
B      4      50         
B      5      100        
B      6      100     

The OP poses his problem as:

Quote
I would like to group rows based on order and similarity in a single
sql-query if possible:

A, 1, 50
A, 2, 50
A, 3, 100
A, 4, 50
A, 5, 100
A, 6, 100

Would come out as:

A, 1, 50, 2 <-- Last value shows number of grouped rows
A, 2, 100, 1
A, 3, 50, 1
A, 4, 100, 2 
UnQuote

Like the previous example, there is no mention in either the question
or the solution of a dense rank, yet that is what the problem calls for,
ie. a dense rank over OtherValue within each Letter and in the ascending
order of RowNum.

We need only check the compound key version of the DenseRank and
IDenseRank operators to insure they accept a table as a parameter
of the same 'type' as TheInput table.

The current compound DenseRank operator accepts a table of type:

table{PKcol1:String,PKcol2:DateTime,Grp:String}

While the Letter column of the TheInput table is the same data type as
PKcol1 (String), the PKcol2 data type (DateTime) is different than the
RowNum column type (Integer). In addition the OtherValue column (Integer),
the target of the rank, is different than the Grp type (String). So all
we have to do is again overload the compound DenseRank operators to accept
the compound key and Grp data types of the TheInput table. It's that simple ☺ .

Here are the operators overloaded to accept a table of type:

table{PKcol1:String,PKcol2:Integer,Grp:Integer}

create operator IDenseRank
 (aTable:table{PKcol1:String,PKcol2:Integer,Grp:Integer}):
       table{PK.col1:String,PK2Min:Integer,PK2Max:Integer,DenseRank:Integer}
begin
result:=table of typeof (result){};            
var D1:=
ToTable(ToList(cursor(
    (
 ToTable(ToList(cursor(
 ToTable(ToList(cursor(aTable order by {PKcol1,PKcol2} )))
                            {PKcol1,PKcol2,Grp,sequence+1 PK12Seq}
     order by {Grp,PK12Seq}  )))  
      add{PK12Seq-(sequence+1) Drank}
       group by {PKcol1,Grp,Drank} add{Min(PKcol2) PK2Min,Max(PKcol2) PK2Max}
      )      
                 order by {PKcol1,PK2Min}  )))  
                   {PKcol1,PK2Min,PK2Max,sequence+1 DenseRank};
result:= 
    D1
     join
      (
        D1 group by {PKcol1} add {Min(DenseRank) OffSet} {PKcol1,OffSet-1 OffSet}
      )                          
        {PKcol1 PK.col1,PK2Min,PK2Max,DenseRank-OffSet DenseRank} ;
end;

create operator DenseRank(aTable:table{PKcol1:String,PKcol2:Integer,Grp:Integer}):
                     table{PKcol1:String,PKcol2:Integer,DenseRank:Integer}
begin
result:=
        table of typeof (result){};                        
result:=
   aTable {PKcol1,PKcol2}
    times
     IDenseRank(aTable)
      where (PKcol1=PK.col1) and (PKcol2 between PK2Min and PK2Max)
            {PKcol1,PKcol2,DenseRank};       
end;

Now we can use the compound dense rank for a table of type TheInput.

select 
 DenseRank(TheInput {Letter PKcol1,RowNum PKcol2,OtherValue Grp}) ;
 
PKcol1 PKcol2 DenseRank 
------ ------ --------- 
A      1      1         
A      2      1         
A      3      2         
A      4      3         
A      5      4         
A      6      4         
B      1      1         
B      2      2         
B      3      3         
B      4      3         
B      5      4         
B      6      4     

select 
 (
  DenseRank(TheInput {Letter PKcol1,RowNum PKcol2,OtherValue Grp})
     {PKcol1 Letter,PKcol2 RowNum,DenseRank}
 )   
  join TheInput
        order by {Letter,RowNum} ; 
        
Letter RowNum DenseRank OtherValue 
------ ------ --------- ---------- 
A      1      1         50         
A      2      1         50         
A      3      2         100        
A      4      3         50         
A      5      4         100        
A      6      4         100        
B      1      1         50         
B      2      2         100        
B      3      3         50         
B      4      3         50         
B      5      4         100        
B      6      4         100        

A simple group by with the Count aggregate answers the OP question.

select 
 (
  DenseRank(TheInput {Letter PKcol1,RowNum PKcol2,OtherValue Grp})
     {PKcol1 Letter,PKcol2 RowNum,DenseRank}
 )   
  join TheInput
    group by {Letter,DenseRank,OtherValue} add{Count() Cnt}
      order by {Letter,DenseRank};
    
Letter DenseRank OtherValue Cnt 
------ --------- ---------- --- 
A      1         50         2   
A      2         100        1   
A      3         50         1   
A      4         100        2   
B      1         50         1   
B      2         100        1   
B      3         50         2   
B      4         100        2   

Summary

The parameter declaration of an operator, ie.

(aTable:table{PKcol1:String,PKcol2:Integer,Grp:Integer})

is called the 'signature' of the operator. Dataphor stores the
same name of an operator for each different signature. In other
words, the same operator name for different parameter types 
(ie. different table types). This is the concept of overloading.
One operator name covering different types (regardless of the
returned result or even if the code differs for different signatures).
This, obviously, significantly benefits developers. For some
types of complicated problems just one clever programmer is all it
takes to write the logic of the operator. To use the operator
other developers merely insure that the signature (parameter type(s))
match their own tables. If not, simply change the signature (and
possibly the data types of the result). It would not be difficult to
overload the dense rank operators to allow for compound keys of
more than two columns. Even the coding change for more than two
PK columns is simple. While overloaded operators are physically
stored multiple times (based on signiture), conceptually there
is but one dense rank operator. When the operator is invoked it
simply looks for the signature (type) that matches the table
being sent in as a parameter. Just think of the many types of
problems typically encountered. Many of these can follow the paradigm
of the dense rank. Imagine a library of these type of operators accessible
from any database(s).

(Feel free to contact me if you would like to see a specific example
 of a type of problem like the dense rank problem).

A final caveat

These operators could be coded in a more sophisticated fashion.
But before the fancy coding, before the modelling of a business problem,
before anything in the database, it all rests on understanding the
key concepts of a relational database like Dataphor. And concepts
like types and variables are just as important as clever code, if
not more so ☺ .

Dataphor - All tables are typed variables

2007-09-09T20:21:00.000-07:00

Everything you work with in D4, every object you define, is a variable.
And since everything is a variable it must typed. Understanding variables
and their types is fundamental to working in the D4 language. There are
5 types: Scalar (like integers and strings), Row, Table, List, and Cursor.
Working with types is not hard. It's all logic and quite simple at that.

Table types

It seems many people have a hard time understanding a table as a variable.
But once you wrap your head around it you'll realize it's just common sense
applied to a database. All we're doing is what you do with a number.

This declares the variable named MyNum to be of type integer.

var MyNum:Integer;

This assigns the number 1 to the variable. So now we can work with MyNum
and its current value of 1.

MyNum:=1;

Why shouldn't we be able to do the same thing with a table?
For example:

var MyLTable:table {ID:Integer,Y:String } ;

We have just defined the variable MyLTable to be a type of table
and specifically a table with column ID of type Integer and column
Y of type String. The '{ID:Integer,Y:String }' is the the heading
and is all that is necessary to define this particular table type.

Just as we assigned the value 1 to the integer variable MyNum we
can assign values to the table variable MyLTable.

The values we assign to a table are 'rows'. So we use the row type
for the assignment.

Suppose we try this:

var MyLTable:table {ID:Integer,Y:String } ;
MyLTable := row { 1 ID,'A' Y };

But this won't work. We are attempting to assign a type row to a
table type. This is analogous to trying to assign a string to a variable
of integer. The two types are incompatible and the assignment wouldn't
work.

var MyNum:Integer;
MyNum:='A';         //This is a type mismatch and will throw an error.

It only makes sense that we can assign a table to a variable of a table
type (and of the same type).
(Of course we can 'insert' into a table which we'll see see later and
'update' a table). This batch of 3 statements shows a type of table
with rows defined with the same heading (column names and data types)
as variable MyLTable on the right of the := that we can assign to MyLTable.
Even the table on the right side of the := is a table variable.

var MyLTable:table {ID:Integer,Y:String } ;
MyLTable := table
                 {
                  row { 1 ID,'A' Y },
                  row { 2 ID,'B' Y }
                  };
select MyLTable;

ID Y
-- -
1  A
2  B

Note that if we want to assign a variable a row value and not a table we
simply define the variable as a row type (with a heading like the heading
required for a table).

var MyLRow:row {ID:Integer,Y:String } ;
MyLRow := row { 1 ID,'A' Y };
select MyLRow;

ID Y
-- -
1  A

The variables defined so far are memory variables. They exist only for
the duration of the batch they are in. Persisted tables can, of course,
be defined. Dataphor uses the concept of a device. A device is an sql
database that Dataphor uses to store table data. For example Sql Server,
Oracle or DB2.
The 'create table' statement is used to persist data in an sql database.
For example, the table SqlServerTable1 uses an Sql Server 2005 database
as data storage.

create table SqlServerTable1
 {
  ID:Integer,
  Y:String,
  key{ID}
 };
 
The SqlServerTable1 table, like the MyLTable table, is a variable. Every
table defined, regardless of where the data is stored, is a table 'variable'.
Also note that only the columns and their data type determine the type of
the table. Other information like keys are not part of the type definition.

Lets assign (persisted) rows to SqlServerTable1.

SqlServerTable1:= table{row{1 ID,'A' Y},row{2 ID,'B' Y}};
select SqlServerTable1; 

ID Y
-- -
1  A
2  B

Lets redefine the definition of the SqlServerTable1 table (variable).

SqlServerTable1:= table{row{3 ID,'A' Y},row{4 ID,'B' Y}};
select SqlServerTable1;

ID Y
-- -
3  A
4  B

Changing the assignment of the table is no different than changing
the assignment of a number to a variable.

var MyNum:Integer;
MyNum:=1;
select MyNum; //1
MyNum:=2;
select MyNum;
2

Always remember that when I refer to a table in D4 it is a table variable.

There are wonderful benefits to this concept. For example, just as
two integer type variables can be compared so too can two table variables.
But you must think in terms of types. When is such a comparison appropriate,
when does it make sense? It only makes sense when the two tables are of the
same type. Just as it only makes sense to compare two integer variables.
Two tables are equal if they have the same columns and corresponding data
types and have the same set of rows (values).

For example, the below select statement displays the rows of MyLTable
since the comparison of tables in the where statement is true (for each row
of MyTable). The two tables are of the same type ({ID:Integer,Y:String})
and have the same rows (values).

var MyLTable:table {ID:Integer,Y:String} ;
MyLTable := table
                 {
                  row {1 ID,'A' Y},
                  row {2 ID,'B' Y}
                  };
var MyLTable1:table {ID:Integer,Y:String} ;
MyLTable1 := table
                 {
                  row {1 ID,'A' Y},
                  row {2 ID,'B' Y}
                  };                 
select MyLTable
 where MyLTable=MyLTable1; //MyLTable=MyLTable1 is true.

ID Y
-- -
1  A
2  B 

Here the two tables are of the same type but do not have the same rows. The
where statement evaluates to false for each row of MyLTable and no rows
are selected.

var MyLTable:table {ID:Integer,Y:String } ;
MyLTable := table
                 {
                  row {1 ID,'A' Y},
                  row {2 ID,'B' Y}
                  };
var MyLTable1:table {ID:Integer,Y:String } ;
MyLTable1 := table { row {1 ID,'A' Y} };                 
select MyLTable
 where MyLTable=MyLTable1; MyLTable=MyLTable1 is false.

ID Y
-- -


Now look at this comparison.

var MyLTable:table {ID:Integer,Y:String } ;
MyLTable := table
                 {
                  row {1 ID,'A' Y},
                  row {2 ID,'B' Y}
                  };
var MyLTable1:table {ID:Integer,Z:String } ;
MyLTable1 := table
                 {
                  row {1 ID,'A' Z},
                  row {2 ID,'B' Z}
                  };                 
select MyLTable
 where MyLTable=MyLTable1; //An invalid comparison, two different table types.
                           //The compiler will raise an error because of the
                           //type 'mismatch'.

Here the two tables are of different type:
{ID:Integer,Y:String} vs.{ID:Integer,Z:String}
One table has a column Y the other a column Z so the comparison
itself is invalid. In other words, it's simply not 'logical' to
compare two different types!

But again we're working with table variables, not a table that is a
file (sql) or a table that is a value (literal). A variable can be
changed which means its type can be changed. One way to change the
type of a table is simply to change a column name. Here we make
the table comparison valid by changing the type of table MyLTable1
to the type of table MyLTable by renaming column Z to Y.

var MyLTable:table {ID:Integer,Y:String} ;
MyLTable := table
                 {
                  row {1 ID,'A' Y},
                  row {2 ID,'B' Y}
                  };
var MyLTable1:table {ID:Integer,Z:String} ;
MyLTable1 := table
                 {
                  row {1 ID,'A' Z},
                  row {2 ID,'B' Z}
                  } ;                 
select MyLTable
 where MyLTable=(MyLTable1 rename {Z Y}); //Now we can compare the tables
                                          //because they are the same type
                                          //(Same column names and data types).

ID Y
-- -
1  A
2  B


Inserting into a table

Previously we created table SqlServerTable1 as a persisted table where
the data is stored in Sql Server. We left it with 2 rows.

select SqlServerTable1;

ID Y
-- -
3  A
4  B

We can of course insert into the table. Here we insert a row.

insert row {10 ID, 'C' Y} into SqlServerTable1;

Inserting a row makes perfect sense. What about inserting multiple rows?

insert
       row {11 ID,'D' Y},
       row {12 ID,'E' Y}
                  into SqlServerTable1;

This does not work. We can insert A row into a table. If we want to insert
multiple rows we can insert each row separately.

insert row {11 ID,'D' Y} into SqlServerTable1;       
insert row {12 ID,'E' Y} into SqlServerTable1;

If we want to insert consecutive rows with a single insert we have to
use the appropriate type to insert into the table. And that type is
a table. So we can insert one table into another.

insert
  table
      {
       row {11 ID,'D' Y},
       row {12 ID,'E' Y}
      }
                  into SqlServerTable1;

Not only do you have to think in terms of types but of the relationship
between types. It is types, their relationships and variables that are
fundamental to a 'relational' database and distinguish D4 from an sql
database, a non-relational database.

The data types of a column of a table

Each column in a tables heading has to have a corresponding data type. This
is usually a Dataphor provided system scalar type. Dataphor provides the
following scalar data types: Decimal, Long, Integer, Short, Byte, Boolean,
String, TimeSpan, DateTime, Date, Time, Money and Binary. But a column type
need not be restricted to these 'scalar' types. For example a 'row' type and
'list' type are as logical as a scalar type. Any 'type' supported by Dataphor
should logically be avaliable for the type of a column. It's that simple :)

Because Sql Server has no support for a 'list' type or 'row' type we
can't create a persisted table in the sql database with these types. But
we can create a persisted table in Dataphor known as a 'session' table.
A session table is like a temporary table (#) in Sql Server. It will
disappear when the session ends. (There are ways to persist such types
in an sql database but that is beyond the scope of this article ☻ ).

Consider session table MySTable with a list and row type and an appropriate
insert statement for a inserting a row.

create session table MySTable
{
  A:Integer,
  B:String,
  LList:list(String),
  LRow:row{X:Integer,Y:String},
  key{A}
}; 
insert row{1 A, 'A1' B, {'J','K','L'} LList, row{10 X,'R1' Y} LRow} into MySTable;

We can extend the idea of the LRow column by using a row type for the columns
of LRow. LRow can be a row consisting of columns which are themselves rows
(we can nest the row type).

create session table MySTable1
{
  A:Integer,
  B:String,
  LRow:row{ R1:row{X1:Integer,Y1:String} , R2:row{X2:Integer,Y2:String} },
  key{A}
}; 
insert row { 1 A,'A1' B,row{ row{10 X1,'S1' Y1} R1,row{20 X2,'S2' Y2} R2 } LRow }
                              into MySTable1;


(I leave the significant benefits of working with tables that contain list
 and row types and just how to work with them for other articles and future
 articles).

The fact that all tables are typed and are variables allows a table to
be passed as a parameter to a procedure just as you would pass a variable
of type integer. For further info on this see:
http://beyondsql.blogspot.com/2007/06/dataphor-13-passing-table-as-parameter.html
http://beyondsql.blogspot.com/2007/08/dataphor-creating-super-function.html

Dataphor - Reply to Michel

2007-08-29T06:02:00.001-07:00

Hello Michel,

It's been a while:)
I do appreciate you taking time to explore this subject!
I hope talented and intellectually mature people like yourself
take a serious interest in this matter.

> So, keep your shields up, :-)

You have my word Kirk :)

>> I decided to take a couple of days in order to get some 'emotional distance
>> toward what you wrote, in your blog, and while I did just that, but not
>> having time to read every thing there, I am still left with an incomplete
>> picture.

Let me give you short overview of the entire subject.
A small group of very talented people, both conceptually and technically,
at Alphora implemented the relational model known as Tutorial D completely
specified in the book:
Databases, Types, and The Relational Model: The Third Manifesto, 3rd edition
by C. J. Date and Hugh Darwenhttp://www.thethirdmanifesto.com/

This relational (non-procedural database access) language is within a Pascal
(declarative/procedural) like language. The language in toto is referred to as
'D4' by alphora. The relational language is conceptually (radically) different
from sql in two fundamental ways. It rests on the concept of 'type' and
'variable'. Whereas sql is absent the concept of 'type' and 'variable'
as it relates to a table. In the absence of type there is only a 'file name',
much like there is only the value of integer 1 in the absence of the
ability to assign integer 1 to a variable. The design of the non-procedural
language is also radically different from sql. There are so many differences
with sql (conceptually, design of language, keys, use of meta-data, constraints,
views, duplicates,procedures etc) that it not possible for me to go into them
all here :) Suffice it to say we are talking qualitive differences not quantitive.

The 'relational' idea of the database is itself in the service of the
overall goal of the whole system. And that is application development.
But a very special method of AD, from procedural to declarative. In other
words, from the traditional procedural method(s) of AD to a methodology
based on 'inference'. It is as if 'performance' as the goal of an sql
optimizer with its emphasis on the physical implementation of sql was
now secondary to the 'logical' model with the emphasis on the compiler
as a logical inference mechanism. In D4 there are no query 'hints' there
are logical 'hints', index hints are replaced with 'key' hints. The user
works with the compiler on a logical level (clarifying which index to
use is replaced with what key(s),reference(s) does a resultant table have).
(As a quick aside, all joins can only be equi-joins. This insures an
unambigious key(s) for the resulting table. Note that the concept
of requiring every result/table (regardless of how it is derived) to
have a key(s) is entirely absent from sql. And yes other predicates can
be used to form tables with other relational operators (see D4 'times' operator).

The ui in Dataphor is capable of making very detailed windows into
the database exclusively based on inference (views which go way beyond
sql, references and meta-data are key components on which inferences are
based).Of course you can modify the gui to your hearts content:) I'm
hoping that the integration between the ui and tables in Access
may lead some to explore the sophisticated inference ideas in Dataphor.
I illustrate these ideas with a mini-application that can be downloaded:
http://beyondsql.blogspot.com/2007/07/dataphor-example-of-rapid-application.html


>>Your points are not about J. C., so even if you want to notify the reader
>>that you take his objections into consideration, you surely can and have to
>>mention it, but not as introduction to the point you develop. I refer to the
>>article where you ... try... to introduce the fact that a table can be a
>>parameter and you start the discussion by bringing one of the poorest answer
>>Joe Celko may have ever done in his entire life, of a mathematician (I say
>>that but take in account that I have a lot of respect for that guy, for the
>>good stuff he produced).

I respect Joe too. I hope you will forgive me if I take some liberties:)
Joe implied the logical implausibility of the idea of a 'super function'
in sql. Of course he is correct, in sql. I used that same idea to show
how logically it fits in a relational database. I used his thought
to 'contrast' sql and D4 on the basis of tables as variables. I think/hope it
worked out:) As a practical matter it is not easy to communicate
many of these ideas. Joe is a known and respected authority. Perhaps the
reader will stick around and see how things worked out:) Also bear in mind
these concepts read in a less than compelling way to most. It is hard to make
them register. That is why I try to show an actual example that derives
from concepts that must be understood for the user to make sense of the
material. I think Joe understands. I have had many exchanges with him:)
For example see:
 (
  microsoft.public.sqlserver.programming
  Tuesday, August 21, 2007
  'Primary key selection'
   http://tinyurl.com/39dgcp
(My point here is sql is immature based on its concept of a key. The key
 is elevated in D4 to where index is in sql. Note that 'all' types including
 tables have a logical 'addressing' mechanism. For tables it is the key(s).
 This concept and its usage is completely alien to sql users).
 I've also used ideas expressed by Itzik Ben-Gan to contrast sql and D4.
 See:
 http://beyondsql.blogspot.com/2007/06/sql-using-dense-rank-for-identifying.html
 So I'm at least in fast company, yourself included :-)
 )

I use MS sql server in much the same way. Some concepts make sense
in sql (some don't:) but take on a whole new meaning in D4. Again I use
the contrast to make my point. In this vein I've used dynamic sql and
'lists' in sql server to name a few. See:
'Sql - History repeats itself simulating lists'
http://beyondsql.blogspot.com/2007/07/sql-history-repeats-itself-simulating.html

>>In my opinion, I must say that for optimization of a query plan, I still find
>>that knowing the table, its physical structure, its indexes, its stats, what
>>is required (what is SELECTed), etc. can make the optimizer find a better plan
>>than when having a ... general ...  table. Not knowing if you SELECT DISTINCT
>>a primary key or a secondary field, as example, can make some difference in
>>the lag the end user is likely to experience, at runtime. Sure, there are
>>cases where the optimized plan will be very fast to obtain (so why not
>>getting it at runtime), or where it is irrelevant, but I cannot say that
>>having a table, as parameter, is on my priority list at all.

The general issue of 'performance' will forever rear its ugly head:(:)
At this point I have little incentive to get into the details of
optimizing D4 code regardless if its queries, non-procedural code
or whatever. The basic concepts are hard enough to get across. I fear
introducing details of how certain D4 constructs will gain a performance
advantage is premature. I would ask that you look at this article for
some perspective on the notion of performance in AD/D4:
http://beyondsql.blogspot.com/2007/06/dataphor-inclusive-sql-vs-exclusive-d4.html

Eventually I will address performance head on.

Let me introduce some additional info. At this point in its development
Dataphor does not possess its own mature 'native' storage (for development
yes, for enterprise work no). Dataphor utilitizes a 'device' for
storage. A device can be one of a number of enterprise sql dbs, ie.
DB2, Oracle, SAS, Sql Server. I use Sql Server as a device. So the device
can simply be the data repository. At the most elementary level all
DDL commands in the synatax of D4 will throw the data into an sql
server database. All DML commands in D4 will, when appropriate,
access data in sql server going thru a client language interface.
The D4 compiler will determine the details of the interaction with
the device. This process is by no means trivial but at the very
least can be transparent to the user. But channels of communication
are availiable for user intervention. A most common one is what
most Access users are familiar with, the pass thru query. So the
user can throw any valid t-sql (queries, procedure calls, ddl) to
the server. For t-sql queries and server side procedures that produce
a result (table) the result will be treated just as if it was derived
via native D4 syntax. Of course there is nothing preventing a user from
using dynamic sql. So we come around in a full circle. If one becomes to
stressed in the relational world of D4 there is always the option to
to go back to the world that time forgot! :-) :-)

>> Your writing style is (still) complex. I know mine is, also, but in this
>> context, my style is not relevant, I think.

You have me at a distint disadvantage. But bear in mind sql has been
around for almost 40 years and talking about it succintly still remains
a burden few can carry. Communicating effectively in the newborn relational
terms, ie types, variables and relvars, is a work in progress. I will
have to take my lumps. But I will try to make effective use of feedback
which I wholeheartedly encourage.

>>I haven't see a 'tour guide'. I am not even sure I started at the right
>>place! Sure, if the purpose is still to start a discussion, that can be
>>expected since we don't know where it will go, but there is already a lot of
>>material (not necessary self contained, I am afraid) and I am still not sure
>>about 'where' in the process of making an application you want to focus. You
>>are not spreading your thought about the user interface, aren't you? Ok, it
>>is NOT SQL, it is maybe Access, but WHAT is it? I am a little bit lost, and
>>have the impression  that I could put my hands on great (I hope) pictures of
>>a movie, but still not sure if the movie is a documentary, or something
>>else.

I understand what you are expressing. Start at the beginning, but just
where is the beginning?
To get a jump start, to actually see what D4 scripts are and the
steps that constitute a working example/application see:
http://beyondsql.blogspot.com/2007/07/dataphor-example-of-rapid-application.html
This is a fully functioning application that includes a ui. But bear in
mind one does not need the 'ui' to explore the system. One can learn
quite a bit about modelling with sql server without the need for a ui.
Creating a single D4 operator/procedure or query will start a user on his way.
Once you start up Dataphor you can tinker with creating tables, queries
or whatever.

I would recommend the serious user:

Go to www.alphora.com and download Dataphor. Install the beast and start
pouring over the documentation to get a feel for this thing (the documentation
is also availiable online). If you have any version of sql server
you can quite easily use it as your data store. You can connect to
a server database and have access to all tables in the database immediately.
For example connect to the Northwind or Pubs database. (If you need a hand
connecting contact me via my blog). Start getting familiar with the
Dataphoria gui. Your going to be doing your development with this ui.
It's straightforward just like query analyzer is. You can easily create
new tables in D4 that will reside on sql server. Start browsing the
relational operators in help. Try a few queries. Your on your way.

My blog
Aside from the mini-app example, the articles consist of pep talks,
explanations and examples (along with their underlying concepts).
The examples can all be copied and pasted into Dataphor and run.
They work:) Think of the examples as snapshots. The show different
slices of constructs and concepts (Dataphor comes with many samples too.
Be sure to check them when you install Dataphor). Joe Celko has said that
at some point in time when learning sql a light in your head will hopefully
appear. Same thing in D4. At some point it will all come together
if you bear with it.


>>It seems you mention an interest for the framework (dot.Net) which is an
>>anti-SQL thing (even if they added some element of SQL in version 3.0),
>>imho. Gone the sub-queries, even gone the joins, just a simple
>>select-from-where and if you need 'join' or 'subqueries', you use the
>>framework objects that refer to these 'simple' statements, and use C#, or
>>whatever, not SQL, to carry the 'joins', but without the memory explosion
>>what SQL-joins do. On the other hand, from the first impression I got, it
>>seems you add complexity to the SQL language. So, it would be like rooting
>>in the wrong soil, no? Since the framework is about simple, simple SQL
>>statements, that is (and C# developers are ready to pay the price by writing
>>complex C# procedural statements, because they know C#, but not SQL, and
>>don't want to invest in a more complex SQL like language). I may have the
>>wrong idea of what you saw, though.

Dataphor is built with C# a typed language. Conceptually it makes a lot of
sense since D4 is a itself a typed system. The concept of type is one of the
foundations of the 'relational' model. In other words it is logical that
D4, with its reliance on type, be built with a framework that is typed.
The fact that D4 is built with a net language in no way implies that
D4 has anything to do with the LINQ/DLINQ project which is what you seem to
be referring to. The idea the MS can 'hide' sql from net developers.
What ever games MS is playing with sql from net has nothing to do with
D4. But it also means that just like channels of communication from D4 to
the storage(device) ie. sql server, there are channels from D4 to net.
So users can write assemblies and interact with the net framework in other
ways. And like pass-thru queries, this communication with net does not
come with any hidden price to be paid:) It can be used to expand the
functionality of D4, for example user defined types defined with a
net language (note that user defined types can be defined quite nicely
from within D4). Aside from channels of communication, the net framework
is 'transparent' to the user. Again MS's communication from net to
sql server is totally independent of D4 and its relational language.

Some parting thoughts.
I have not said anything specific about the relational (query) language.
As an expert sql developers you will find it quite different which
is to be expected. You will also find that some query concepts in
sql will carry over. You will find there are many new constructs,
you will find many sql constructs greatly expanded in functionality
(constraints, views, procedures to name just a few) and you
will discover new ways of doing things in comparison to sql. Most
of the historical objections to sql (both conceptually and design of the
language) are corrected in D4.
Mastery of D4 will take an effort, this is not a trivial exercise.
But I think you will find it is a great picture:)
If you have any questions or just want to chat please feel free to
contact me (thru the blog is fine). People like yourself will only
make this system better.
Lastly, as I state in several articles, D4 does not necessary
eclipse sql. Sql has its place but it is misplaced as a foundation
for application development. For that D4 is best suited.

best,
steve

Dataphor - Creating a Super Function

2007-08-11T21:24:00.000-07:00

The Concept of the Super Function

In the thread:
comp.databases.ms-sqlserver
Jul 25, 6:52 am
'Pass Table as a parameter to a function'
http://tinyurl.com/2j4dvy

Joe Celko makes the following comment on passing a table as a
parameter to a procedure:

>> Is it possible to pass a table as a parameter to a function.  <<

'Please read a book, any book, on data modeling, and RDBMS.  A table is
 an entity or a relationship.  That would mean you have a magical,
 super function that works on Squids, Automobiles, Britney Spears,
 Geographical locations or anything in the whole of creation.'

While his reply is pejorative the name 'super function' is perfectly
valid! It is actually a more catchy name for what is commonly referred
to as 'reusable code'. And such code rests on a fundamental concept:
being able to represent objects as 'variables'. In a database the
objects we are referring to are most notably tables. A table as a
variable allows us to pass it as a parameter to a function (procedure).
A procedure that takes a table as a parameter and returns a virtual
table (result) allows us to realize a procedure as a variable, a super
function. The super function allows an infinite number of representations
of the same logic (in the form of a table) for tables of the same type,
ie. tables of Squids, Automobiles and Britney Spears ☺ This is not possible
in sql since what are talking about as a variable (tables,procedures)
can only be represented as literals in sql. In sql we can only reference
a table by its name since there is no type for a table. An sql
procedure/function dependent on a table can only express logic for
that particular table and hence, like the sql table, is a literal
representation of an object (table) not a variable that can represent
an infinite number of solutions.

An example of a super function in the D4 language of Dataphor.

A common requirement is find a dense rank given that a grouping column
can have repeating values. Procedure DenseRank takes a table as a parameter
with an integer column named PK which represents the tables primary key
and a column named Grp which represents a group column. The operator returns
a table with the primary key column (PK) and an integer column which is
the dense rank (DenseRank). The DenseRank procedures calls another
procedure, IDenseRank, which takes the same DenseRank table as a parameter.
Both procedures are overloaded with the group (Grp) column of the table
defined as a string. The DenseRank super function can therefore represent
the dense rank of any table in the database that has a single integer column
that is unique (not necessarily the primary key) and a column
(either integer or string) that represents a group column. The group column
may very well represent Squids, Automobiles or Britney Spears ☺ It does
not matter as the super function is predicated on column type only. The
logic of the procedure (a dense rank) is independent of what the meaning
of the predicate (rows of the input table) is. This independence is
achieved by the concept of a what a table type is. In other words,
the dense rank super function is only possible by substituting the
significance of the type of a column(s) for the name of a column. In
sql all logical significance rests in the name of a column. The only
significance of the type of a column is computational. The significance
of type, its logical significance, is a major advancement of D4 over
sql and allows objects like super functions. The examples that follow
will make clear what these concepts mean in practice.

(If you are just beginning D4 it is more important to understand the
 the general concepts of database objects being variables and the
 importance of type than specific programming techniques. If you have
 programming questions about the procedure(s) please feel free to post
 them in comments.)

Operator (function) DenseRank returns a table (virtual) with a unique
integer column (PK) and an integer column with the dense rank (DenseRank).

create operator DenseRank(aTable:table{PK:Integer,Grp:Integer}):
                         table{PK:Integer,DenseRank:Integer}
begin
result:=
        table of typeof (result){};                        
result:=
         aTable {PK}
            times
             IDenseRank(aTable)
               where PK between PKMin and PKMax
                 {PK,DenseRank};
end;               

This operator returns a single row for each dense rank. It also has the
starting (min) and ending (max) primary key values of the rank. The ranks
are consecutively numbered in case there is a query that involves a
meaningful ordering of the dense ranks (in the direction of the primary key).

create operator IDenseRank(aTable:table{PK:Integer,Grp:Integer}):
                      table{PKMin:Integer,PKMax:Integer,DenseRank:Integer}
begin
result:=table of typeof (result){}; 
result:=                  
   ToTable(ToList(cursor(
       (
        ToTable(ToList(cursor(
       //Have to get PK consecutively numbered (PKSeq).
          ToTable(ToList(cursor(aTable order by {PK} ))) {PK,Grp,sequence+1 PKSeq}
            order by {Grp,PKSeq}  ))) 
              add{PKSeq-(sequence+1) Drank}
                group by {Grp,Drank} add{Min(PK) PKMin,Max(PK) PKMax}
       )     
                  order by {PKMin}  ))) 
                    {PKMin,PKMax,sequence+1 DenseRank} ; 
end;        

Here we overload the DenseRank and IDenseRank procedures to accept a group
column (Grp) of type string. The overload allows us to essentially ignore
the type of the group column (as long as it's an integer or a string type).
The system will determine which overload of the procedures to use based on
the table (and its group column type) passed.

create operator DenseRank(aTable:table{PK:Integer,Grp:String}):
                         table{PK:Integer,DenseRank:Integer}
begin
result:=
        table of typeof (result){};                        
result:=
         aTable {PK}
            times
             IDenseRank(aTable)
               where PK between PKMin and PKMax
                 {PK,DenseRank};
end;                
 
create operator IDenseRank(aTable:table{PK:Integer,Grp:String}):
                      table{PKMin:Integer,PKMax:Integer,DenseRank:Integer}
begin
result:=table of typeof (result){}; 
result:=                  
   ToTable(ToList(cursor(
       (
        ToTable(ToList(cursor(
         ToTable(ToList(cursor(aTable order by {PK} ))) {PK,Grp,sequence+1 PKSeq}
            order by {Grp,PKSeq}  ))) 
              add{PKSeq-(sequence+1) Drank}
                group by {Grp,Drank} add{Min(PK) PKMin,Max(PK) PKMax}
       )     
                  order by {PKMin}  ))) 
                    {PKMin,PKMax,sequence+1 DenseRank} ; 
end;        

(D4 will store two signatures for each procedure accounting for an integer and
 a string type for Grp. The table returned (PK and DenseRank) is exactly the
 same regardless of Grp type.)

Examples of using the DenseRank super function (the use of the term function
or procedure or operator is a matter of choice, they all mean the same thing
and here they all imply super ☺ )

(All tables use MS Sql Server 2005 as the storage device.)

Consider table Squids:

create table Squids
{
 SquidPK:Integer, //Primary key.
 Squid:Integer,   //Indicates a squid group that can repeat.
 Length:Integer,  //Length of the squid in squidly centimeters.
 Weight:Integer,  //Weight of the squid in squidly globs.
 key{SquidPK}
};
insert
table
{
row{1 SquidPK,1 Squid,5 Length,10 Weight},
row{5,1,4,6},
row{7,2,3,8},
row{9,2,7,3},
row{11,3,5,8},
row{12,3,6,7},
row{15,3,6,8},
row{17,1,3,5},
row{19,1,6,7},
row{21,2,8,5},
row{23,2,6,8},
row{25,2,6,8}
} into Squids;

select Squids;

SquidPK Squid Length Weight
------- ----- ------ ------
1       1     5      10    
5       1     4      6     
7       2     3      8     
9       2     7      3     
11      3     5      8     
12      3     6      7     
15      3     6      8     
17      1     3      5     
19      1     6      7     
21      2     8      5     
23      2     6      8     
25      2     6      8     

To obtain the dense rank we need:
 1. compatibility (type) of columns SquidPK and Squid with the columns of
    the generic table parameter aTable (PK,Grp) which we have.
 2. the elimination of all columns except the primary key and group.
 3. header agreement (column name equivalency) between the table passed
    and the generic names of the columns in the function.
   
We can accomplish 2 and 3 using remove and rename and obtain the ranks with:

select DenseRank(Squids remove{Length,Weight} rename {SquidPK PK,Squid Grp});

We can even combine the remove and replace into one statement using specify { }.

select DenseRank(Squids{SquidPK PK,Squid Grp});

PK DenseRank
-- ---------
1  1        
5  1        
7  2        
9  2        
11 3        
12 3        
15 3        
17 4        
19 4        
21 5        
23 5        
25 5        

What we have is the variable DenseRank representing the dense ranks of
Squids. But the variable is still generic in the sense that PK is not
meaningful for table Squids. So we simply reverse the process used in
calling the function, we rename the PK column to the primary key column
in Squids (SquidPK):

select DenseRank(Squids{SquidPK PK,Squid Grp}) {PK SquidPK,DenseRank}; 

SquidPK DenseRank
------- ---------
1       1        
5       1        
7       2        
9       2        
11      3        
12      3        
15      3        
17      4        
19      4        
21      5        
23      5        
25      5        

By joining the Squids table to variable DenseRank we can represent
all the data of Squids along with its dense rank:

select
 (DenseRank(Squids{SquidPK PK,Squid Grp}) {PK SquidPK,DenseRank})
//A natural join using the primary key (SquidPK) from tables DenseRank and Squids.
  join
   Squids
    {SquidPK,Squid,DenseRank,Length,Weight};

SquidPK Squid DenseRank Length Weight
------- ----- --------- ------ ------
1       1     1         5      10    
5       1     1         4      6     
7       2     2         3      8     
9       2     2         7      3     
11      3     3         5      8     
12      3     3         6      7     
15      3     3         6      8     
17      1     4         3      5     
19      1     4         6      7     
21      2     5         8      5     
23      2     5         6      8     
25      2     5         6      8  

Grouping by Squid and the dense rank (DenseRank) allows aggregate functions
to summarize the data:

select
 (
  (DenseRank(Squids{SquidPK PK,Squid Grp}) {PK SquidPK,DenseRank})
  join
   Squids
//We rename DenseRank to SquidRank in the specify statement.
    {SquidPK,Squid,DenseRank SquidRank,Length,Weight}
 )  
     group by {Squid,SquidRank}
       add{Count() Cnt,Avg(Length) AvgLen,Avg(Weight) AvgWt}
         redefine {AvgLen:=Round(AvgLen,1),AvgWt:=Round(AvgWt,1)};

Squid SquidRank Cnt AvgLen AvgWt
----- --------- --- ------ -----
1     1         2   4.5    8    
1     4         2   4.5    6    
2     2         2   5      5.5  
2     5         3   6.7    7    
3     3         3   5.7    7.7  
   
We can do cars as well as squid:

create table Cars
{
  ID :Integer,
  Car:String,
  Price:Money,
  key{ID}
};
insert
table
{
 row{ 1 ID,'Impala' Car,$18000 Price},
 row{ 2,'Impala',$19000},
 row{ 3,'Impala',$17000},
 row{ 5,'Mustang',$26000},
 row{ 7,'Mustang',$25000},
 row{ 9,'Impala',$16000},
 row{11,'Impala',$14000},
 row{13,'Mustang',$22000},
 row{17,'Mustang',$21000},
 row{19,'Mustang',$28000},
 row{23,'Mustang',$25000},
 row{29,'Impala',$20000},
 row{31,'Impala',$21000},
 row{37,'Impala',$16000}
} into Cars;

select DenseRank(Cars{ID PK,Car Grp}) {PK ID,DenseRank};
 
ID DenseRank
-- ---------
1  1        
2  1        
3  1        
5  2        
7  2        
9  3        
11 3        
13 4        
17 4        
19 4        
23 4        
29 5        
31 5        
37 5      

select
  (DenseRank(Cars{ID PK,Car Grp}) {PK ID,DenseRank})
    join
     Cars
      {ID,DenseRank,Car,Price};
 
ID DenseRank Car     Price     
-- --------- ------- ----------
1  1         Impala  $18,000.00
2  1         Impala  $19,000.00
3  1         Impala  $17,000.00
5  2         Mustang $26,000.00
7  2         Mustang $25,000.00
9  3         Impala  $16,000.00
11 3         Impala  $14,000.00
13 4         Mustang $22,000.00
17 4         Mustang $21,000.00
19 4         Mustang $28,000.00
23 4         Mustang $25,000.00
29 5         Impala  $20,000.00
31 5         Impala  $21,000.00
37 5         Impala  $16,000.00

(Note that you can't just rename columns in sql. When you rename a column
 in sql the table is recreated with the column name. This is because the
 column name is inseparable from the table name. This is a consequence of
 sql tables being literal or just a file as opposed to being a variable.)

Like any function DenseRank is only concerned with the parameter (table)
being passed. The PK column need not be a column stored in the database
as belonging to a table(s) but can be a virtual column that is generated
from an expression. The fact that the parameter is based on an expression
does not in any way invalidate the meaningfulness of the dense rank function.

So instead of Britney Spears consider the Orders table in the Northwind database.
Suppose we want a dense rank over customers. We can use an expression that
orders the table by CustomerID and within customers by OrderID. The following
expression uses the generated sequence number (renamed to PK) for the ordering
as a unique key from which we can get a dense rank over customers (note that
the DenseRank function can return a dense rank regardless of whether or not
the group column (Grp) repeats or not.)

select
  (ToTable(ToList(cursor(Orders return 12 by {CustomerID,OrderID}
                                     with {IgnoreUnsupported = 'true'})))
          {sequence+1 PK,OrderID,CustomerID,EmployeeID,ShipCountry});        


PK OrderID CustomerID EmployeeID ShipCountry
-- ------- ---------- ---------- -----------
1  10643   ALFKI      6          Germany    
2  10692   ALFKI      4          Germany    
3  10702   ALFKI      4          Germany    
4  10835   ALFKI      1          Germany    
5  10952   ALFKI      1          Germany    
6  11011   ALFKI      3          Germany    
7  10308   ANATR      7          Mexico     
8  10625   ANATR      3          Mexico     
9  10759   ANATR      3          Mexico     
10 10926   ANATR      4          Mexico     
11 10365   ANTON      3          Mexico     
12 10507   ANTON      7          Mexico     

Based on the expression we can get the dense rank for customers (we only
return the first 12 rows.)

select
  DenseRank(
            (ToTable(ToList(cursor(Orders return 12 by {CustomerID,OrderID}
                                      with {IgnoreUnsupported = 'true'})))
              {sequence+1 PK,CustomerID Grp})
           ) ;
          
PK DenseRank
-- ---------
1  1        
2  1        
3  1        
4  1        
5  1        
6  1        
7  2        
8  2        
9  2        
10 2        
11 3        
12 3               
          
And just like squid and cars we can combine the DensRank function (variable)
with the expression (variable) to work with it in any form we want (ie a memory
variable, a view or perhaps another function).

create view CustomerRanks 
//Note that procedures can be used in views.
//We could remove the PK column from the result if we wanted.
  DenseRank(
           (ToTable(ToList(cursor(Orders return 12 by {CustomerID,OrderID}
                                     with {IgnoreUnsupported = 'true'})))
             {sequence+1 PK,CustomerID Grp})
           )
            join  //The join is a natural join on PK.   
              (
               ToTable(ToList(cursor(Orders return 12 by {CustomerID,OrderID}
                                        with {IgnoreUnsupported = 'true'}))
              )
                {sequence+1 PK,OrderID,CustomerID,EmployeeID,ShipCountry}); 

PK DenseRank OrderID CustomerID EmployeeID ShipCountry
-- --------- ------- ---------- ---------- -----------
1  1         10643   ALFKI      6          Germany    
2  1         10692   ALFKI      4          Germany    
3  1         10702   ALFKI      4          Germany    
4  1         10835   ALFKI      1          Germany    
5  1         10952   ALFKI      1          Germany    
6  1         11011   ALFKI      3          Germany    
7  2         10308   ANATR      7          Mexico     
8  2         10625   ANATR      3          Mexico     
9  2         10759   ANATR      3          Mexico     
10 2         10926   ANATR      4          Mexico     
11 3         10365   ANTON      3          Mexico     
12 3         10507   ANTON      7          Mexico     

Summary

Hopefully you can see that Joe Celko's idea of 'a magical super function
that works on Squids, Automobiles, Britney Spears, Geographical locations
or anything in the whole of creation' is not magical but very real! What
is meant by magic is simply the concept of a table as a variable and
the underlying importance of columns and their types. There is nothing
magical going on at all. The idea of a variable holding any integer or
string is just extended in the D4 language to tables. Sql users have been
trying to simulate this concept for a long time, perhaps unwittingly. The
sql use of dynamic sql is an attempt to use variable sql, the attempt
to go beyond the simple static (file) structure to use tables as variables.

Finally, in D4 when using tables as parameters to functions all such
functions can be super functions. It is but one way to turn a database
into a super database. And can you guess what type of man they call
such programmers ☺

steve

Learning D4 is like learning two systems, D4 and relearning sql all over
again. D4 will bring a new clarity to sql and will make clear when the
use of each system is most appropriate. It is not a one or the other
contest, but a learning of when to use which where.

Dataphor - Simplifying Relational Division

2007-07-17T21:30:00.000-07:00

If your not familiar with relational division a popular article
for the translation of the concept into sql is:

Relational Division
by Joe Celko

Basically relational division is the question of whether a set
of predicates exist in a given table. Generally the predicates
represent a set of 'rows' and therefore the question becomes
does a predicate in the form of a 'table' exist in another
table. The predicate rows are referred to as the divisor table,
the given table as the dividend table and the result as the 
quotient table. Hence the idea of division by tables. 

Since sql does not support a table 'type' various solutions
have been offered to target the rows of the given (dividend)
table. Most often the means of targeting rows are the use
of aggregate functions like count. Because sql cannot directly
compare one set of rows (ie. a table) to another set of rows (a table)
the forced simulation is not unlike the attempt to simulate
a list with the substring function as explained here. The resulting
sql solutions can at the very least be unintuitive and convoluted 
and at the very most not even possible to specify (at least for
most human beings ☺. 

Given that Dataphor supports direct table comparisons the logic
of a relational division problem can be greatly simplified.

This example uses MS Sql Server 2005 and is based on the post:

Thursday, July 12, 2007 11:07 PM
microsoft.public.sqlserver.programming
Expert Challenge - Complex Join, Group By, Having

Here is some sample data. This is the dividend table.

create table CTable
{
 ID:Integer,
 Answer:String,
 Wcnt:Integer,
 Word:String,
 key{ID}
}; 
insert
table
{
row{1 ID,'first' Answer, 2 Wcnt,'george' Word},
row{2,'first', 2, 'burns'},
row{3,'second',2, 'burns'},
row{4,'second',2, 'burns'},
row{5,'third', 3,    'go'},
row{6,'third', 3,'george'},
row{7,'third', 3, 'burns'},
row{8,'fourth',2,  'fred'},   
row{9,'fourth',2,'george'}  
} into CTable;

Given a match (divisor) table:

(test #1)
declare @match table([ID] INT IDENTITY, [word] varchar(20))
insert @match([word] ) select 'burns'
insert @match([word] ) select 'george'

what the op refers to as a 'match' table, the relational division
question becomes find the rows in the dividend table that correspond
to the divisor rows. And it follows that we compare the Word and
ID columns of the tables. 

We now make two important observations:
1. The Answer column of the dividend table defines rows that belong
   together. These rows form a table.
2. Since we are interesting in comparing columns of two tables we note
   that it is unfeasible to try to compare the ID columns of the dividend
   and divisor tables since the ID column of the divisor table can take
   any value based on the Identity function. With a little insight we
   see that we can compare them if we first 'standardize' the ID column
   in both tables. We can do this by computing the 'rank' of Words in
   each table and compare the ranks instead of IDs. We also note that
   the way the original problem was posed the Word column is of primary
   significance. In other words, it is the presence of the divisor Words
   that determines a match. There is no significance attached to ID, we
   are simply standarizing it to include it in a comparison. Because it 
   has no meaning as to determining a match which ever way we standarize 
   as long as we are consistent within both tables it will be sufficient.

Sql solution   

Here is one of many sql solutions. The row_number() function is used
to standarize the ID columns in both tables and Rank is used in the
comparison. 

-- Match/divisor table (test#1).
declare @match table([ID] INT IDENTITY, [word] varchar(20))
insert @match([word] ) select 'burns'
insert @match([word] ) select 'george'

select C.ID,C.Answer,C.Wcnt,C.Word
from CTable as C
where C.Answer in
(
select B.Answer 
from
-- Note that if we order by 'Word desc' for the ranks in both selects it
-- will not make a difference, ie. (consistency for ranks between tables).
(select ID,Word,row_number()over(order by Word) Rank
     from @match as M) as A
right join
(select ID,Answer,Wcnt,Word,
    row_number()over(partition by Answer order by Word) Rank
     from CTable) as B
on A.Rank=B.Rank and A.Word=B.Word
group by B.Answer
having (count(A.rank)=count(*)) and (count(*)=(select count(*) from @match))
)

ID          Answer               Wcnt        Word                 
----------- -------------------- ----------- -------------------- 
1           first                2           george
2           first                2           burns

Note that the join is necessary but not sufficient. The rows of Answer
must be further restricted by aggregate comparisons using the count
aggregate. This methodology rests on being able to express a quotient
table using aggregate functions. But as the criteria for such a table
becomes increasingly more complex so does the query. It is interesting
that proponents of this methodology think it is quite helpful in 
understanding relation divison. For example the authors of:

A Simpler (and Better) SQL Approach to Relational Division

state 'We believe the syntactical construction of Q0 allows the student to
grasp the concepts of implementing SQL division in a more intuitive way'.
Perhaps most telling is their framing of the question of relational division
where they state:

'A common type of database query requires one to find all tuples of some table
 that are related to each and every one of the tuples of a second group.'
 
This seems to a reflection of the fact that there is no direct way to
compare tables in sql. What would appear much more to the point is simply:

'A common type of database query is to compare two tables'.

And this is precisely what we will do in the D4 language of Dataphor.

Developing a D4 solution

Keep in mind the very precise and simple definition of table equality:
Two tables are equal if they have the same named columns of the same 
data types and they have the same set of rows. In other words, they
are the same 'type' and have the same rows.

The following simple D4 query illustrates how easy relational division
can be. The query ignores ID and only compares tables based on Word.
It uses match test#1:

select
//Each row in CTable that has the particular Answer value given by the where
//predicate being true will be returned by the having operator.
CTable 
having
 ( 
//Every Answer value forms a (dividend) table. We get the Answer value 
//by comparing the match (divisor) table to each table formed by a 
//particular Answer.
 CTable {Answer AnswerI} 
//We compare two tables, each row in the match table must exist
//in the table formed by an Answer value.
  where 
   (table{row{'burns' Word},row{'george'}}) //match table.
   = 
  ((CTable where Answer=AnswerI with { ShouldSupport="false"}) {Word} )
   {AnswerI Answer }
 );   

ID Answer Wcnt Word   
-- ------ ---- ------ 
1  first  2    george 
2  first  2    burns  

The above query will work for matches #1 and #3 but will fail for #2.

select
 CTable 
  having
  ( 
   CTable {Answer AnswerI} 
    where 
  //Using test #2.
    (table{row{'burns' Word},row{'burns'}}) 
     = 
   ((CTable where Answer=AnswerI with { ShouldSupport="false"}) {Word} )
   {AnswerI Answer }
 );   
-- Internal Index Error: "Duplicate key violation."

Which makes the point that ID is necessary given duplicate Words.

Now lets get serious with how we're going to do this example in D4 ☺.

First we're going to create a view in D4 using an sql pass-thru query with
row_number(). This will give us a rank (from 1 to Wcnt) in the direction of
Word for each Answer (the partition column).

create view CTableViewWord 
SQLQuery('select ID,Answer,Wcnt,Word,
    row_number()over(partition by Answer order by Word) Rank
     from CTable','key{Answer,Rank}') {ID,ToInteger(Rank) Rank,Answer,Wcnt,Word};

select 
 CTableViewWord 
        join
           (CTable group by {Answer} add{Min(ID) Asort})
             with {IgnoreUnsupported = 'true'}
             return 50 by {Asort,Rank}
                remove{Asort};

ID Rank Answer Wcnt Word   
-- ---- ------ ---- ------ 
2  1    first  2    burns  
1  2    first  2    george 
3  1    second 2    burns  
4  2    second 2    burns  
7  1    third  3    burns  
6  2    third  3    george 
5  3    third  3    go     
8  1    fourth 2    fred   
9  2    fourth 2    george 

D4 operator Words is the sql query expressed in a relational way. It too
standarizes ID by using a rank for comparison. Like the sql query it
only finds tables where there is the same occurrance of the Words. The
values of ID are not pertinent to the intent of the problem. The intent
is only to find the same Words ignoring the IDs. The operator takes a match
table as an argument and returns matching tables from CTable based on Answers.

//Matches with Word occurrence only.
create operator Words(MatchTable:table{ID:Integer,Word:String}):typeof(CTable)
begin      
result:=table of typeof(result){};  
//Create a table variable with a rank in the direction of Word. This is similar
//to sql row_number() function.               
var MatchTableRank:= 
              (ToTable(ToList(cursor(MatchTable order by {Word})))
                    {sequence+1 Rank,Word}) ;
var RTable:=
CTable
  having
        ( 
         (CTable {Answer AnswerI})
          where 
           MatchTableRank
            =
             (  
              (CTableViewWord where Answer=AnswerI)
               {Rank,Word} 
             ) with {IgnoreUnsupported = 'true'} 
              {AnswerI Answer} 
        );
//Result table will show nil values if RTable is empty.
if exists(RTable) then
result:=RTable
 else
  insert row{nil ID,nil Answer,nil Wcnt,nil Word} into result;
end;

To clarify just what operator Words is doing lets insert some more data
into table CTable.

insert
table
{
row{10 ID,'fifth' Answer,3 Wcnt,'burns' Word}, 
row{11,'fifth', 3,   'burns'}, 
row{12,'fifth', 3,   'burns'}, 
row{13,'sixth',4,   'arthur'},  
row{14,'sixth',4,   'arthur'}, 
row{15,'sixth',4,    'burns'}, 
row{16,'sixth',4,    'burns'}, 
row{17,'seventh',4,  'burns'},  
row{18,'seventh',4,  'burns'}, 
row{19,'seventh',4, 'arthur'}, 
row{20,'seventh',4, 'arthur'}, 
row{21,'eight',4,   'arthur'},  
row{22,'eight',4,    'burns'}, 
row{23,'eight',4,   'arthur'}, 
row{24,'eight',4,    'burns'}, 
row{25,'nineth',4,   'burns'},  
row{26,'nineth',4,   'burns'}, 
row{27,'nineth',4,   'burns'}, 
row{28,'nineth',4,  'arthur'}, 
row{29,'tenth',4,    'burns'},  
row{32,'tenth',4,    'burns'}, 
row{35,'tenth',4,    'burns'}, 
row{38,'tenth',4,   'arthur'}, 
row{39,'eleventh',4, 'burns'},  
row{41,'eleventh',4, 'burns'}, 
row{43,'eleventh',4, 'burns'}, 
row{45,'eleventh',4,'arthur'} 
} into CTable;

Examples:

select Words(table{row{11 ID,'burns' Word},row{22,'burns'}});

ID Answer Wcnt Word  
-- ------ ---- ----- 
3  second 2    burns 
4  second 2    burns 

select Words(table{row{11 ID,'burns' Word},row{12,'burns'},row{17,'burns'}});

ID Answer Wcnt Word  
-- ------ ---- ----- 
10 fifth  3    burns 
11 fifth  3    burns 
12 fifth  3    burns 

The following three selects all return the same table since table
equality is based on the occurrence of the three Words. The value of 
ID and the sequence of Words in the match table is immaterial since the
comparison of Rank between the divisor (match) and dividend tables is
based on Word.

select Words(table{row{11 ID,'burns' Word},row{12,'george'},row{17,'go'}});
select Words(table{row{11 ID,'go' Word},row{12,'george'},row{17,'burns'}});
select Words(table{row{12 ID,'george' Word},row{13,'go'},row{14,'burns'}});

ID Answer Wcnt Word   
-- ------ ---- ------ 
5  third  3    go     
6  third  3    george 
7  third  3    burns  

These two selects return 3 Answers and conceptually 3 tables because the two
Words, 'arthur' and 'burns', occur twice within each Answer.

select Words(table{row{11 ID,'arthur' Word},row{17,'burns'},row{20,'arthur'},row{21,'burns'}});
select Words(table{row{11 ID,'burns' Word},row{17,'arthur'},row{20,'arthur'},row{21,'burns'}});

ID Answer  Wcnt Word   
-- ------- ---- ------ 
13 sixth   4    arthur 
14 sixth   4    arthur 
15 sixth   4    burns  
16 sixth   4    burns  
17 seventh 4    burns  
18 seventh 4    burns  
19 seventh 4    arthur 
20 seventh 4    arthur 
21 eight   4    arthur 
22 eight   4    burns  
23 eight   4    arthur 
24 eight   4    burns  

The idea of using the aggregate count in sql as a solution to relational
division problems is well suited to finding occurrences of things, ie. Words.
With D4 it is obviously not necessary to 'invent' an idea like count.
With table comparisons it is not necessary to go beyond the 'data' in the rows.

Up till now the ID value has not been meaningful. Lets change that. Here
we want to match the occurrence of Word and the ID column. Said another way,
we want to match the sequence of Words within an Answer.

We create another view where the rank is now based on ID.

create view CTableViewID 
SQLQuery('select ID,Answer,Wcnt,Word,
    row_number()over(partition by Answer order by ID) Rank
     from CTable','key{Answer,Rank}') {ID,ToInteger(Rank) Rank,Answer,Wcnt,Word};
     
select 
 CTableViewID 
        join
           (CTable group by {Answer} add{Min(ID) Asort})
             with {IgnoreUnsupported = 'true'}
              where Answer in ({'first','second','third','fourth','sixth','eight'})
               return 50 by {Asort,Rank}
                remove{Asort};
     
ID Rank Answer Wcnt Word   
-- ---- ------ ---- ------ 
1  1    first  2    george 
2  2    first  2    burns  
3  1    second 2    burns  
4  2    second 2    burns  
5  1    third  3    go     
6  2    third  3    george 
7  3    third  3    burns  
8  1    fourth 2    fred   
9  2    fourth 2    george 
13 1    sixth  4    arthur 
14 2    sixth  4    arthur 
15 3    sixth  4    burns  
16 4    sixth  4    burns  
21 1    eight  4    arthur 
22 2    eight  4    burns  
23 3    eight  4    arthur 
24 4    eight  4    burns  

Operator IDandWord is similar to operator Words but here Rank reflects
a meaningful sequence. Now the order of the match table is relevant to
the equality of the table comparison.

create operator IDandWord(MatchTable:table{ID:Integer,Word:String}):typeof(CTable)
begin      
result:=table of typeof(result){};  
//Create a table variable with a rank in the direction of ID. This is similar
//to sql row_number() function.               
var MatchTableRank:= 
              (ToTable(ToList(cursor(MatchTable order by {ID})))
                    {sequence+1 Rank,Word}) ;
var RTable:=
CTable
  having
        ( 
         (CTable {Answer AnswerI})
          where 
           MatchTableRank
            =
             (  
              (CTableViewID where Answer=AnswerI)
               {Rank,Word} 
             ) with {IgnoreUnsupported = 'true'} 
              {AnswerI Answer} 
        );
//Result table will show nil values if RTable is empty.
if exists(RTable) then
result:=RTable
 else
  insert row{nil ID,nil Answer,nil Wcnt,nil Word} into result;
end;

Here the sequence of the ID/Word combination in the match table, when
standarized, is the same as the sequence in CTable so we have a match.

select IDandWord(table{row{11 ID,'go' Word},row{12,'george'},row{17,'burns'}});

ID Answer Wcnt Word   
-- ------ ---- ------ 
5  third  3    go     
6  third  3    george 
7  third  3    burns  

But these sequences of ID/Word do not match any sequence in CTable.

select IDandWord(table{row{11 ID,'burns' Word},row{12,'george'},row{17,'go'}});
select IDandWord(table{row{12 ID,'george' Word},row{13,'go'},row{14,'burns'}});

ID         Answer     Wcnt       Word       
---------- ---------- ---------- ---------- 
<No Value> <No Value> <No Value> <No Value> 

select IDandWord(table{row{11 ID,'arthur' Word},row{17,'burns'},row{20,'arthur'},row{21,'burns'}});

ID Answer Wcnt Word   
-- ------ ---- ------ 
21 eight  4    arthur 
22 eight  4    burns  
23 eight  4    arthur 
24 eight  4    burns  

No match:
select IDandWord(table{row{11 ID,'burns' Word},row{17,'arthur'},row{20,'arthur'},row{21,'burns'}});

ID         Answer     Wcnt       Word       
---------- ---------- ---------- ---------- 
<No Value> <No Value> <No Value> <No Value> 

Note that operators Words and IDandWord are the same except for the view
being used, the type of standarizing on the ID column. The sql solution
implies that aggregates would have to test for the sequence of IDs given
the Word values. This seems to mean that sql has to use a separate set of
constructs for each meaningful column involved in the match. And this
certainly raises the complexity level of such a query.

Now lets raise the bar one more notch in matching. Now not only the
sequence of Words is meaningful (Words and IDs) but so is the relationship
between the IDs. Given the three Answers that match the sequence of Words:

select 
 IDandWord(table{row{11 ID,'burns' Word},row{17,'burns'},row{20,'burns'},row{21,'arthur'}});

ID Answer   Wcnt Word   
-- -------- ---- ------ 
25 nineth   4    burns  
26 nineth   4    burns  
27 nineth   4    burns  
28 nineth   4    arthur 
29 tenth    4    burns  
32 tenth    4    burns  
35 tenth    4    burns  
38 tenth    4    arthur 
39 eleventh 4    burns  
41 eleventh 4    burns  
43 eleventh 4    burns  
45 eleventh 4    arthur 

We want to be able to distinguish (match) each of these Answer tables. To do this we
have to examine the relationship between ID values. The only way to target each
Answer is to test the difference between the IDs. So we have to test for differences
of 1,2 or 3.

Operator BetweenIDs is similar to operator IDandWord but adds a column of the
difference between consecutive IDs. The table comparison now involves looking
at three columns, Word, Rank and TestID.

create operator BetweenIDs(MatchTable:table{ID:Integer,Word:String}):typeof(CTable)
begin     
result:=table of typeof(result){};  
var MatchTableRank:=
        (ToTable(ToList(cursor(MatchTable order by {ID})))
          {ID,sequence+1 Rank,Word}) ;
//Compare the IDs (next to current) in sequence (ascending order) of IDs.
//An indexer ([]) expression is used to directly access a particular Rank.
var MatchTableID:=
           MatchTableRank          
            add{IfNil(((MatchTableRank adorn{key{Rank}})[Rank+1].ID-
              (MatchTableRank adorn{key{Rank}})[Rank].ID),0) TestID} 
                  {Rank,Word,TestID} ;       
var RTable:=
CTable
  having
        ( 
         (CTable {Answer AnswerI})
          where
           MatchTableID
            =
             (  
               (CTableViewID where Answer=AnswerI)
                   add{IfNil((CTableViewID[AnswerI,Rank+1].ID-
                                     CTableViewID[Answer,Rank].ID),0) TestID} 
                   {Rank,Word,TestID} 
             )  with {IgnoreUnsupported = 'true'} 
              {AnswerI Answer} 
        );
var Resultable:=table of typeof(CTable){}; 
//Result table will show nil values if RTable is empty.
if exists(RTable) then
result:=RTable
 else
  insert row{nil ID,nil Answer,nil Wcnt,nil Word} into result;
end;

Match consecutivec IDs.
select BetweenIDs(table{row{10 ID,'burns' Word},row{11,'burns'},row{12,'burns'},row{13,'arthur'}});

ID Answer Wcnt Word   
-- ------ ---- ------ 
25 nineth 4    burns  
26 nineth 4    burns  
27 nineth 4    burns  
28 nineth 4    arthur 

Match IDs separated by 2.
select BetweenIDs(table{row{1 ID,'burns' Word},row{3,'burns'},row{5,'burns'},row{7,'arthur'}});

ID Answer   Wcnt Word   
-- -------- ---- ------ 
39 eleventh 4    burns  
41 eleventh 4    burns  
43 eleventh 4    burns  
45 eleventh 4    arthur 

Match IDs separated by 3.
select BetweenIDs(table{row{4 ID,'burns' Word},row{7,'burns'},row{10,'burns'},row{13,'arthur'}});

ID Answer Wcnt Word   
-- ------ ---- ------ 
29 tenth  4    burns  
32 tenth  4    burns  
35 tenth  4    burns  
38 tenth  4    arthur 

There is no match for these divisor tables:

select BetweenIDs(table{row{10 ID,'burns' Word},row{11,'burns'},row{12,'burns'},row{14,'arthur'}});
select BetweenIDs(table{row{2 ID,'burns' Word},row{3,'burns'},row{5,'burns'},row{7,'arthur'}});
select BetweenIDs(table{row{14 ID,'burns' Word},row{11,'burns'},row{8,'burns'},row{5,'arthur'}});

ID         Answer     Wcnt       Word       
---------- ---------- ---------- ---------- 
<No Value> <No Value> <No Value> <No Value> 


Hopefully you can see how D4 can take the mystery out of relational division.

bye for now,
steve

For those who like to explore D4 here is a bonus operator ☺. Operator
WordsLimit is similar to operator Words but limits the possible Answer values
used for comparing the divisor and dividend tables.

//This operator limits possible Answers for the table comparison.
create operator WordsLimit(MatchTable:table{ID:Integer,Word:String}):typeof(CTable)
begin     
result:=table of typeof(result){}; 
//Get the Word/ID  combination corresponding to the last Rank (when ording by Word)
//for each Answer.
var LastRow:=
    CTableViewWord 
     having 
       (
        CTableViewWord 
        group by {Answer} add{Max(Rank) Rank}
       )   
        {ID,Answer,Wcnt,Word,Rank};
var MatchTableRank:= 
              (ToTable(ToList(cursor(MatchTable order by {Word})))
                    {sequence+1 Rank,Word}) ;
//Get Last row of Match table                    
var LRow:=
        ((MatchTableRank adorn{key{Rank}}) return 1 by {Rank desc})[];
//PTable are Answer(s) from CTable where the last row of an Answer is also
//the last row of the MatchTable, ie the Rank is the same and the Word is
//the same as the last row in the MatchTable. It returns only those Answer(s)
//that 'could' be true in a table comparison using all the rows of MatchTable
//and CTable. It limits the possible Answer(s) to only those that could be
//true.
var PTable:=
CTable {Answer}
where
IsNotNil
(
(LastRow adorn{key{Answer,Rank,Word}})[Answer,LRow.Rank,LRow.Word by{Answer,Rank,Word}]
) with {IgnoreUnsupported = 'true'} ;
//Now using PTable which reduces number of table comparisons.
//Note that the entire 'having' relation is derived independently of CTable.
//It must be derived first so the rows of CTable can be compared to the 'having'
//relation.
var RTable:=
     CTable
      having
       ( 
        (PTable {Answer AnswerI})
          where 
           MatchTableRank
             =
            ( 
             (CTableViewWord where Answer=AnswerI)
              {Rank,Word} 
            ) 
              {AnswerI Answer} 
        ) with {IgnoreUnsupported = 'true'} ;
var Resultable:=table of typeof(CTable){}; 
//Result table will show nil values if RTable is empty.
if exists(RTable) then
result:=RTable
 else
  insert row{nil ID,nil Answer,nil Wcnt,nil Word} into result;
end;

select WordsLimit(table{row{10 ID,'burns' Word},row{13,'burns'},row{16,'burns'},row{19,'arthur'}});

ID Answer   Wcnt Word   
-- -------- ---- ------ 
25 nineth   4    burns  
26 nineth   4    burns  
27 nineth   4    burns  
28 nineth   4    arthur 
29 tenth    4    burns  
32 tenth    4    burns  
35 tenth    4    burns  
38 tenth    4    arthur 
39 eleventh 4    burns  
41 eleventh 4    burns  
43 eleventh 4    burns  
45 eleventh 4    arthur 

select WordsLimit(table{row{10 ID,'arthur' Word}});

ID         Answer     Wcnt       Word       
---------- ---------- ---------- ---------- 
<No Value> <No Value> <No Value> <No Value>

Dataphor - An example of rapid application development

2007-07-13T22:57:00.000-07:00

The SqlWindow example involves populating tables thru a form that is derived
by the Dataphor frontend server. The fact that the form is derived by the
system services is a key concept in Dataphors ability to be a vehicle of
rapid application development. The example covers a wide range of Dataphor
capabilities that are used in application development. Concepts covered include
the presentation layer, metadata, forms, event handlers, references, constraints,
cursors and views as well as many other programming aspects of D4. 

What does it do
Based on user input several tables are populated and can be viewed thru
a form. Complete editing is available. The example uses Sql Server 2005
as the data respository (device). 

Requirements 
The application was developed using Dataphor v2677 and MS Sql Server 2005 sp1.
Except for an sql stored procedure (which is not required) the example should
run on any version of Sql Server or any other database for that matter.

Download the demo
Everything needed to run the demo including screenshots and setup scripts for
tables, views, constraints, references, operators etc. can be download @
http://www.geocities.com/d4tosql/D4/sqlwindowexample.zip

All scripts for Dataphor objects are documented with comments that should
help explain the what and why of the code. 

Screenshots
You can view screenshots of the demo here and here.
View a D4 operator used here.

What is the context of the example
Two different methods are used to compute cumulative aggregates that are based 
on the concept of an sql window as outlined in the Sql-99 standard. The sql window
is intended to simplify and make more efficient the computing of cumulative sums. 
While an sql window is available in DB2 and Oracle, it is not in Sql Server. 
The two methods used in the example simulate two different conceptual approaches
to deriving cumulative/running sums.

Who is this intended for
For anyone interested in exploring in greater detail key programming aspects
of Dataphor used in application development. There are many operators and
other Dataphor objects to go thru. The code is relatively straightforward
and easy to follow. The example can also be used as a demo for those not
familiar with Dataphor. The setup script will do everything required to run
the example. It is only necessary to have a library using Sql Server as the
device. The form in this example is 'completely' derived by the frontend server
using metadata supplied in various places. Even without a customized form
entering appropriate data is very easy and straightforward. If an entry is
invalid and violates a constraint the custom error message(s) fully describe
how to correct the entry. A modified form with menus will be available in the 
future.

Sql background for computing cumulative/running sums
Traditionally computing a running sum involved a subquery with an inequality
comparision. Given the sample table Stocks, the following sql query will
compute running sums for each Stock in ascending order of QTime. The query
can be run from Dataphor using the SQLQuery operator to pass it directly
to Sql Server. The use of the row_number() function allows the ranks to
be easily compared.

select SQLQuery
 ('
   select A.Stock,A.PRank,A.QTime,A.Quote,
    (select Sum(B.Quote) 
      from 
       (select Stock,QTime,Quote,
         row_number()over(Partition by Stock Order by QTime,Quote) as PRank
          from Stocks) as B
           where B.Stock=A.Stock and B.PRank<=A.PRank) as YSum 
   from
   (select Stock,QTime,Quote,
      row_number()over(Partition by Stock Order by QTime,Quote) as PRank
        from Stocks) as A    
 ');
Stock    PRank QTime                  Quote YSum 
-------- ----- ---------------------- ----- ---- 
IBM      1     4/3/2006 2:47:00 PM    107   107  
IBM      2     7/16/2006 7:39:00 AM   157   264  
IBM      3     8/26/2006 6:24:00 PM   125   389  
IBM      4     9/8/2006 1:11:00 PM    171   560  
IBM      5     11/22/2006 12:01:00 PM 155   715  
IBM      6     2/2/2007 12:03:00 AM   171   886  
IBM      7     4/13/2007 3:30:00 AM   219   1105 
IBM      8     10/18/2007 12:12:00 AM 133   1238 
IBM      9     12/25/2007 10:48:00 AM 139   1377 
MS       1     3/17/2006 7:12:00 PM   257   257  
.

This method is similar to computing ranks as discussed here and is also
just as inefficient. The impact of the subquery is more compelling when
it is expressed as a table. 
The following query gives the counts for the stocks:

select Stocks group by {Stock} add{Count() Cnt};

Stock    Cnt 
-------- --- 
IBM      9   
MS       11  
MySQL_AB 12  
Oracle   11  
Sybase   9   

For a table to represent the subquery it must be able to accommodate the
maximum number of rows in a Stock. Based on MySQL_AB, we need at least
12 rows. Consider the following D4 batch which populates a table assuming
the current row is the 6th row (PRank=6) in the 'IBM' Stock partition.
The table represents all rows up to and including the 6th row. Non-existent
rows, thru the use of IfNil, are given a PRank and nil (null) value for
Quote. A sum of the Quote values from the table would represent the
running sum up thru the 6th rank (PRank) for the 'IBM' Stock.

//Shows table based on table var S for a particular Stock and PRank.
var S:=SQLQuery('select Stock,QTime,Quote,
    row_number()over(Partition by Stock Order by QTime,Quote) as PRank
          from Stocks') {Stock,PRank,Quote} ;     
var Stock:='IBM';
var PRank:=6;          
select
      table
           {
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-11],row{-11 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-10],row{-10 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-9], row{ -9 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-8], row{ -8 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-7], row{ -7 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-6], row{ -6 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-5], row{ -5 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-4], row{ -4 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-3], row{ -3 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-2], row{ -2 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-1], row{ -1 PRank,nil Stock,nil Quote}),
           (S adorn{key{Stock,PRank}})[Stock,PRank]
           } order by {PRank desc};

Stock      PRank Quote      
---------- ----- ---------- 
IBM        6     171        
IBM        5     155        
IBM        4     171        
IBM        3     125        
IBM        2     157        
IBM        1     107        
<No Value> -6    <No Value> 
<No Value> -7    <No Value> 
<No Value> -8    <No Value> 
<No Value> -9    <No Value> 
<No Value> -10   <No Value> 
<No Value> -11   <No Value> 

The fact that the table has to be repopulated for each rank underscores the
inefficiency of the subquery. There is no reuse of information from one
rank to the next. When we compute the running sum for 'IBM':

var S:=SQLQuery('select Stock,QTime,Quote,
    row_number()over(Partition by Stock Order by QTime,Quote) as PRank
          from Stocks') {Stock,PRank,Quote} ;     
select
     S 
       add
          {
Sum
    (
     Quote
      from
     (
      table
           {
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-11],row{-11 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-10],row{-10 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-9], row{ -9 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-8], row{ -8 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-7], row{ -7 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-6], row{ -6 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-5], row{ -5 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-4], row{ -4 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-3], row{ -3 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-2], row{ -2 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-1], row{ -1 PRank,nil Stock,nil Quote}),
           (S adorn{key{Stock,PRank}})[Stock,PRank]
           }
     )
    )
    SumQte
          } ;       

Stock    PRank Quote SumQte 
-------- ----- ----- ------ 
IBM      1     107   107    
IBM      2     157   264    
IBM      3     125   389    
IBM      4     171   560    
IBM      5     155   715    
IBM      6     171   886    
IBM      7     219   1105   
IBM      8     133   1238   
IBM      9     139   1377   
MS       1     257   257    
.

We are making 45 inserts into the table since we are starting over from the
1st rank for each row of 'IBM':
1+2+3+4+5+6+7+8+9=45 inserts.

In the example operators D4ApplyforForm (used on the before insert event)
and D4ApplyforFormUpdate (used on the before update event) represent the
the idea of having to build a solution independently for each row. It therefore
represents an inefficient solution just as a subquery does.

Obviously it would make a lot more sense to simply add each new row to the
table which had all the previous rows and then get the sum. We could represent
the addition of a single row to the table as follows:

//Shows reusing information from one PRank to another.
var S:=SQLQuery('select Stock,QTime,Quote,
    row_number()over(Partition by Stock Order by QTime,Quote) as PRank
          from Stocks') {Stock,PRank,Quote} ;     
var Stock:='IBM';
var PRank:=6;          
var T1:=
      table
           {
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-11],row{-11 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-10],row{-10 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-9], row{ -9 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-8], row{ -8 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-7], row{ -7 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-6], row{ -6 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-5], row{ -5 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-4], row{ -4 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-3], row{ -3 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-2], row{ -2 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-1], row{ -1 PRank,nil Stock,nil Quote}),
           (S adorn{key{Stock,PRank}})[Stock,PRank]
           } ;
select
   T1
   union
   table
       {
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank+1],row{-99 PRank,nil Stock,nil Quote})
     } 
     where IsNotNil(Quote)
         order by {PRank desc};

Stock PRank Quote 
----- ----- ----- 
IBM   7     219   
IBM   6     171   
IBM   5     155   
IBM   4     171   
IBM   3     125   
IBM   2     157   
IBM   1     107   
 
In this way there would be as many inserts into the table as rows for the
Stock. In other words we would be scanning the data just once for the nine
rows vs. the multiple scans for the subquery (table).

We can show the idea of accumulating sums incrementally. For example the
running sum for row 6, adding the next row to the previously stored table
and then getting the sum for row 7:

var S:=SQLQuery('select Stock,QTime,Quote,
    row_number()over(Partition by Stock Order by QTime,Quote) as PRank
          from Stocks') {Stock,PRank,Quote} ;     
var Stock:='IBM';
var PRank:=6;          
var T1:=
      table
           {
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-11],row{-11 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-10],row{-10 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-9], row{ -9 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-8], row{ -8 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-7], row{ -7 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-6], row{ -6 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-5], row{ -5 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-4], row{ -4 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-3], row{ -3 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-2], row{ -2 PRank,nil Stock,nil Quote}),
     IfNil((S adorn{key{Stock,PRank}})[Stock,PRank-1], row{ -1 PRank,nil Stock,nil Quote}),
           (S adorn{key{Stock,PRank}})[Stock,PRank]
           } ;
select 
 table
     {
      row{'Accumulation of Sum for row 6' Desc,Sum(Quote from T1) SumQte},
      row{'Accumulation of Sum for row 7',
Sum(Quote
      from
         (
          T1
          union
          table
          {
         IfNil((S adorn{key{Stock,PRank}})[Stock,PRank+1],row{-99 PRank,nil Stock,nil Quote})
        } 
         where IsNotNil(Quote)
        )) }
     };     
        
Desc                          SumQte 
----------------------------- ------ 
Accumulation of Sum for row 6 886    
Accumulation of Sum for row 7 1105             
          
In the example operators D4WindowforForm (used on the before insert event)
and D4WindowforFormUpdate (used on the before update event) represent the
the idea of reusing prior information when appropriate by incrementally
adding (and when appropriate deleting) rows from a work table that is used
to obtain cumulative aggregates. View operator D4WindowforForm here.

The Sql window
The idea of accumulating running sums based on a new construct was introduced
in the sql-99 standard:

ISO/ANSI: Introduction to OLAP functions
http://tinyurl.com/2taahc

'An <OLAP function> is defined using a window. A window may specify a partitioning,
 an ordering of rows within partitions, and an aggregation group. The aggregation
 group specifies which rows of a partition, relative to the current row, should
 participate in the calculation of an aggregate. Through aggregation groups, windows
 support such important OLAP capabilities as cumulative sums and moving averages. 
 Windows may be specified either in the new WINDOW clause, or in-line in the SELECT list.'

The running sums for Stock is expressed by:

SELECT Stock,QTime,Quote,
   Sum(Quote) OVER (PARTITION BY Stock ORDER BY QTime,Quote 
                    'ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW') AS SumQte
   FROM Stocks;       

where the window is:

 Sum(Quote) OVER (PARTITION BY Stock ORDER BY QTime,Quote 
                 'ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW') AS SumQte

The 'ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW' is called the aggregate
group clause/statement. The definition of the terms is as follows:

window-aggregation-group-clause 
The aggregation group of a row R is a set of rows, defined relative to R in
the ordering of the rows of R's partition. This clause specifies the aggregation group. 

ROWS 
Indicates the aggregation group is defined by counting rows. 

group-BETWEEN 
Specifies the aggregation group start and end based on ROWS. 

UNBOUNDED PRECEDING 
Includes the entire partition preceding the current row. 

UNBOUNDED FOLLOWING 
Includes the entire partition following the current row. 

CURRENT ROW 
Specifies the start or end of the aggregation group as the current row. 

PRECEDING 
Specifies the number of rows preceding the current row.as a positive integer
indicating a number of rows. 

FOLLOWING 
Specifies the number of rows following the current row. as a positive integer
indicating a number of rows. 

The definition 'ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW' or its
abbreviated form 'ROWS UNBOUNDED PRECEDING' means include all the rows in
the partition prior to and including the current row in the computation of
the sum.

The sql-99 window does not introduce anything that couldn't be done previously.
What it does is simplify the way it is done and most importantly make accumulation
much more efficient.

We can express the idea of the sql window in Sql Server with the following
procedure:

create procedure SqlWindow
@From INT,
@To INT
as
SELECT A.PRank,A.Stock,A.QTime,A.Quote,
       WCnt,SumQte,MinQte,MaxQte,AvgQte
 FROM 
  (SELECT Stock,QTime,Quote,
   ROW_NUMBER() OVER(PARTITION BY Stock ORDER BY QTime) AS PRank
   FROM Stocks) AS A
    CROSS APPLY
     (
      SELECT COUNT(*) AS WCnt,SUM(Quote) AS SumQte,MIN(Quote) AS MinQte,
      MAX(Quote) AS MaxQte,CAST(AVG(1.*Quote) AS DECIMAL(6,1)) AS AvgQte
      FROM
       (SELECT Stock,Quote,ROW_NUMBER() OVER(ORDER BY QTime) AS PRank
        FROM Stocks AS B
        WHERE B.Stock=A.Stock) AS C
         WHERE C.PRank BETWEEN A.PRank+@From AND A.PRank+@To
       ) AS D

The @From and @To parameters are the bound of rows relative to the current
row that the running/cumulative sum is obtained for.

To obtain the running sums (and other aggregates) we can execute the procedure
using an aggregate group definition that includes all prior rows in the sum:

select 
SQLQuery
(
'Exec SqlWindow @From,@To',
 WList('ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW') {Wfrom From,Wto To}
);

PRank Stock    QTime                  Quote WCnt SumQte MinQte MaxQte AvgQte 
----- -------- ---------------------- ----- ---- ------ ------ ------ ------ 
1     IBM      4/3/2006 2:47:00 PM    107   1    107    107    107    107.0  
2     IBM      7/16/2006 7:39:00 AM   157   2    264    107    157    132.0  
3     IBM      8/26/2006 6:24:00 PM   125   3    389    107    157    129.7  
4     IBM      9/8/2006 1:11:00 PM    171   4    560    107    171    140.0  
5     IBM      11/22/2006 12:01:00 PM 155   5    715    107    171    143.0  
6     IBM      2/2/2007 12:03:00 AM   171   6    886    107    171    147.7  
7     IBM      4/13/2007 3:30:00 AM   219   7    1105   107    219    157.9  
8     IBM      10/18/2007 12:12:00 AM 133   8    1238   107    219    154.8  
9     IBM      12/25/2007 10:48:00 AM 139   9    1377   107    219    153.0  
1     MS       3/17/2006 7:12:00 PM   257   1    257    257    257    257.0  
.
.

The D4 operator WList parses the aggregate definition to obtain the @From and
@To values.

Executing the WList operator with the definition we obtain the bounds of
the cumulative sum. WList returns a row with the From and To values:

select WList('ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW');

Wfrom       Wto 
----------- --- 
-1000000000 0 

The Wfrom value of -1000000000 simply represents a large negative number
guaranteed to capture all prior rows. The value of 0 for Wto represents the
current row. The specify construct '{Wfrom From,Wto To}' simply renames the
columns of the row returned by operator WList to the names of the parameters
expected by the sql procedure SqlWindow.

If we wanted cumulative sums from the current row thru all rows available
for a Stock we can use either of the following:

select WList('ROWS UNBOUNDED FOLLOWING'); 
select WList('ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING');  

Wfrom Wto        
----- ---------- 
0     1000000000 

The Wfrom value of 0 is the current row and the Wto value of 1000000000
guarantees we will include all available 'following' rows in the partition
for the cumulative sum.

Executing the SqlWindow procedure:

select 
SQLQuery
(
'Exec SqlWindow @From,@To',
 WList('ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING') {Wfrom From,Wto To}
);

PRank Stock    QTime                  Quote WCnt SumQte MinQte MaxQte AvgQte 
----- -------- ---------------------- ----- ---- ------ ------ ------ ------ 
1     IBM      4/3/2006 2:47:00 PM    107   9    1377   107    219    153.0  
2     IBM      7/16/2006 7:39:00 AM   157   8    1270   125    219    158.8  
3     IBM      8/26/2006 6:24:00 PM   125   7    1113   125    219    159.0  
4     IBM      9/8/2006 1:11:00 PM    171   6    988    133    219    164.7  
5     IBM      11/22/2006 12:01:00 PM 155   5    817    133    219    163.4  
6     IBM      2/2/2007 12:03:00 AM   171   4    662    133    219    165.5  
7     IBM      4/13/2007 3:30:00 AM   219   3    491    133    219    163.7  
8     IBM      10/18/2007 12:12:00 AM 133   2    272    133    139    136.0  
9     IBM      12/25/2007 10:48:00 AM 139   1    139    139    139    139.0 <- Last row
1     MS       3/17/2006 7:12:00 PM   257   11   2254   100    290    204.9  
.
.
You can see that it is the opposite of 'ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW'.
For the last row in the partition we are left with only the current row being
used for the cumulative sum.

Suppose we want sums for the current row, 2 prior rows and 1 following row.
The aggregate group definition would be:

select WList('ROWS BETWEEN 2 PRECEDING AND 1 FOLLOWING');  

Wfrom Wto 
----- --- 
-2    1   

This definition therefore can use 4 rows, the 2 prior rows from the current row,
the current row and the next row.

select 
SQLQuery
(
'Exec SqlWindow @From,@To',
 WList('ROWS BETWEEN 2 PRECEDING AND 1 FOLLOWING') {Wfrom From,Wto To}
);

PRank Stock    QTime                  Quote WCnt SumQte MinQte MaxQte AvgQte 
----- -------- ---------------------- ----- ---- ------ ------ ------ ------ 
1     IBM      4/3/2006 2:47:00 PM    107   2    264    107    157    132.0 <- First row
2     IBM      7/16/2006 7:39:00 AM   157   3    389    107    157    129.7  
3     IBM      8/26/2006 6:24:00 PM   125   4    560    107    171    140.0  
4     IBM      9/8/2006 1:11:00 PM    171   4    608    125    171    152.0  
5     IBM      11/22/2006 12:01:00 PM 155   4    622    125    171    155.5  
6     IBM      2/2/2007 12:03:00 AM   171   4    716    155    219    179.0  
7     IBM      4/13/2007 3:30:00 AM   219   4    678    133    219    169.5  
8     IBM      10/18/2007 12:12:00 AM 133   4    662    133    219    165.5  
9     IBM      12/25/2007 10:48:00 AM 139   3    491    133    219    163.7  
1     MS       3/17/2006 7:12:00 PM   257   2    424    167    257    212.0  
.
.

Note that not all rows have 4 values. For the 1st row (PRank=1) there are no
prior rows, there is only the current row and the one following. Therefore the
aggregates for the 1st row are based on cumulative values from only 2 rows.

Note that a cumulative sum need not include the current row. We can base the
accumulation on prior or following rows. For example we can use the prior
rows between 4 and 2:

select WList('ROWS BETWEEN 4 PRECEDING AND 2 PRECEDING')       

Wfrom Wto 
----- --- 
-4    -2  

select 
SQLQuery
(
'Exec SqlWindow @From,@To',
 WList('ROWS BETWEEN 4 PRECEDING AND 2 PRECEDING') {Wfrom From,Wto To}
);          

PRank Stock    QTime                  Quote WCnt SumQte     MinQte     MaxQte     AvgQte     
----- -------- ---------------------- ----- ---- ---------- ---------- ---------- ---------- 
1     IBM      4/3/2006 2:47:00 PM    107   0    <No Value> <No Value> <No Value> <No Value> 
2     IBM      7/16/2006 7:39:00 AM   157   0    <No Value> <No Value> <No Value> <No Value> 
3     IBM      8/26/2006 6:24:00 PM   125   1    107        107        107        107.0      <-First value
4     IBM      9/8/2006 1:11:00 PM    171   2    264        107        157        132.0      
5     IBM      11/22/2006 12:01:00 PM 155   3    389        107        157        129.7      
6     IBM      2/2/2007 12:03:00 AM   171   3    453        125        171        151.0      
7     IBM      4/13/2007 3:30:00 AM   219   3    451        125        171        150.3      
8     IBM      10/18/2007 12:12:00 AM 133   3    497        155        171        165.7      
9     IBM      12/25/2007 10:48:00 AM 139   3    545        155        219        181.7      
1     MS       3/17/2006 7:12:00 PM   257   0    <No Value> <No Value> <No Value> <No Value> 
2     MS       5/9/2006 3:12:00 PM    167   0    <No Value> <No Value> <No Value> <No Value> 
3     MS       6/24/2006 1:51:00 AM   277   1    257        257        257        257.0      
.
.

It's not until the 3rd row of a partition that a row in included.

We can lead or lag by using the same value from both From and To. For example
to lead by 1 row we can use:

select 
SQLQuery
(
'Exec SqlWindow @From,@To',
 WList('ROWS BETWEEN 1 FOLLOWING AND 1 FOLLOWING') {Wfrom From,Wto To}
);          

PRank Stock    QTime                  Quote WCnt SumQte     MinQte     MaxQte     AvgQte     
----- -------- ---------------------- ----- ---- ---------- ---------- ---------- ---------- 
1     IBM      4/3/2006 2:47:00 PM    107   1    157        157        157        157.0      
2     IBM      7/16/2006 7:39:00 AM   157   1    125        125        125        125.0      
3     IBM      8/26/2006 6:24:00 PM   125   1    171        171        171        171.0      
4     IBM      9/8/2006 1:11:00 PM    171   1    155        155        155        155.0      
5     IBM      11/22/2006 12:01:00 PM 155   1    171        171        171        171.0      
6     IBM      2/2/2007 12:03:00 AM   171   1    219        219        219        219.0      
7     IBM      4/13/2007 3:30:00 AM   219   1    133        133        133        133.0      
8     IBM      10/18/2007 12:12:00 AM 133   1    139        139        139        139.0      
9     IBM      12/25/2007 10:48:00 AM 139   0    <No Value> <No Value> <No Value> <No Value> <-Last row
1     MS       3/17/2006 7:12:00 PM   257   1    167        167        167        167.0      
2     MS       5/9/2006 3:12:00 PM    167   1    277        277        277        277.0      
3     MS       6/24/2006 1:51:00 AM   277   1    250        250        250        250.0      
4     MS       8/18/2006 6:46:00 PM   250   1    290        290        290        290.0      
5     MS       1/16/2007 5:56:00 PM   290   1    177        177        177        177.0      
6     MS       1/22/2007 1:28:00 AM   177   1    150        150        150        150.0      
7     MS       5/30/2007 4:30:00 PM   150   1    263        263        263        263.0      
8     MS       6/21/2007 1:54:00 PM   263   1    153        153        153        153.0      
9     MS       7/17/2007 12:20:00 PM  153   1    170        170        170        170.0      
10    MS       11/26/2007 1:45:00 AM  170   1    100        100        100        100.0      
11    MS       12/25/2007 3:54:00 PM  100   0    <No Value> <No Value> <No Value> <No Value> <-Last row
1     MySQL_AB 2/8/2006 3:47:00 PM    21    1    25         25         25         25.0       
.
.

The last row for each partition (Stock) has no data in the accumulation. Note that
the lead precludes the use of previous information.

Modeling the Sql Window
The D4 operators that compute cumulative aggregates, (D4ApplyforForm, D4ApplyforFormUpdate,
D4WindowforForm and D4WindowforFormUpdate) take the sql procedure SqlWindow
as their logical model. The operators implement the logic of the procedure using
cursors as a way to illustrate their use. As stated previously the D4Apply operators
follow the sql procedure more faithfully and reflect the same inefficiency
as the procedure. The D4Window operators follow the same basic logic but make use
of previous rows. This interdependency reflects a much more efficient methodology
and hence is more faithful to the idea of the efficient sql window.

Dataphor - Create string intervals

2007-07-05T16:57:00.000-07:00

This problem appeared on the sqlservercentral.com forums:
'Compare two varchar arrays one character at time until different'
http://tinyurl.com/yochaa

The original poster explains:
Some background info first. We have an ASP.NET Web Application that serves up a list of
names in a dynamic tree control. The nodes of the tree are generated on demand and loaded
into the tree dynamically. When this list of names become very big (and indeed it will
be), the delay takes too long. So to minimize the delay we will group the names into
dynamically created folders. Each folder will contain n number of names based on some
passed in limit (In my code below this is the variable @NthRow). This way we can mimimize
the number of nodes to fetch yet still support a large number of names.

The final output of our function must be a table of strings in the form of X - Y where X
is the starting name and Y is the ending name of the group.

For example, given the following names and @NthRow = 5

Baker, Ms. Jamie
Espinosa, Ms. Jean
Gonzales, Mr. Robert
Holland, Ms. Julia
Jimenez, Ms. Soni
Macdonald, Mr. Mickey
Miller, Ms. Jana
Noriega, Ms. Michelle
Owen, Ms. Alma
Ramirez, Ms. Stephanie
Salas, Ms. Jeanine
Sipes, Mr. Tyler
Tamayo, Ms. Laura
Timmer, Ms. Julie
Trevino, Ms. Yvonne
Young, Ms. Annia

Produce an output table of:
Ba-Ji
Ma-Ra
Sa-Tr
Y-Y

Where:
Ba-Ji represents the names Baker, Ms. Jamie thru Jimenez, Ms. Soni
Ma-Ra represents the names Macdonald, Mr. Mickey thru Ramirez, Ms. Stephanie
Sa-Tr represents the names Salas, Ms. Jeanine thru Trevino, Ms. Yvonne
Y-Y represents the name Young, Ms. Annia
------------------------------------------------------

This solution uses the D4 language of Dataphor and MS Sql Server 2005 as
the data repository,

Basically we want to take a table of names ordered by last name and divide it
into intervals of NthRow names. We want at least the 1st 2 characters of
of the starting and ending last names for each interval. We will allow that
if the last interval contains only 1 name we want to show a lesser number
of characters than other intervals.

Here is some sample data entered in Dataphor and stored in sql server:

create table MyNames
{
ID:Integer,
Name:String tags{Storage.Length = "50"},
key{ID}
};
insert
table
{
row{1 ID,' Owen , Ms. Alma' Name},
row{2,'Ramirez, Ms. Stephanie'},
row{3,' Salas, Ms.  Jeanine'},
row{4,'  Sipes, Mr. Tyler'},
row{5,'Tamayo, Ms. Laura'},
row{6,'Timmer  , Ms. Julie'},
row{7,'  Trevino, Ms. Yvonne'},
row{8,'  Young, Ms. Annia'},
row{9,'Baker, Ms. Jamie'},
row{10,' Espinosa, Ms. Jean'},
row{11,'  Gonzales, Mr. Robert'},
row{12,'  Holland, Ms. Julia'},
row{13,'Jimenez, Ms. Soni'},
row{14,' Macdonald, Mr. Mickey'},
row{15,'   Miller, Ms. Jana'},
row{16,'Noriega, Ms. Michelle'}
}
into MyNames;    

Operator ArrayVarNames takes parameters of a 'table' with columns ID and Name,
NthRow and NChars which is the number of characters +1 to identity a name.
The operators returns a table with the interval information, the count of
names in the interval and the identifying characters.

create operator
    ArrayVarNames(MyNames:table{ID:Integer,Name:String},NthRow:Integer,NChars:Integer):
                  table{Index:Integer,NFrom:Integer,NTo:Integer,NCnt:Integer,
                        CharFrom:String,CharTo:String}
begin
result:=table of typeof(result){};
//Return a table variable with a sequence number (RowID) reflecting Lastname
//sorted in ascending order.
var T:=
     ToTable(ToList(cursor(
       (MyNames add{Name.Split()[0].Trim() LastName}
          with {IgnoreUnsupported = 'true'}) order by {LastName})))
            {sequence+1 RowID,Name,LastName};
var MaxID:=Max(RowID from T); //Max RowID in table T.           
//NGroups is a table variable that uses NthRow and the numbers table to form
//intervals. For each interval (Index) we get the count (Cnt), the max RowID
//(MaxRowID) for the interval (we really are only interested in MaxRowID
//for the last interval).
var NGroups:=
            (
               numbers
                where num between 1 and MaxID
                 {num Index,((num-1)*NthRow)+1 From,num*NthRow To}
               )
                  times
                    T with {IgnoreUnsupported = 'true'}
                      where RowID between From and To
                       group by {Index} add{Count() Cnt,Max(RowID) MaxRowID}
                          {Index,Cnt,MaxRowID};
//LRow is a row variable with the max Index (MaxIndex), which is the max interval
//given data MyNames and NthRow, the count (Cnt) of the rows for the max Index
//and the max RowID for MyNames. Note that LRow uses the table (variable) NGroups.                        
var LRow:=    
           ( NGroups adorn{key{Index}} return 1 by {Index desc}
                {Index MaxIndex,MaxRowID,Cnt})[];
//We get NChars+1 characters of the Lastname for the starting (NFrom) RowID
//and ending RowID (NTo) for each interval by using RowID as a key to look
//up LastName in table T. We use some boolean logic to check if there is
//only 1 LastName in the last interval (MaxIndex). If true we only want
//the NChars character of LastName. We put the boolean logic directly in SubString.
//We also add the count (NCnt) for each interval.                       
result:=
numbers rename{num Index}
where Index between 1 and LRow.MaxIndex
 add{(NGroups adorn{key{Index}})[Index].Cnt NCnt} with {IgnoreUnsupported = 'true'}
 add{((Index-1)*NthRow)+1 NFrom,Index*NthRow NTo}
  add
     {
      SubString((T adorn{key{RowID}})[NFrom].LastName,0,
      (NChars+ToInteger(not((Index=LRow.MaxIndex) and (LRow.Cnt=1))))) CharFrom,
   IfNil(
        SubString((T adorn{key{RowID}})[NTo].LastName,0,
      (NChars+ToInteger(not((Index=LRow.MaxIndex) and (LRow.Cnt=1))))),
      SubString((T adorn{key{RowID}})[LRow.MaxRowID].LastName,0,
      (NChars+ToInteger(not((Index=LRow.MaxIndex) and (LRow.Cnt=1)))))
        ) CharTo
     }
     {Index,NFrom,NTo,NCnt,CharFrom,CharTo}
     with {IgnoreUnsupported = 'true'} ;
end;    

For example given the original data:

select ArrayVarNames(MyNames,5,1);

Index NFrom NTo NCnt CharFrom CharTo
----- ----- --- ---- -------- ------
1     1     5   5    Ba       Ji   
2     6     10  5    Ma       Ra   
3     11    15  5    Sa       Tr   
4     16    20  1    Y        Y  

select ArrayVarNames(MyNames,6,2);

Index NFrom NTo NCnt CharFrom CharTo
----- ----- --- ---- -------- ------
1     1     6   6    Bak      Mac  
2     7     12  6    Mil      Sip  
3     13    18  4    Tam      You  

We can use the operator with any table by simply renaming the columns of
the table. Here we use the Orders table of the Sql Server Northwind database
to create intervals for customers by simply renaming the columns of the
Orders table to agree with those expected by ArrayVarNames:

select ArrayVarNames
 (Orders group by {CustomerID} add{Max(OrderID) ID}
            {ID,CustomerID Name},15,4);


Index NFrom NTo NCnt CharFrom CharTo
----- ----- --- ---- -------- ------
1     1     15  15   ALFKI    COMMI
2     16    30  15   CONSH    GOURL
3     31    45  15   GREAL    LILAS
4     46    60  15   LINOD    QUEEN
5     61    75  15   QUICK    THEBI
6     76    90  14   THECR    WOLZA

We can create a similar table using the Products table of the Northwind db
using ProductID and ProductName.

select ArrayVarNames
 (Products {ProductID ID,ProductName Name},10,30);
 
Index NFrom NTo NCnt CharFrom                      CharTo                          
----- ----- --- ---- ----------------------------- ------------------------------- 
1     1     10  10   Alice Mutton                  Chef Anton's Gumbo Mix          
2     11    20  10   Chocolade                     Grandma's Boysenberry Spread    
3     21    30  10   Gravad lax                    Jack's New England Clam Chowder 
4     31    40  10   Konbu                         Mishi Kobe Niku                 
5     41    50  10   Mozzarella di Giovanni        Queso Cabrales                  
6     51    60  10   Queso Manchego La Pastora     Scottish Longbreads             
7     61    70  10   Singaporean Hokkien Fried Mee Tofu                            
8     71    80  7    Tourtière                     Zaanse koeken                   
 
Hopefully examples like these will encourage you to explore Dataphor -:)