Using QoQ to intentionally make Cartesian products

For those that don’t remember their database theory classes, a Cartesian product is what you get when you take every element of one matrix (query) and match it up with every element of another matrix (query) so that you end up with every combination of every element. Like so:

1 3
1 4
2 3
2 4

When doing database work we generally try very hard to avoid making unrestricted Cartesian products. Most of the time when we do make them, it is an accident, probably because we left off a key clause in a JOIN expression. On the occasion that we do want to do them intentionally, we can do them easily with a CROSS JOIN expression — or JOIN ON (1 = 1) if your DBMS doesn’t support that.

But! Cross products can be useful things, when used correctly.

Today’s blog post by Ben Nadel is about iterating over an unknown number of nested lists. Long story short for the attention-deficit crowd, here’s the skinny:

Given an array of lists, where neither the length of the array nor the length of the lists are fixed, how can we quickly build a set of every combination of the elements on that list?

Note that last part: “every combination of the elements”. Sound familiar?

I’m going to use the same array that Ben used:

<cfset arrLists = [
    "Full Sleeve,Short Sleeve"
  ] />

Here’s the code:

<cfloop from="1" to="#arrayLen(arrLists)#" index="i">
    <cfset p=queryNew("")>
    <cfset queryAddColumn(p, "a#i#", "VARCHAR", listToArray(arrLists[i]))>
    <cfif NOT isDefined("q")>
        <cfset q=p>
        <cfquery dbtype="query" name="q">
        SELECT * FROM q, p

If we then cfdump that query “q”, we get this:

  A1 A2 A3 A4
1 Blue Small Mens Full Sleeve
2 Blue Small Mens Short Sleeve
3 Blue Small Womens Full Sleeve
4 Blue Small Womens Short Sleeve
21 Black Large Mens Full Sleeve
22 Black Large Mens Short Sleeve
23 Black Large Womens Full Sleeve
24 Black Large Womens Short Sleeve

So what did we do? In English:

  1. For each of the lists in the array,

    1. Create a new query “p” with a no columns.

    2. Add a column to query “p”. That column is named “A1” or “A2” or whatever the number of the current list is.

    3. Set its single column equal to the items. (Such as “Blue”.) That is, we’ve taken the current list and turned it into a query resultset.

    4. If we haven’t yet created our result query “q”, and are thus on our first iteration of the loop, then just set it equal to query “p”.

    5. But if we have query “q” already, and are thus at least on our second trip through the loop, then do some magic:

      1. Make a Cartesian product of our existing query “q” and the current list query “p”. We do this by joining the two without any restrictive clauses.

      2. Assign that Cartesian product back to our result query “q” via name="q".

  2. Repeat, making the Cartesian product bigger and bigger, until you run out of lists.

Or, visually, here’s how our result query “q” looks at the end of the first two iterations through the array loop:

1 Blue
2 Black
  A1 A2
1 Blue Small
2 Blue Medium
3 Blue Large
4 Black Small
5 Black Medium
6 Black Large

And so on and so forth.

Now … let’s make things a little trickier …

What if we wanted to add an “id” column to our resultset that was an automatically-generated unique identifier for that combination? In real-world terms, we want to build a SKU for that particular item configuration. And, just to make it even more realistic, lets say that each item in each list has a particular key, like so:

<cfset arrLists = [
    "l:Full Sleeve,s:Short Sleeve"
  ] />

We’ve added a character to identify each option uniquely. There’s no real reason for limiting it to a single character, nor limiting it to alpha or numeric. I’ve just done so for simplicity. The modifications to make a unique SKU are then:

<cfset columns="">
<cfloop from="1" to="#arrayLen(arrLists)#" index="i">
    <cfset columns=listAppend(columns, "a#i#")>
    <cfset p=queryNew("pid,a#i#","VARCHAR,VARCHAR")>
    <cfloop list="#arrLists[i]#" index="j">
        <cfset queryAddRow(p)>
        <cfset querySetCell(p, "pid", listFirst(j,":"), p.recordCount)>
        <cfset querySetCell(p, "a#i#", listRest(j,":"), p.recordCount)>
    <cfif NOT isDefined("q")>
        <cfquery dbtype="query" name="q">
        SELECT pid AS id, #columns#
        FROM p
        <cfquery dbtype="query" name="q">
        SELECT id + pid AS id, #columns#
        FROM q, p
    <cfdump var="#q#">

Most of the really weird stuff is actually just working around bugs in the QoQ engine. I’ll explain the differences.

  • The columns variable is going to hold those A1, A2, etc, column names for us. We got away without needing to keep an explicit list before, but keeping one now will make things much simpler. And there’s not much to it really: just append the latest column name to the list at each iteration.

  • We’ve added a “pid” column to our query. No surprise there.

  • You’ll notice that we now need to specify in the queryNew call that we are working with VARCHAR columns. This is needed if any of your identifying character strings are numeric, such as we have done with sizes. The QoQ engine will throw an error later if we don’t do this, as it “helpfully” converts those numbers to BIGINT columns, then can’t figure out how to add a VARCHAR and BIGINT together later.

  • In the loop where we set up the list query, we now have to set both the ID and value columns.

  • Our nice, simple, cfset q=p had to go away, unfortunately. To make the query work later, the ID column in the two queries could not be named the same thing. So, when we just copy over the query without the join, we need to change the column name as we do it.

    We know that A1 is our only column at the time, but I used #columns# instead so that if we decide to use a different naming scheme this is one less place we’ll have to change it later.

  • The Cartesian product query has only gotten a little more complex. We concatenate the old ID and the new ID, then bring over all of the data columns.

Again, with the visuals:

  A1 ID
1 Blue u
2 Black k
  A1 A2 ID
1 Blue Small u1
2 Blue Medium u2
3 Blue Large u3
4 Black Small k1
5 Black Medium k2
6 Black Large k3
  A1 A2 A3 ID
1 Blue Small Mens u1m
2 Blue Small Womens u1f
3 Blue Medium Mens u2m
4 Blue Medium Womens u2f
5 Blue Large Mens u3m
6 Blue Large Womens u3f
7 Black Small Mens k1m
8 Black Small Womens k1f
9 Black Medium Mens k2m
10 Black Medium Womens k2f
11 Black Large Mens k3m
12 Black Large Womens k3f

It’s a little contrived, but there you go.

Categorized as Web

By Rick Osborne

I am a web geek who has been doing this sort of thing entirely too long. I rant, I muse, I whine. That is, I am not at all atypical for my breed.