Jacob Mastel

Welcome to the new site

2026-07-28T09:00:00-07:00

Welcome to the rebuilt mastel.org. If you're reading this, the new static setup is working.

Why the change

The old site ran on a full CMS. That was useful when I was rolling my own comments, forms, and galleries — but over time I moved each of those into a dedicated, self-hosted app. What was left was essentially a blog and a photo gallery being served by a lot of machinery I no longer needed.

So I collapsed it down to what it actually is: Markdown files and a template, built into static HTML.

What's under the hood

Pelican turns Markdown into HTML.
A small custom theme, typhoon-dark, keeps the look I liked from Grav's Typhoon theme — but darker, with a royal-purple accent.
The whole thing is served as static files from an Nginx container.

# The entire publish workflow
pelican content -s publishconf.py
docker build -t mastel-web .

Visually, the whole pipeline is just:

flowchart LR MD["Markdown + images"] --> P["Pelican build"] P --> H["Static HTML"] H --> N["Nginx container"] N --> U["Visitor"]

No database, no PHP, no plugin update treadmill. Just files.

What's next

More writing, and a real photography gallery to replace the placeholder frames. Onward.

Aggregating Using Unpivot

2017-08-18T00:00:00-07:00

The most common solution I see to this particular problem is a series of case statements. This can be a huge pain to write and even worse to maintain. Recently, I've learned that Oracle's unpivot functionality can be a better solution.

The Problem

Recently, I was asked to report on course performance based on student demographics. I have a table that includes many indicators (values are mostly Y or N) and looks something like this.

 demographics
     person_uid
     oregon_resident_ind
     us_minority_ind
     first_generation_ind

I need to use this table to build a report has a format that looks something like what's below.

 course id | course avg grade | course dwfu rate
     group | avg grade | dwfu rate | Δ course avg grade | Δ course dwfu rate

Getting the Course Info

This part is pretty easy; a simple group by expression will give me the three values I need to calculate the desired course wide values in report.

 select
     course_id
     , count(*)                  as course_cnt
     , avg(final_grade)          as course_avg_grade
     , sum(is_dwfu(final_grade)) as course_dwfu_cnt
 from  ...
 where ...
 group by
    course_id

The difficulty now lies in dealing with all the demographic information.

Getting the Demographic Info

Solution 1 - Lots of Case Statements

As I said before, the most common solution I see for this is using a series of case statements. If you've done this before, you can probably you guess what this particular nightmare is going to look like.

 sum(
    case when us_minority_ind = 'Y'
        then 1
         else 0
     end
 ) as us_minority_cnt
 , avg(
    case when us_minority_ind = 'Y'
        then final_grade
         else null
     end
 ) as us_minority_avg
 , sum(
    case when us_minority_ind = 'Y' and is_dwfu(final_grade) = 1
        then 1
         else 0
     end
 ) as us_minority_dwfu_cnt

For this one demographic, that's 18 lines of code! The example I listed is 3 demographics. Basic math tells us that's 54 lines of code just for this section of my example. In reality, I currently have 11 different demographic indicators to deal with, and there's a good chance that there will be more in the future. Show me a programmer that wants to deal with 198 lines of this nonsense and I'll show you a crazy person.

Want to or not, sometimes the fact that it works is enough. This does provide the various fields needed to calculate the desired reports, which is why I've used this solution in the past.

Unfortunately, this solution gives me one massively flat line. SSRS doesn't like that. Since SSRS does not include an unpivot feature and I'm loath to start hard coding textboxes, I've traditionally unpivoted these large lines. As I started to implement that here, it gave me the idea for solution 2.

Solution 2 - Calculations in the Unpivot

An unpivot query has two primary parts, the source query and the actual unpivot step. For my source query, I want to maintain the overall course information, and bring in the indicators. To do this, I used Oracle's window clause. This allowed me to define grouping, while still getting results row by row.

The Source Query

 select
     course_identification
     , count(*) over (
         partition by
             course_identification
     ) as course_cnt
     , sum(final_grade) over (
             partition by
                 course_identification
         ) / count(*) over (
             partition by
                 course_identification
         )
     ) as course_avg_grade
     , sum(
         case when is_dwfu(final_grade)
             then 1
             else 0
         end
     ) over (
         partition by
             course_identification
     ) as course_cnt_dwfu
     , oregon_residency_ind
     , us_minority_ind
     , first_gen_ind
     , final_grade
 from
  ...

Notice I'm just including the demographic fields that I care about? I haven't included the 198 lines of hell that I mentioned earlier. Even if I were to include all of my 11 demographics, the select clause would still only be around 35 lines long. So onto the unpivot!

The Unpivot Clause

The unpivot clause is the portion of a query where you define the actual unpivoting action you want to perform. Within this clause, you need to list the fields as they'll be available in the select clause. I've decided to name my fields ind (since it's identifying if they're in the group or not), grade (an alias for final_grade) and pop (as in population). The lines beneath that list what fields will be unpivoted. The first set puts oregon_residency_ind into ind, final_grade into grade and Resident into pop.

 ) unpivot (
     (ind, grade) for pop in (
         (oregon_residency_ind, final_grade) as 'Resident'
         , (us_minority_ind, final_grade) as 'US Minority'
         , (first_gen_ind, final_grade) as 'First Generation'
     )
 )

Once that's done I'll have a bunch of rows. In fact, you'll see one row per person for each demographic that was originally selected. Since I've selected 3 demographics, I'll have three rows per person. Our last step will be to aggregate a second time to get our final result.

Putting it all Together

Finally, we can put it all together. We'll need to use our same 3 case statements from earlier, but now we only need to do this once! We can get away with this since each row is now only relevant to a single population, a normal group by clause will wrap everything up.

 select
     course_identification
     , course_cnt
     , course_avg_grade
     , course_cnt_dwfu
     , pop
     , sum(
         case when ind = 'Y'
             then 1
             else 0
         end
     ) as pop_cnt
     , sum(
         case when ind = 'Y'
                 then bicreprt.f_letter_to_points_osu(grade)
             else 0
         end
     ) / sum(case when ind = 'Y' then 1 else 0 end) as pop_avg_grade
     , sum(
         case when ind = 'Y' and is_dwfu(grade)
             then 1
             else 0
         end
     ) as pop_cnt_dwfu
 from (
     {{ source query goes here }}
 ) unpivot (
     (ind, grade) for pop in (
         (oregon_residency_ind, final_grade) as 'Resident'
         , (us_minority_ind, final_grade) as 'US Minority'
         , (first_gen_ind, final_grade) as 'First Generation'
     )
 )
 group by
     course_identification
     , course_cnt
     , course_avg_grade
     , course_cnt_dwfu
     , pop

In Conclusion

The concept of pivoting and unpivoting can be a little hard to get your head around. A lot of SQL is that way for most people. This trick however, does a good job of eliminating a huge amount of code redundancy (probably some error causing typos...), and in my testing actually performed significantly better than having 33 different case statements.

When Join Order Matters

2017-07-28T09:00:00-07:00

It's a popular belief that the order of a SQL query's join order doesn't matter so long as the joins are done as an inner join 1. In a simple world, this is true. Unfortunately, life is rarely so simple.

A common question among new SQL query users is "Does the order of my inner joins matter?". The answer to that question, like many things in life, tends to be more complicated than one might originally think.

Logical Equivalence

First we should look at the mathematical point of view. The symmetric property tells us that if $a = b$, then $b = a$. In plain English, that means the results of our query won't change, regardless of how we input our join order. From a logical standpoint, the order of the join won't effect the results. Simple, right?

The Optimizer

The question gets more complicated in how the query plays out in the RDBMS's optimizer. Most of my work is done with Oracle's RDBMS, so I will focus on their implementation and documentation. Regardless, the information should apply to MySQL, Microsoft's SQL Server, PostgreSQL and any other RDBMS you can think of.

The optimizer, in case you're unaware, is the part of the RDBMS that is responsible for interpreting your query and building an execution plan. Most of the optimizer's work is done through estimations. As a result, the plan given is what the RDBMS believes to be the best plan available. What the RDBMS believes and what is true however, aren't necessarily the same.

The plan generator explores various plans for a query block by trying out different access paths, join methods, and join orders. The number of possible plans for a query block is proportional to the number of join items in the FROM clause. This number rises exponentially with the number of join items.

Oracle Database Performance Tuning Guide 11.4.1.4 (Emphasis mine)

The second bolded section is the most important aspect in relation to this article. A short query joining two tables isn't likely to give the optimizer much trouble. Longer queries with more complex relationships, like joining complex views, can create more possible join orders than the optimizer has time to examine. If this happens, you might get stuck with a sub optimial execution plan.

Fortunately, the optimizer is good at taking hints from the query writer. It turns out that the order in which you write your joins acts as a kind of hint in of itself. If you specify a join order in a particular way, the optimizer will try that order first.

An Example

I wrote a query relatively recently where the performance was completely unusable. My mistake was that I placed one of my database's largest views, Address Current under the driving table on my query's join tree. Worse yet, the next table up the join tree was Person Detail. That's also one of the database's largest views.

--- title: Bad Join --- flowchart LR CTE --> PSRS["Person Sensitive Race Slot"] CTE --> PS["PreStudent"] --> PD["Person Detail"] --> AC["Address Current"] CTE --> SD["Student Demographics"]

--- title: Good Join --- flowchart LR PD["Person Detail"] --> PSRS["Person Sensitive Race Slot"] PD --> PS["PreStudent"] PD --> AC["Address Current"] PS --> SD["Student Demographics"] SD --> CTE

As the image demonstrates, the Good Join contains all the same views as the Bad Join. The key difference is that my driving table, the CTE, is at the bottom of the good join. In the Bad Join the database pulls and joins all the records in Person Detail and Address Current and only starts to filter them later when the CTE is applied.

How Much Does it Matter?

Well, the Bad Join takes so long to run that it might as well not. The Good Join on the other hand, returns fairly quickly. Looking at the explain plan, the difference is pretty stark. The Bad Join has a cost of 13,500, but the Good Join has a cost of 9,727. That's an estimated 37% increase just based on the order I typed in the joins!

Finally, looking the autotrace we see that the performance of the two queries diverge even further. The cost returned from the autotrace lists the Bad Join at 28,429 and the query took 1701.556 seconds to return. The Good Join however returned in just 1.782 seconds with a cost of 18,178%. That's a 67% increase in cost and a 95,385% increase in execution time!

Lessions Learned Logically, your join order may not matter, but if you want your query to return in a reasonable amount of time, you need to pay attention to how you're building your query. Make sure that your driving tables are at the bottom of your join tree, and focus on building the join tree taller as opposed to wider.