Alternative Ranking Modifications: An Analysis

The following article was a contribution by Groger Ranks member Arjun Nageswaran.

Before beginning this post, it is worth noting that all the potential rankings in this article will exclude Saratoga A from California. Two players from the Saratoga club were recently (and rightfully) removed for misconduct, and it was an error on our part to not remove them from the rankings before posting the pre-HSNCT rankings. We obviously have zero-tolerance for any misconduct in quizbowl and all future rankings (starting with the ones discussed in this article) will discount any performances from the Saratoga A team with those two players. A special thanks to Nicole Pruitt for catching this error in our pre-HSNCT rankings.

In this article, I wanted to go through three alternative ranking modifications and see how that would change the order of teams. This article is more for public interest rather than anything else, but I hope you guys find some things interesting here! These rankings will take into account all teams rather than just teams going to HSNCT and/or PACE NSC. Additionally, there will be no mACF-only rankings in this article since we plan to release that separately as part of our pre-PACE NSC rankings.

We’ll start with the first one:

NAQT-Only Rankings

RANKTEAMSCOREBEST SET
1Stanton College Prep B (FL)111.23IS-197
2Hunter A (NY)107.38IS-201
3Lambert A (GA)105.64IS-199
4Arcadia A (CA)105.24IS-197
5Richard Montgomery A (MD)104.23IS-199
6Beavercreek A (OH)103.13IS-195
7St. Mark’s School of Texas A (TX)102.61IS-199
8Phillips Academy A (MA)101.72IS-199
9Stanford Online A (CA)101.50IS-197
10Ladue A (MO)101.24IS-195
11Wayzata A (MN)100.94IS-197
12Detroit Country Day A (MI)100.52IS-201
13East Brunswick B (NJ)100.49IS-199
14Solon A (OH)98.85IS-195
15East Brunswick A (NJ)98.73IS-199
16Greenhill A (TX)98.45IS-199
17TAG Magnet A (TX)98.36IS-195
18East Chapel Hill A (NC)97.28IS-199
19Burnsville A (MN)96.93IS-201
20Georgetown Day A (DC)95.47IS-195
21Russell A (KY)95.25IS-199
22Raffles A (SG)94.96IS-195
23Chattahoochee A (GA)94.10IS-195
24Mira Loma A (CA)94.09IS-197
25High Tech A (NJ)94.03IS-195
26Hunter B (NY)93.99IS-197
27Hamilton A (AZ)93.81IS-199
28Great Valley A (PA)93.12IS-197
29Strake Jesuit A (TX)91.62IS-195
30Santa Monica A (CA)90.79IS-197
31Carmel A (IN)90.71IS-201
32Centennial A (MD)90.59IS-197
33duPont Manual A (KY)90.49IS-195
34Miami Valley A (OH)89.99IS-199
35Hotchkiss A (CT)89.80IS-201
36Mounds View A (MN)89.67IS-197
37Belmont A (MA)89.47IS-197
38Ed W. Clark A (NV)89.26IS-201
39Detroit Catholic Central A (MI)88.88IS-201
40Detroit Country Day B (MI)88.84IS-199
41Homestead A (CA)88.69IS-195
42Livingston A (NJ)88.31IS-197
43Del Norte A (CA)87.99IS-197
44Choate A (CT)87.99IS-201
45Manheim Township A (PA)87.68IS-199
46Irvington A (CA)87.43IS-195
47Grady A (GA)87.02IS-199
48BASIS Silicon Valley A (CA)86.84IS-195
49North Carolina Science and Math A (NC)86.20IS-199
50Woodford County A (KY)86.16IS-195
51Wayzata C (MN)86.07IS-197
52Seven Lakes A (TX)85.47IS-199
53John P. Stevens A (NJ)84.75IS-201
54Eden Prairie A (MN)84.58IS-201
55Westminster A (GA)84.52IS-201
56St. Joseph A (IN)84.25IS-201
57Wayzata B (MN)84.10IS-197
58Solon B (OH)84.01IS-195
59Lowell A (CA)83.99IS-197
60Maggie Walker Governor’s School B (VA)83.80IS-195
61Raymore-Peculiar A (MO)83.69IS-199
62Paideia A (GA)83.39IS-201
63Caddo Magnet A (LA)82.79IS-195
64Maggie Walker Governor’s School A (VA)81.98IS-195
65Cinco Ranch A (TX)81.60IS-201
66Northmont A (OH)81.15IS-199
67Johns Creek A (GA)81.08IS-201
68Lynbrook A (CA)80.87IS-199
69Westview A (OR)80.50IS-199
70Buchholz A (FL)80.44IS-201
71Freeman A (VA)80.42IS-195
72Carmel B (IN)80.22IS-195
73William Henry Harrison A (IN)80.05IS-201
74St. Joseph Central A (MO)79.93IS-199
75Archbishop Mitty B (CA)79.51IS-199
76Monta Vista A (CA)79.51IS-197
77Ithaca A (NY)79.47IS-199
78Baton Rouge Episcopal A (LA)79.36IS-199
79Bellarmine College Prep A (CA)79.34IS-197
80Tanglin Trust B (SG)79.30IS-195
81Chattahoochee B (GA)78.82IS-195
82LASA A (TX)78.64IS-195
83Winston Churchill A (MD)78.60IS-199
84BASIS Scottsdale A (AZ)78.14IS-199
85Northfield A (MN)77.97IS-197
86Acton-Boxborough A (MA)77.25IS-197
87Jasper A (TX)77.23IS-195
88Folsom A (CA)77.23IS-195
89Washington A (CA)77.12IS-197
90Prior Lake A (MN)76.77IS-197
91Coppell A (TX)76.74IS-197
92Dunbar A (KY)76.55IS-201
93Darien A (CT)76.41IS-197
94Archbishop Mitty A (CA)76.25IS-195
95Wheeler A (GA)76.09IS-201
96Bergen Tech-Teterboro A (NJ)76.04IS-195
97Troy A (MI)75.90IS-201
98James E. Taylor A (TX)75.84IS-201
99BASIS McLean A (VA)75.74IS-195
100Mission San Jose A (CA)75.65IS-199

These rankings imagine what the top 100 would look like if we only took into account performances on NAQT sets. Our pre-HSNCT rankings would have ordinarily been NAQT only rankings only including teams attending HSNCT. Why couldn’t we do that this year? If you look at the discrepancy between these NAQT-only rankings and our pre-HSNCT rankings, you’ll notice that we miss NAQT data from a lot of the teams attending HSNCT including the #1 in our pre-HSNCT rankings, TJ A, and essentially all the Illinois teams. As a result, it would probably be misleading to label such a ranking as “pre-HSNCT rankings.” Of course by including both mACF and NAQT sets this makes our current pre-HSNCT rankings not the best measure possible, but that was probably the better of the two choices. This had no bearing on the decision to not have NAQT-only pre-HSNCT rankings, but additionally since there is no SCT set this year, this would have meant that all of the sets for an NAQT-only ranking can only include IS sets and SSNCT. These sets are around HS regular difficulty, and as a result there would be little information about teams’ ability to scale up to harder difficulties, less than say a performance on DII SCT would have in past years.

For me, the most interesting difference between the pre-HSNCT and the NAQT-only rankings (besides several teams being omitted from NAQT-only rankings) is the ordering of the top 4. With TJ A having no NAQT performances, it would be expected that Arcadia A as the #2 ranked team would be the new #1 for NAQT-only rankings. However, Arcadia got bumped down to #4 with the top 3 being 1. SCP B (ranked #5 in the pre-HSNCT rankings), 2. Hunter A (ranked #8 in the pre-HSNCT rankings), and 3. Lambert A (ranked #18 in the pre-HSNCT rankings). Lambert probably has the biggest jump when only NAQT sets are taken into account.

What does this mean? There are certainly teams who are much stronger on NAQT than mACF and vice versa, and as a result being able to look at the NAQT-only rankings could provide clarification into potential HSNCT predictions in a way that overall rankings can not. Of course like stated before there are several teams being omitted from the NAQT-only rankings, and the sets are all much easier than what HSNCT will end up being, but it’s still pretty interesting to look at.

This discussion of considering teams’ abilities to scale takes us to our second alternative ranking modification we will be looking at:

“Hard Sets”-Only Rankings

RANKTEAMSCOREBEST SET
1Thomas Jefferson Science & Tech A (VA)109.45BHSAT
2Arcadia A (CA)101.50STASH
3Richard Montgomery A (MD)98.05Prison Bowl
4Wayzata A (MN)97.02STASH
5Georgetown Day A (DC)96.69LIT
6Montgomery Blair A (MD)95.20BHSAT
7Stanton College Prep B (FL)94.32Prison Bowl
8University Lab A (IL)93.74LIT
9Beavercreek A (OH)92.89SATURNALIA
10Belmont A (MA)92.01SATURNALIA
11Hunter A (NY)91.74WORKSHOP
12Westview A (CA)91.52IKEA
13Lambert A (GA)90.89WORKSHOP
14St. Mark’s School of Texas A (TX)90.52STASH
15Adlai E. Stevenson A (IL)90.50Prison Bowl
16Detroit Country Day A (MI)89.69STASH
17Greenhill A (TX)89.05WORKSHOP
18Stanford Online A (CA)88.96STASH
19Solon A (OH)88.90BHSAT
20Kinkaid A (TX)88.68Prison Bowl
21Barrington A (IL)88.24BHSAT
22Ladue A (MO)87.56BHSAT
23Chattahoochee A (GA)86.68SATURNALIA
24TAG Magnet A (TX)86.52Prison Bowl
25Buffalo Grove A (IL)85.89STASH
26East Chapel Hill A (NC)85.34LIT
27High Tech A (NJ)84.36BHSAT
28Mira Loma A (CA)83.88SATURNALIA
29Santa Monica A (CA)82.20CATT
30Strake Jesuit A (TX)80.75LIT
31Detroit Country Day B (MI)80.63STASH
32Great Valley A (PA)79.05SATURNALIA
33Phillips Academy A (MA)78.23STASH
34Cinco Ranch A (TX)77.85STASH
35Mounds View A (MN)77.46LIT
36North Carolina Science and Math A (NC)77.04SATURNALIA
37Hunter B (NY)76.90WORKSHOP
38Homestead A (CA)76.85STASH
39Ithaca A (NY)76.77IKEA
40Choate A (CT)76.48BHSAT
41East Brunswick B (NJ)75.88BHSAT
42Ridgewood A (NJ)75.87WORKSHOP
43Chattahoochee B (GA)75.63SATURNALIA
44Detroit Catholic Central A (MI)75.57WORKSHOP
45Hotchkiss A (CT)75.49Prison Bowl
46Carl Sandburg A (IL)75.21STASH
47Del Norte A (CA)74.52LIT
48Harker A (CA)74.51SATURNALIA
49Bellarmine College Prep A (CA)74.29STASH
50Woodford County A (KY)73.37WORKSHOP
51Wayzata C (MN)73.07Prison Bowl
52Mission San Jose A (CA)72.18STASH
53Centennial A (MD)72.16STASH
54Livingston A (NJ)71.96Prison Bowl
55Thomas Jefferson Science & Tech B (VA)71.73BHSAT
56Baker A (AL)71.73CATT
57Carmel A (IN)71.47BHSAT
58Stanton College Prep A (FL)71.09Prison Bowl
59Macomb A (IL)70.95STASH
60IMSA A (IL)70.94IKEA
61Raleigh Charter A (NC)70.80SATURNALIA
62Boulder A (CO)70.76BHSAT
63Raymore-Peculiar A (MO)70.72Prison Bowl
64Lexington A (MA)70.31BHSAT
65Hoover A (AL)69.96CATT
66Wayzata B (MN)69.80Prison Bowl
67Rockford Auburn A (IL)69.47SATURNALIA
68St. Joseph Central A (MO)69.25BHSAT
69Irvington A (CA)69.23STASH
70East Brunswick A (NJ)68.77SATURNALIA
71Archbishop Mitty B (CA)68.56STASH
72Westview A (OR)68.53Prison Bowl
73Miami Valley A (OH)68.22BHSAT
74Manheim Township A (PA)68.13SATURNALIA
75Grady A (GA)67.64CATT
76College Heights Christian A (MO)67.54BHSAT
77Lowell A (CA)67.13IKEA
78LASA A (TX)66.75STASH
79Northfield A (MN)66.57STASH
80Millburn A (NJ)66.56SATURNALIA
81Adlai E. Stevenson B (IL)65.95LIT
82Trinity School A (NY)65.53SATURNALIA
83Archbishop Mitty A (CA)65.18LIT
84Westview B (CA)63.92LIT
85Buchholz A (FL)63.55Prison Bowl
86Naperville North A (IL)63.48BHSAT
87Winston Churchill A (MD)62.64STASH
88BASIS McLean A (VA)62.49SATURNALIA
89Dublin A (CA)61.79STASH
90Canyon Crest A (CA)61.72SATURNALIA
91Paideia A (GA)61.54Prison Bowl
92Caddo Magnet A (LA)61.53STASH
93La Jolla A (CA)61.30STASH
94John P. Stevens A (NJ)60.55Prison Bowl
95St. Joseph A (IN)60.49STASH
96Beavercreek B (OH)60.30BHSAT
97Thomas Jefferson Science & Tech C (VA)60.28Prison Bowl
98Santa Monica B (CA)60.17STASH
99Barrington B (IL)60.03Prison Bowl
100Darien A (CT)59.96Prison Bowl

For the “Hard Sets”-only rankings, we remove all the HS regular and similar difficulty sets from the system, keeping only the HS regs+, HS Nats, and the college sets in order to rank teams based on how they did on harder sets. The scores are a little bit deflated here since we had to change the baseline set since IS-195 would not be included as one of the hard sets, but the actual scores probably don’t matter as much as the relative difference in scores/ordering of teams.

The top 4 in the pre-HSNCT and hard sets only rankings are the same with #5 being different probably only due to the fact that GDS A is not attending HSNCT. There are slight variances here and there – Lambert for example jumps from #18 in pre-HSNCT rankings to #13 in the hard sets only rankings and Westview jumps from #17 in pre-HSNCT rankings to #12 in the hard sets only rankings. There are also changes in the other direction (my team of Stevenson unfortunately being one of the ones that slip when only hard sets are considered).

How much does this mean? Not sure. For some teams the discrepancy can be explained by the fact that they have not had as many opportunities to play harder sets, for others maybe playing harder sets allowed them to separate themselves from the rest of the field more than they could on regular difficulty sets. “Scaling up” or being able to play nearly as well on harder difficulty sets as on regular difficulty sets is a good skill to have for nationals where the questions will be much harder than on regular difficulty, so seeing how teams do on harder difficulty sets is useful for this measure.

Personally, I think the combination of NAQT-only rankings and hard sets only rankings allows us to better appreciate Lambert’s status as a title contender for HSNCT than the actual pre-HSNCT rankings do. That to me is the fun of looking through these alternative modifications – people can decide what they value as important in a team’s past performances and look at the appropriate modified rankings.

Of course whenever discussion of past performances on hard sets comes up, there comes the argument that hard sets are in fact detrimental to accurate ranking of teams. The argument goes that since college sets in particular have such high adjustments, this leads to inflated rankings for teams that play college sets. So the question naturally arises: how different would the rankings be if we excluded college sets?

That takes us to the third and final alternative ranking modification we will be looking at:

HS Set-Only Rankings

RANKTEAMSCOREBEST SET
1Thomas Jefferson Science & Tech A (VA)127.43BHSAT
2Arcadia A (CA)115.81STASH
3Richard Montgomery A (MD)114.70Prison Bowl
4Montgomery Blair A (MD)113.91BHSAT
5Wayzata A (MN)113.02STASH
6Stanton College Prep B (FL)111.60Prison Bowl
7Beavercreek A (OH)111.28LONE STAR
8Hunter A (NY)110.58DART
9Georgetown Day A (DC)109.24BHSAT
10Solon A (OH)109.18MCMT
11Adlai E. Stevenson A (IL)107.85Prison Bowl
12St. Mark’s School of Texas A (TX)107.61STASH
13Belmont A (MA)107.17SATURNALIA
14Chattahoochee A (GA)107.13RAFT
15TAG Magnet A (TX)106.72DART
16Detroit Country Day A (MI)106.68MCMT
17University Lab A (IL)106.56RAFT
18Kinkaid A (TX)105.03Prison Bowl
19Ladue A (MO)104.33BHSAT
20Barrington A (IL)103.77BHSAT
21Lambert A (GA)103.43IS-199
22Greenhill A (TX)102.99MCMT
23Stanford Online A (CA)102.37STASH
24East Chapel Hill A (NC)101.29LONE STAR
25Westview A (CA)101.19MCMT
26Buffalo Grove A (IL)100.71STASH
27High Tech A (NJ)100.69BHSAT
28Phillips Academy A (MA)99.79IS-199
29Mira Loma A (CA)99.25LONE STAR
30Santa Monica A (CA)98.52CATT
31Russell A (KY)98.07Scottie
32Detroit Country Day B (MI)97.61CATT
33Burnsville A (MN)96.49IS-201
34Chattahoochee B (GA)96.06DART
35Great Valley A (PA)95.73RAFT
36East Brunswick B (NJ)95.24IS-199
37Woodford County A (KY)95.08Scottie
38Raffles A (SG)94.96IS-195
39Hunter B (NY)94.03IS-197
40Hamilton A (AZ)93.75IS-199
41Mounds View A (MN)93.28LONE STAR
42Macomb A (IL)93.16DART
43Carmel A (IN)92.60LONE STAR
44North Carolina Science and Math A (NC)92.22SATURNALIA
45Westview A (OR)92.07DART
46Homestead A (CA)91.71IS-195
47Detroit Catholic Central A (MI)91.53DART
48Strake Jesuit A (TX)91.42IS-195
49Miami Valley A (OH)91.39Scottie
50Hotchkiss A (CT)90.96Prison Bowl
51Centennial A (MD)90.70IS-197
52Choate A (CT)90.66BHSAT
53East Brunswick A (NJ)90.62IS-199
54William Fremd A (IL)90.52RAFT
55duPont Manual A (KY)90.21IS-195
56Carl Sandburg A (IL)89.49STASH
57St. Joseph Central A (MO)89.20Scottie
58Ridgewood A (NJ)89.13BHSAT
59Del Norte A (CA)88.89LONE STAR
60Ed W. Clark A (NV)88.61IS-201
61Wayzata C (MN)88.34IS-197
62Manheim Township A (PA)88.10IS-199
63Cinco Ranch A (TX)87.92STASH
64Raymore-Peculiar A (MO)87.86DART
65BASIS Silicon Valley A (CA)87.79IS-195
66Rockford Auburn A (IL)87.75DART
67Wayzata B (MN)87.61LONE STAR
68Livingston A (NJ)87.33IS-197
69Hoover A (AL)87.33Spring Novice
70Irvington A (CA)87.12IS-195
71Baker A (AL)86.95CATT
72Dunbar A (KY)86.82DART
73Grady A (GA)86.21IS-199
74College Heights Christian A (MO)86.17MCMT
75Harker A (CA)85.67SATURNALIA
76IMSA A (IL)85.65RAFT
77Seven Lakes A (TX)85.44IS-199
78Lowell A (CA)85.23IS-197
79Thomas Jefferson Science & Tech B (VA)85.17BHSAT
80Archbishop Mitty B (CA)84.48LONE STAR
81Mission San Jose A (CA)84.48STASH
82Caddo Magnet A (LA)84.45RAFT
83Eden Prairie A (MN)84.32IS-201
84Boulder A (CO)84.25BHSAT
85Bellarmine College Prep A (CA)84.23STASH
86Raleigh Charter A (NC)84.09Spring Novice
87Maggie Walker Governor’s School B (VA)83.80IS-195
88Solon B (OH)83.61IS-195
89John P. Stevens A (NJ)83.43IS-199
90Paideia A (GA)83.01IS-201
91Kirksville A (MO)82.97MCMT
92Lexington A (MA)82.67BHSAT
93Stanton College Prep A (FL)82.17Prison Bowl
94St. Joseph A (IN)81.72IS-201
95Waukee A (IA)81.55LONE STAR
96Adlai E. Stevenson B (IL)81.49RAFT
97St. Louis Patriots A (MO)81.47MCMT
98Northmont A (OH)81.20IS-199
99Westminster A (GA)81.03IS-201
100Troy A (MI)80.99LONE STAR

You can look at the pre-HSNCT rankings again. To me, though, these rankings don’t seem that different with or without college sets. I really don’t have much commentary to add here – I think people were likely overly concerned about the inflating impact of college sets more last semester and that makes sense since there were two big college sets (LIT and IKEA) that a lot of the top teams could have played, and not that many actual high school sets. As the season went on these teams played more high school sets and also a lot of them happened to improve, so it became more likely that their top performance was no longer a college set but a high school set.

It is also harder to distinguish teams at the high school regular level just because there is a ceiling at some point on how many powers per game and PPB that even top teams can put up on an IS set or whatever. That also makes sense then that the adjustments would elevate stronger performances on harder sets – since its easier for the top teams to differentiate into tiers then when not every bonus will be 30d, or every tossup powered.

(I separately have the take that top teams in circuits with an abundance of sets should not play high school regular sets at a certain point, but that is probably a discussion for another day.)

Even when removing college sets, the top scores for most of these teams comes from HS regs+ or HS nats level sets. I’m sure you could then ponder what the scores would look like if only high school regular sets were considered in order to remove “inflation” from more difficult sets, but that doesn’t seem particularly interesting to look at (at least in my opinion.)

Anyways, with that, that concludes this analysis of sorts. Hopefully you guys enjoyed it! A reminder that our pre-nationals poll https://hsquizbowl.org/forums/viewtopic.php?f=4&t=25112 is still open until Thursday, May 27, 2021 at 11:59 PM CDT.

Good luck at HSNCT for those participating, and for the rest of you hope you have a great rest of the year! On a personal note, with this being my last year of high school quizbowl I just want to take one last moment to thank the quizbowl community for everything it has given me. Obviously this community is far from perfect, but I’m not sure what my high school experience would have been like without it.

Thanks!

Arjun Nageswaran is currently a senior at Adlai E. Stevenson High School in Lincolnshire, Illinois

One thought on “Alternative Ranking Modifications: An Analysis

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s