Alternative Ranking Modifications: An Analysis

The following article was a contribution by Groger Ranks member Arjun Nageswaran.

Before beginning this post, it is worth noting that all the potential rankings in this article will exclude Saratoga A from California. Two players from the Saratoga club were recently (and rightfully) removed for misconduct, and it was an error on our part to not remove them from the rankings before posting the pre-HSNCT rankings. We obviously have zero-tolerance for any misconduct in quizbowl and all future rankings (starting with the ones discussed in this article) will discount any performances from the Saratoga A team with those two players. A special thanks to Nicole Pruitt for catching this error in our pre-HSNCT rankings.

In this article, I wanted to go through three alternative ranking modifications and see how that would change the order of teams. This article is more for public interest rather than anything else, but I hope you guys find some things interesting here! These rankings will take into account all teams rather than just teams going to HSNCT and/or PACE NSC. Additionally, there will be no mACF-only rankings in this article since we plan to release that separately as part of our pre-PACE NSC rankings.

We’ll start with the first one:

NAQT-Only Rankings

RANK	TEAM	SCORE	BEST SET
1	Stanton College Prep B (FL)	111.23	IS-197
2	Hunter A (NY)	107.38	IS-201
3	Lambert A (GA)	105.64	IS-199
4	Arcadia A (CA)	105.24	IS-197
5	Richard Montgomery A (MD)	104.23	IS-199
6	Beavercreek A (OH)	103.13	IS-195
7	St. Mark’s School of Texas A (TX)	102.61	IS-199
8	Phillips Academy A (MA)	101.72	IS-199
9	Stanford Online A (CA)	101.50	IS-197
10	Ladue A (MO)	101.24	IS-195
11	Wayzata A (MN)	100.94	IS-197
12	Detroit Country Day A (MI)	100.52	IS-201
13	East Brunswick B (NJ)	100.49	IS-199
14	Solon A (OH)	98.85	IS-195
15	East Brunswick A (NJ)	98.73	IS-199
16	Greenhill A (TX)	98.45	IS-199
17	TAG Magnet A (TX)	98.36	IS-195
18	East Chapel Hill A (NC)	97.28	IS-199
19	Burnsville A (MN)	96.93	IS-201
20	Georgetown Day A (DC)	95.47	IS-195
21	Russell A (KY)	95.25	IS-199
22	Raffles A (SG)	94.96	IS-195
23	Chattahoochee A (GA)	94.10	IS-195
24	Mira Loma A (CA)	94.09	IS-197
25	High Tech A (NJ)	94.03	IS-195
26	Hunter B (NY)	93.99	IS-197
27	Hamilton A (AZ)	93.81	IS-199
28	Great Valley A (PA)	93.12	IS-197
29	Strake Jesuit A (TX)	91.62	IS-195
30	Santa Monica A (CA)	90.79	IS-197
31	Carmel A (IN)	90.71	IS-201
32	Centennial A (MD)	90.59	IS-197
33	duPont Manual A (KY)	90.49	IS-195
34	Miami Valley A (OH)	89.99	IS-199
35	Hotchkiss A (CT)	89.80	IS-201
36	Mounds View A (MN)	89.67	IS-197
37	Belmont A (MA)	89.47	IS-197
38	Ed W. Clark A (NV)	89.26	IS-201
39	Detroit Catholic Central A (MI)	88.88	IS-201
40	Detroit Country Day B (MI)	88.84	IS-199
41	Homestead A (CA)	88.69	IS-195
42	Livingston A (NJ)	88.31	IS-197
43	Del Norte A (CA)	87.99	IS-197
44	Choate A (CT)	87.99	IS-201
45	Manheim Township A (PA)	87.68	IS-199
46	Irvington A (CA)	87.43	IS-195
47	Grady A (GA)	87.02	IS-199
48	BASIS Silicon Valley A (CA)	86.84	IS-195
49	North Carolina Science and Math A (NC)	86.20	IS-199
50	Woodford County A (KY)	86.16	IS-195
51	Wayzata C (MN)	86.07	IS-197
52	Seven Lakes A (TX)	85.47	IS-199
53	John P. Stevens A (NJ)	84.75	IS-201
54	Eden Prairie A (MN)	84.58	IS-201
55	Westminster A (GA)	84.52	IS-201
56	St. Joseph A (IN)	84.25	IS-201
57	Wayzata B (MN)	84.10	IS-197
58	Solon B (OH)	84.01	IS-195
59	Lowell A (CA)	83.99	IS-197
60	Maggie Walker Governor’s School B (VA)	83.80	IS-195
61	Raymore-Peculiar A (MO)	83.69	IS-199
62	Paideia A (GA)	83.39	IS-201
63	Caddo Magnet A (LA)	82.79	IS-195
64	Maggie Walker Governor’s School A (VA)	81.98	IS-195
65	Cinco Ranch A (TX)	81.60	IS-201
66	Northmont A (OH)	81.15	IS-199
67	Johns Creek A (GA)	81.08	IS-201
68	Lynbrook A (CA)	80.87	IS-199
69	Westview A (OR)	80.50	IS-199
70	Buchholz A (FL)	80.44	IS-201
71	Freeman A (VA)	80.42	IS-195
72	Carmel B (IN)	80.22	IS-195
73	William Henry Harrison A (IN)	80.05	IS-201
74	St. Joseph Central A (MO)	79.93	IS-199
75	Archbishop Mitty B (CA)	79.51	IS-199
76	Monta Vista A (CA)	79.51	IS-197
77	Ithaca A (NY)	79.47	IS-199
78	Baton Rouge Episcopal A (LA)	79.36	IS-199
79	Bellarmine College Prep A (CA)	79.34	IS-197
80	Tanglin Trust B (SG)	79.30	IS-195
81	Chattahoochee B (GA)	78.82	IS-195
82	LASA A (TX)	78.64	IS-195
83	Winston Churchill A (MD)	78.60	IS-199
84	BASIS Scottsdale A (AZ)	78.14	IS-199
85	Northfield A (MN)	77.97	IS-197
86	Acton-Boxborough A (MA)	77.25	IS-197
87	Jasper A (TX)	77.23	IS-195
88	Folsom A (CA)	77.23	IS-195
89	Washington A (CA)	77.12	IS-197
90	Prior Lake A (MN)	76.77	IS-197
91	Coppell A (TX)	76.74	IS-197
92	Dunbar A (KY)	76.55	IS-201
93	Darien A (CT)	76.41	IS-197
94	Archbishop Mitty A (CA)	76.25	IS-195
95	Wheeler A (GA)	76.09	IS-201
96	Bergen Tech-Teterboro A (NJ)	76.04	IS-195
97	Troy A (MI)	75.90	IS-201
98	James E. Taylor A (TX)	75.84	IS-201
99	BASIS McLean A (VA)	75.74	IS-195
100	Mission San Jose A (CA)	75.65	IS-199

These rankings imagine what the top 100 would look like if we only took into account performances on NAQT sets. Our pre-HSNCT rankings would have ordinarily been NAQT only rankings only including teams attending HSNCT. Why couldn’t we do that this year? If you look at the discrepancy between these NAQT-only rankings and our pre-HSNCT rankings, you’ll notice that we miss NAQT data from a lot of the teams attending HSNCT including the #1 in our pre-HSNCT rankings, TJ A, and essentially all the Illinois teams. As a result, it would probably be misleading to label such a ranking as “pre-HSNCT rankings.” Of course by including both mACF and NAQT sets this makes our current pre-HSNCT rankings not the best measure possible, but that was probably the better of the two choices. This had no bearing on the decision to not have NAQT-only pre-HSNCT rankings, but additionally since there is no SCT set this year, this would have meant that all of the sets for an NAQT-only ranking can only include IS sets and SSNCT. These sets are around HS regular difficulty, and as a result there would be little information about teams’ ability to scale up to harder difficulties, less than say a performance on DII SCT would have in past years.

For me, the most interesting difference between the pre-HSNCT and the NAQT-only rankings (besides several teams being omitted from NAQT-only rankings) is the ordering of the top 4. With TJ A having no NAQT performances, it would be expected that Arcadia A as the #2 ranked team would be the new #1 for NAQT-only rankings. However, Arcadia got bumped down to #4 with the top 3 being 1. SCP B (ranked #5 in the pre-HSNCT rankings), 2. Hunter A (ranked #8 in the pre-HSNCT rankings), and 3. Lambert A (ranked #18 in the pre-HSNCT rankings). Lambert probably has the biggest jump when only NAQT sets are taken into account.

What does this mean? There are certainly teams who are much stronger on NAQT than mACF and vice versa, and as a result being able to look at the NAQT-only rankings could provide clarification into potential HSNCT predictions in a way that overall rankings can not. Of course like stated before there are several teams being omitted from the NAQT-only rankings, and the sets are all much easier than what HSNCT will end up being, but it’s still pretty interesting to look at.

This discussion of considering teams’ abilities to scale takes us to our second alternative ranking modification we will be looking at:

“Hard Sets”-Only Rankings

RANK	TEAM	SCORE	BEST SET
1	Thomas Jefferson Science & Tech A (VA)	109.45	BHSAT
2	Arcadia A (CA)	101.50	STASH
3	Richard Montgomery A (MD)	98.05	Prison Bowl
4	Wayzata A (MN)	97.02	STASH
5	Georgetown Day A (DC)	96.69	LIT
6	Montgomery Blair A (MD)	95.20	BHSAT
7	Stanton College Prep B (FL)	94.32	Prison Bowl
8	University Lab A (IL)	93.74	LIT
9	Beavercreek A (OH)	92.89	SATURNALIA
10	Belmont A (MA)	92.01	SATURNALIA
11	Hunter A (NY)	91.74	WORKSHOP
12	Westview A (CA)	91.52	IKEA
13	Lambert A (GA)	90.89	WORKSHOP
14	St. Mark’s School of Texas A (TX)	90.52	STASH
15	Adlai E. Stevenson A (IL)	90.50	Prison Bowl
16	Detroit Country Day A (MI)	89.69	STASH
17	Greenhill A (TX)	89.05	WORKSHOP
18	Stanford Online A (CA)	88.96	STASH
19	Solon A (OH)	88.90	BHSAT
20	Kinkaid A (TX)	88.68	Prison Bowl
21	Barrington A (IL)	88.24	BHSAT
22	Ladue A (MO)	87.56	BHSAT
23	Chattahoochee A (GA)	86.68	SATURNALIA
24	TAG Magnet A (TX)	86.52	Prison Bowl
25	Buffalo Grove A (IL)	85.89	STASH
26	East Chapel Hill A (NC)	85.34	LIT
27	High Tech A (NJ)	84.36	BHSAT
28	Mira Loma A (CA)	83.88	SATURNALIA
29	Santa Monica A (CA)	82.20	CATT
30	Strake Jesuit A (TX)	80.75	LIT
31	Detroit Country Day B (MI)	80.63	STASH
32	Great Valley A (PA)	79.05	SATURNALIA
33	Phillips Academy A (MA)	78.23	STASH
34	Cinco Ranch A (TX)	77.85	STASH
35	Mounds View A (MN)	77.46	LIT
36	North Carolina Science and Math A (NC)	77.04	SATURNALIA
37	Hunter B (NY)	76.90	WORKSHOP
38	Homestead A (CA)	76.85	STASH
39	Ithaca A (NY)	76.77	IKEA
40	Choate A (CT)	76.48	BHSAT
41	East Brunswick B (NJ)	75.88	BHSAT
42	Ridgewood A (NJ)	75.87	WORKSHOP
43	Chattahoochee B (GA)	75.63	SATURNALIA
44	Detroit Catholic Central A (MI)	75.57	WORKSHOP
45	Hotchkiss A (CT)	75.49	Prison Bowl
46	Carl Sandburg A (IL)	75.21	STASH
47	Del Norte A (CA)	74.52	LIT
48	Harker A (CA)	74.51	SATURNALIA
49	Bellarmine College Prep A (CA)	74.29	STASH
50	Woodford County A (KY)	73.37	WORKSHOP
51	Wayzata C (MN)	73.07	Prison Bowl
52	Mission San Jose A (CA)	72.18	STASH
53	Centennial A (MD)	72.16	STASH
54	Livingston A (NJ)	71.96	Prison Bowl
55	Thomas Jefferson Science & Tech B (VA)	71.73	BHSAT
56	Baker A (AL)	71.73	CATT
57	Carmel A (IN)	71.47	BHSAT
58	Stanton College Prep A (FL)	71.09	Prison Bowl
59	Macomb A (IL)	70.95	STASH
60	IMSA A (IL)	70.94	IKEA
61	Raleigh Charter A (NC)	70.80	SATURNALIA
62	Boulder A (CO)	70.76	BHSAT
63	Raymore-Peculiar A (MO)	70.72	Prison Bowl
64	Lexington A (MA)	70.31	BHSAT
65	Hoover A (AL)	69.96	CATT
66	Wayzata B (MN)	69.80	Prison Bowl
67	Rockford Auburn A (IL)	69.47	SATURNALIA
68	St. Joseph Central A (MO)	69.25	BHSAT
69	Irvington A (CA)	69.23	STASH
70	East Brunswick A (NJ)	68.77	SATURNALIA
71	Archbishop Mitty B (CA)	68.56	STASH
72	Westview A (OR)	68.53	Prison Bowl
73	Miami Valley A (OH)	68.22	BHSAT
74	Manheim Township A (PA)	68.13	SATURNALIA
75	Grady A (GA)	67.64	CATT
76	College Heights Christian A (MO)	67.54	BHSAT
77	Lowell A (CA)	67.13	IKEA
78	LASA A (TX)	66.75	STASH
79	Northfield A (MN)	66.57	STASH
80	Millburn A (NJ)	66.56	SATURNALIA
81	Adlai E. Stevenson B (IL)	65.95	LIT
82	Trinity School A (NY)	65.53	SATURNALIA
83	Archbishop Mitty A (CA)	65.18	LIT
84	Westview B (CA)	63.92	LIT
85	Buchholz A (FL)	63.55	Prison Bowl
86	Naperville North A (IL)	63.48	BHSAT
87	Winston Churchill A (MD)	62.64	STASH
88	BASIS McLean A (VA)	62.49	SATURNALIA
89	Dublin A (CA)	61.79	STASH
90	Canyon Crest A (CA)	61.72	SATURNALIA
91	Paideia A (GA)	61.54	Prison Bowl
92	Caddo Magnet A (LA)	61.53	STASH
93	La Jolla A (CA)	61.30	STASH
94	John P. Stevens A (NJ)	60.55	Prison Bowl
95	St. Joseph A (IN)	60.49	STASH
96	Beavercreek B (OH)	60.30	BHSAT
97	Thomas Jefferson Science & Tech C (VA)	60.28	Prison Bowl
98	Santa Monica B (CA)	60.17	STASH
99	Barrington B (IL)	60.03	Prison Bowl
100	Darien A (CT)	59.96	Prison Bowl

For the “Hard Sets”-only rankings, we remove all the HS regular and similar difficulty sets from the system, keeping only the HS regs+, HS Nats, and the college sets in order to rank teams based on how they did on harder sets. The scores are a little bit deflated here since we had to change the baseline set since IS-195 would not be included as one of the hard sets, but the actual scores probably don’t matter as much as the relative difference in scores/ordering of teams.

The top 4 in the pre-HSNCT and hard sets only rankings are the same with #5 being different probably only due to the fact that GDS A is not attending HSNCT. There are slight variances here and there – Lambert for example jumps from #18 in pre-HSNCT rankings to #13 in the hard sets only rankings and Westview jumps from #17 in pre-HSNCT rankings to #12 in the hard sets only rankings. There are also changes in the other direction (my team of Stevenson unfortunately being one of the ones that slip when only hard sets are considered).

How much does this mean? Not sure. For some teams the discrepancy can be explained by the fact that they have not had as many opportunities to play harder sets, for others maybe playing harder sets allowed them to separate themselves from the rest of the field more than they could on regular difficulty sets. “Scaling up” or being able to play nearly as well on harder difficulty sets as on regular difficulty sets is a good skill to have for nationals where the questions will be much harder than on regular difficulty, so seeing how teams do on harder difficulty sets is useful for this measure.

Personally, I think the combination of NAQT-only rankings and hard sets only rankings allows us to better appreciate Lambert’s status as a title contender for HSNCT than the actual pre-HSNCT rankings do. That to me is the fun of looking through these alternative modifications – people can decide what they value as important in a team’s past performances and look at the appropriate modified rankings.

Of course whenever discussion of past performances on hard sets comes up, there comes the argument that hard sets are in fact detrimental to accurate ranking of teams. The argument goes that since college sets in particular have such high adjustments, this leads to inflated rankings for teams that play college sets. So the question naturally arises: how different would the rankings be if we excluded college sets?

That takes us to the third and final alternative ranking modification we will be looking at:

HS Set-Only Rankings

RANK	TEAM	SCORE	BEST SET
1	Thomas Jefferson Science & Tech A (VA)	127.43	BHSAT
2	Arcadia A (CA)	115.81	STASH
3	Richard Montgomery A (MD)	114.70	Prison Bowl
4	Montgomery Blair A (MD)	113.91	BHSAT
5	Wayzata A (MN)	113.02	STASH
6	Stanton College Prep B (FL)	111.60	Prison Bowl
7	Beavercreek A (OH)	111.28	LONE STAR
8	Hunter A (NY)	110.58	DART
9	Georgetown Day A (DC)	109.24	BHSAT
10	Solon A (OH)	109.18	MCMT
11	Adlai E. Stevenson A (IL)	107.85	Prison Bowl
12	St. Mark’s School of Texas A (TX)	107.61	STASH
13	Belmont A (MA)	107.17	SATURNALIA
14	Chattahoochee A (GA)	107.13	RAFT
15	TAG Magnet A (TX)	106.72	DART
16	Detroit Country Day A (MI)	106.68	MCMT
17	University Lab A (IL)	106.56	RAFT
18	Kinkaid A (TX)	105.03	Prison Bowl
19	Ladue A (MO)	104.33	BHSAT
20	Barrington A (IL)	103.77	BHSAT
21	Lambert A (GA)	103.43	IS-199
22	Greenhill A (TX)	102.99	MCMT
23	Stanford Online A (CA)	102.37	STASH
24	East Chapel Hill A (NC)	101.29	LONE STAR
25	Westview A (CA)	101.19	MCMT
26	Buffalo Grove A (IL)	100.71	STASH
27	High Tech A (NJ)	100.69	BHSAT
28	Phillips Academy A (MA)	99.79	IS-199
29	Mira Loma A (CA)	99.25	LONE STAR
30	Santa Monica A (CA)	98.52	CATT
31	Russell A (KY)	98.07	Scottie
32	Detroit Country Day B (MI)	97.61	CATT
33	Burnsville A (MN)	96.49	IS-201
34	Chattahoochee B (GA)	96.06	DART
35	Great Valley A (PA)	95.73	RAFT
36	East Brunswick B (NJ)	95.24	IS-199
37	Woodford County A (KY)	95.08	Scottie
38	Raffles A (SG)	94.96	IS-195
39	Hunter B (NY)	94.03	IS-197
40	Hamilton A (AZ)	93.75	IS-199
41	Mounds View A (MN)	93.28	LONE STAR
42	Macomb A (IL)	93.16	DART
43	Carmel A (IN)	92.60	LONE STAR
44	North Carolina Science and Math A (NC)	92.22	SATURNALIA
45	Westview A (OR)	92.07	DART
46	Homestead A (CA)	91.71	IS-195
47	Detroit Catholic Central A (MI)	91.53	DART
48	Strake Jesuit A (TX)	91.42	IS-195
49	Miami Valley A (OH)	91.39	Scottie
50	Hotchkiss A (CT)	90.96	Prison Bowl
51	Centennial A (MD)	90.70	IS-197
52	Choate A (CT)	90.66	BHSAT
53	East Brunswick A (NJ)	90.62	IS-199
54	William Fremd A (IL)	90.52	RAFT
55	duPont Manual A (KY)	90.21	IS-195
56	Carl Sandburg A (IL)	89.49	STASH
57	St. Joseph Central A (MO)	89.20	Scottie
58	Ridgewood A (NJ)	89.13	BHSAT
59	Del Norte A (CA)	88.89	LONE STAR
60	Ed W. Clark A (NV)	88.61	IS-201
61	Wayzata C (MN)	88.34	IS-197
62	Manheim Township A (PA)	88.10	IS-199
63	Cinco Ranch A (TX)	87.92	STASH
64	Raymore-Peculiar A (MO)	87.86	DART
65	BASIS Silicon Valley A (CA)	87.79	IS-195
66	Rockford Auburn A (IL)	87.75	DART
67	Wayzata B (MN)	87.61	LONE STAR
68	Livingston A (NJ)	87.33	IS-197
69	Hoover A (AL)	87.33	Spring Novice
70	Irvington A (CA)	87.12	IS-195
71	Baker A (AL)	86.95	CATT
72	Dunbar A (KY)	86.82	DART
73	Grady A (GA)	86.21	IS-199
74	College Heights Christian A (MO)	86.17	MCMT
75	Harker A (CA)	85.67	SATURNALIA
76	IMSA A (IL)	85.65	RAFT
77	Seven Lakes A (TX)	85.44	IS-199
78	Lowell A (CA)	85.23	IS-197
79	Thomas Jefferson Science & Tech B (VA)	85.17	BHSAT
80	Archbishop Mitty B (CA)	84.48	LONE STAR
81	Mission San Jose A (CA)	84.48	STASH
82	Caddo Magnet A (LA)	84.45	RAFT
83	Eden Prairie A (MN)	84.32	IS-201
84	Boulder A (CO)	84.25	BHSAT
85	Bellarmine College Prep A (CA)	84.23	STASH
86	Raleigh Charter A (NC)	84.09	Spring Novice
87	Maggie Walker Governor’s School B (VA)	83.80	IS-195
88	Solon B (OH)	83.61	IS-195
89	John P. Stevens A (NJ)	83.43	IS-199
90	Paideia A (GA)	83.01	IS-201
91	Kirksville A (MO)	82.97	MCMT
92	Lexington A (MA)	82.67	BHSAT
93	Stanton College Prep A (FL)	82.17	Prison Bowl
94	St. Joseph A (IN)	81.72	IS-201
95	Waukee A (IA)	81.55	LONE STAR
96	Adlai E. Stevenson B (IL)	81.49	RAFT
97	St. Louis Patriots A (MO)	81.47	MCMT
98	Northmont A (OH)	81.20	IS-199
99	Westminster A (GA)	81.03	IS-201
100	Troy A (MI)	80.99	LONE STAR

You can look at the pre-HSNCT rankings again. To me, though, these rankings don’t seem that different with or without college sets. I really don’t have much commentary to add here – I think people were likely overly concerned about the inflating impact of college sets more last semester and that makes sense since there were two big college sets (LIT and IKEA) that a lot of the top teams could have played, and not that many actual high school sets. As the season went on these teams played more high school sets and also a lot of them happened to improve, so it became more likely that their top performance was no longer a college set but a high school set.

It is also harder to distinguish teams at the high school regular level just because there is a ceiling at some point on how many powers per game and PPB that even top teams can put up on an IS set or whatever. That also makes sense then that the adjustments would elevate stronger performances on harder sets – since its easier for the top teams to differentiate into tiers then when not every bonus will be 30d, or every tossup powered.

(I separately have the take that top teams in circuits with an abundance of sets should not play high school regular sets at a certain point, but that is probably a discussion for another day.)

Even when removing college sets, the top scores for most of these teams comes from HS regs+ or HS nats level sets. I’m sure you could then ponder what the scores would look like if only high school regular sets were considered in order to remove “inflation” from more difficult sets, but that doesn’t seem particularly interesting to look at (at least in my opinion.)

Anyways, with that, that concludes this analysis of sorts. Hopefully you guys enjoyed it! A reminder that our pre-nationals poll https://hsquizbowl.org/forums/viewtopic.php?f=4&t=25112 is still open until Thursday, May 27, 2021 at 11:59 PM CDT.

Good luck at HSNCT for those participating, and for the rest of you hope you have a great rest of the year! On a personal note, with this being my last year of high school quizbowl I just want to take one last moment to thank the quizbowl community for everything it has given me. Obviously this community is far from perfect, but I’m not sure what my high school experience would have been like without it.

Thanks!

Arjun Nageswaran is currently a senior at Adlai E. Stevenson High School in Lincolnshire, Illinois