Computer vs. Human Rankings

Older Topics, Not the current discussion

Moderators: Mitch Hawker, east hockey, karl(east)

Post Reply
scorekeeper
Posts: 569
Joined: Tue Oct 25, 2011 1:08 am

Post by scorekeeper »

karl(east) wrote:
scorekeeper wrote: It will be interesting to see what the actual rankings are. First release is December 5th and will begin tightening up in accuracy by December 12th and onward.
What do you mean by "actual" rankings? AP/Let's Play Hockey/MNHockeyHub are all the same thing, and they're already putting out rankings.
I mean fact based rankings. Mathematical.

The ones you mention are all great fun, but they are just opinion polls. They are a great conversation piece, and a lot of fun to debate but in terms of accurate quantification of team strength, they have inherent flaws and are notoriously inferior to mathematical quanitifcation.

Of course, mathematically, it's impossible to have a pre-season poll, so there is definitely a place for opinion polls in that regard, but once the players take the ice and the games are being players, results replace opinions.

East Ridge looks to me like a 18 or 20 win team and they will definitely be a threat in the Section.

Fingers crossed on the return of our vets, and you're welcome on the blue colored recap! :)
observer
Posts: 2225
Joined: Wed Apr 02, 2008 8:45 pm

Post by observer »

I mean fact based rankings. Mathematical.

The ones you mention are all great fun, but they are just opinion polls.
You'll do better in your opinion poll than the mathematical one.
scorekeeper
Posts: 569
Joined: Tue Oct 25, 2011 1:08 am

Post by scorekeeper »

observer wrote:
I mean fact based rankings. Mathematical.

The ones you mention are all great fun, but they are just opinion polls.
You'll do better in your opinion poll than the mathematical one.
Has nothing to do with me. I'm not in either. Like your moniker, I am just an observer

As it is, at least one mathematical module already has Eagan listed higher than the opinion polls
http://www.bgoski.com/b/KRACH_AA_20121130.htm

Not near enough data for this to be accurate. As I said, it'll start coming into focus by December 12th
karl(east)
Posts: 6462
Joined: Tue Jul 17, 2007 9:03 pm
Contact:

Post by karl(east) »

scorekeeper wrote:
karl(east) wrote:
scorekeeper wrote: It will be interesting to see what the actual rankings are. First release is December 5th and will begin tightening up in accuracy by December 12th and onward.
What do you mean by "actual" rankings? AP/Let's Play Hockey/MNHockeyHub are all the same thing, and they're already putting out rankings.
I mean fact based rankings. Mathematical.

The ones you mention are all great fun, but they are just opinion polls. They are a great conversation piece, and a lot of fun to debate but in terms of accurate quantification of team strength, they have inherent flaws and are notoriously inferior to mathematical quanitifcation.

Of course, mathematically, it's impossible to have a pre-season poll, so there is definitely a place for opinion polls in that regard, but once the players take the ice and the games are being players, results replace opinions.
Alright, makes sense--no doubt the math-based ones have some major benefits that us silly humans don't have. Within the confines of what they measure, they are 100% correct. The problem comes from those confines, which are selected by a human somewhere along the line. The good ones (Minnhock's PS2, USHSHO, MyHockeyRankings) have been tested so often that they are often very accurate. Their weakness is that, for the sake of clean, hard accuracy based on a few principles (basically, scores and SOS), they strip away all nuance.

The AP/Hub/LPH poll has the benefit of using many different voters' opinions, so that one person's bias can't affect the poll too much; the problem with their process is that it is very opaque, and we have little to no way of seeing why team X is listed ahead of team Y. We can guess, but sometimes we're just left scratching our heads.

Which is why I try to do my own on here/at mnhockeyprospects. I'd never say mine are better, or that I don't have biases; obviously I have to, since I can't possibly see every minute of every game. But I can check my work against the computers while still factoring in some of the things they miss (injuries, momentum, tactics and strategies, empty-netters, when goals were scored in a game, control of play, etc.), and also give explanations that are impossible with the AP/Hub/LPH poll.
scorekeeper
Posts: 569
Joined: Tue Oct 25, 2011 1:08 am

Post by scorekeeper »

karl(east) wrote:
scorekeeper wrote:
karl(east) wrote: What do you mean by "actual" rankings? AP/Let's Play Hockey/MNHockeyHub are all the same thing, and they're already putting out rankings.
I mean fact based rankings. Mathematical.

The ones you mention are all great fun, but they are just opinion polls. They are a great conversation piece, and a lot of fun to debate but in terms of accurate quantification of team strength, they have inherent flaws and are notoriously inferior to mathematical quanitifcation.

Of course, mathematically, it's impossible to have a pre-season poll, so there is definitely a place for opinion polls in that regard, but once the players take the ice and the games are being players, results replace opinions.
Alright, makes sense--no doubt the math-based ones have some major benefits that us silly humans don't have. Within the confines of what they measure, they are 100% correct. The problem comes from those confines, which are selected by a human somewhere along the line.
Strong mathematical rankings are the simplest thing in the world. It's very simply margin of victory over strength of schedule. There is no human element to it whatsoever. This has been tried and true in Las Vegas for decades and billions and billions of dollars have been made pitting math against humans.

The problem comes when humans try and mess with the math. The BCS is a perfect example. The original formula was perfect, but then got messed with as humans wanted more influence over the results. Adding in coaches and AP for example made it worse, not better.

Las Vegas uses a strict margin of victory over strength of schedule. Adjustments are only made for weather and injuries to key players, and even these are very rare.

It's not uncommon to find football teams higher ranked in the AP, for example, as underdogs to lower ranked teams and even unranked teams. While opinion polls are fun, you can actually book big money - and it's done everyday - on mathematical polls.

MYHockeyRankings.com has been notoriously accurate - MORE accurate on Minnesota hockey than experts from right within the state.

The MYHockey algarythm was written by a man in Indiana who likely has never seen a Minnesota team play yet his algarythm is decidedly more accurate than the local experts watching these games first hand.

I believe the sole flaw in the MyHockeyRankings is in the cap at 7 goals. I believe the cap should no less than the initial margin between the two teams.

For example, if a team with a Rating of 14 plays a team with a Rating of 2, the highest Rating the better team can get for the win is 9. Even on a score of 16-1, for example. I believe they should at least be able to recoup their own initial Rating (in this case of 14). Other than that, the MyHockeyRanking system is a mirror of the oddsmaking systems in Las Vegas.

I do keep a personal set that doesn't limit the margin, but as rare as blowouts are, the MyHockeyRankings set is the best public ranking system available for Minnesota Hockey teams.

Anyways, probably hi-jacked this thred enough. It'll be an interesting comparison on December 12th. I think the inital set comes out December 5th, but there won't be enough data to get an accurate read. You really need about 5 scores on each team and good crossover between conferences to get an accurate read.
Last edited by scorekeeper on Fri Nov 30, 2012 3:52 pm, edited 1 time in total.
HShockeywatcher
Posts: 6848
Joined: Thu Aug 31, 2006 10:21 pm

Post by HShockeywatcher »

In your simple "margin over SOS" equation, how exactly is SOS determined?

To have a SOS you need to be basing it on something, which would be a ranking system, with some sort of inherent bias depending on what factors are used.

Ultimately, there is no "correct" ranking system. What makes one team "better" than another in many cases is subjective.
scorekeeper
Posts: 569
Joined: Tue Oct 25, 2011 1:08 am

Post by scorekeeper »

HShockeywatcher wrote:To have a SOS you need to be basing it on something, which would be a ranking system, with some sort of inherent bias depending on what factors are used.
It's just an algorythm based on win percentage and scoring margins. There is no bias whatsoever. It's 100% mathematical, objective and unbias

There may not be a 100% "correct" system, but mathematical systems are more ACCURATE than opinion polls, which is why you can base a billion dollar industry on them
BodyShots
Posts: 1921
Joined: Fri Feb 29, 2008 9:44 am

Post by BodyShots »

Scorekeeper, welcome the the High School Hockey Forum....

I'm going to love reading your drivel throughout the year. :wink:
Last edited by BodyShots on Fri Nov 30, 2012 4:44 pm, edited 1 time in total.
mnmouth
Posts: 621
Joined: Mon Mar 05, 2007 4:06 pm

Post by mnmouth »

scorekeeper wrote:
HShockeywatcher wrote:To have a SOS you need to be basing it on something, which would be a ranking system, with some sort of inherent bias depending on what factors are used.
It's just an algorythm based on win percentage and scoring margins. There is no bias whatsoever. It's 100% mathematical, objective and unbias

There may not be a 100% "correct" system, but mathematical systems are more ACCURATE than opinion polls, which is why you can base a billion dollar industry on them
But mathematical systems are boring, just like your explanation of why they are superior.

There is a place for both. Computers can't sit in the stands with a bag of popcorn in their laps. They can't watch the action on the ice. They can't compare, contrast and argue over what they have seen with other computers. Flawed or not, computers can't make points, accept criticism and become wiser because another computer has witnessed more, learned hard lessons and become wiser. Half the fun of a Sunday in January is to read the human rankings (and computer rankings) on this site. The other half is reading about what we think of them. The other half is the arguing that occurs. Sorry. My math sucks.
scorekeeper
Posts: 569
Joined: Tue Oct 25, 2011 1:08 am

Post by scorekeeper »

mnmouth wrote:
scorekeeper wrote:
HShockeywatcher wrote:To have a SOS you need to be basing it on something, which would be a ranking system, with some sort of inherent bias depending on what factors are used.
It's just an algorythm based on win percentage and scoring margins. There is no bias whatsoever. It's 100% mathematical, objective and unbias

There may not be a 100% "correct" system, but mathematical systems are more ACCURATE than opinion polls, which is why you can base a billion dollar industry on them
But mathematical systems are boring, just like your explanation of why they are superior.

There is a place for both. Computers can't sit in the stands with a bag of popcorn in their laps. They can't watch the action on the ice. They can't compare, contrast and argue over what they have seen with other computers. Flawed or not, computers can't make points, accept criticism and become wiser because another computer has witnessed more, learned hard lessons and become wiser. Half the fun of a Sunday in January is to read the human rankings (and computer rankings) on this site. The other half is reading about what we think of them. The other half is the arguing that occurs. Sorry. My math sucks.
I don't disagree with you at all. Opinion polls are a lot fun to make, read and debate. Especially when they are substantive like Karls are, with good editorial and sound logic. They are also all we have right now. But soon opinion will give way to results and then we will have both
karl(east)
Posts: 6462
Joined: Tue Jul 17, 2007 9:03 pm
Contact:

Post by karl(east) »

**I've split this topic off from the scores thread, since we've veered so far away**
scorekeeper wrote:It's just an algorythm based on win percentage and scoring margins. There is no bias whatsoever. It's 100% mathematical, objective and unbias

There may not be a 100% "correct" system, but mathematical systems are more ACCURATE than opinion polls, which is why you can base a billion dollar industry on them
Well, somewhere along the line, someone did make a subjective decision that goal differential and SOS were all that matters. This was a smart, informed decision that has held up well under extensive testing (just look at the archived comparsion of coaches' seedings vs. PS2 on the "past section tournaments" page at Minnhock), but it was still a human decision. And the system, while perhaps the best we have now, *could* conceivably be improved upon if we were to ever find better ways to quantify everything that goes on in hockey games.

I am reminded of an essay by Bill James (a numbers guy if there ever was one) in which he denounced the use of mathematical rankings as "the end of the conversation." His broader point was that we all need to have a healthy respect for our own ignorance in our ability to measure these things, because as far as we've come with our advanced stats and formulas, we're still only scraping the surface. Because of that, we need to continue to observe, test new theories, and argue about it all.

We do pretty well on that last count on this forum. :lol:
scorekeeper
Posts: 569
Joined: Tue Oct 25, 2011 1:08 am

Post by scorekeeper »

James sounds like a wise man.
karl(east)
Posts: 6462
Joined: Tue Jul 17, 2007 9:03 pm
Contact:

Post by karl(east) »

scorekeeper wrote:James sounds like a wise man.
Indeed he is. :) Reading his stuff completely changed the way I watch sports.

That's the second time I've referenced this particular essay in a month on here, so I may have to type it up and post it when I find the time.
scorekeeper
Posts: 569
Joined: Tue Oct 25, 2011 1:08 am

Post by scorekeeper »

BodyShots wrote:Scorekeeper, welcome the the High School Hockey Forum....

I'm going to love reading your dribble throughout the year. :wink:
Ok. I'll try not to get any on your shoe
ghshockeyfan
Posts: 6132
Joined: Thu Jun 05, 2003 2:33 pm
Location: Inver Grove Heights, MN
Contact:

KRACH Computer Ranking

Post by ghshockeyfan »

I'll be curious to see how KRACH computer ranking compares against some of the others mentioned here (Minnhock's PS2, USHSHO, MyHockeyRankings). KRACH is probably as "pure" as any algorithm in that it doesn't even care about score differential. That's why I prefer it. No additional human intervention.

Interestingly, it is supposed to give you odds on a game taking place. A team with a "10" rating value is considered the average team in the state regardless of Class designation. A team with, for example, a "100" rating is a 10:1 favorite over the average team. Similarly, you can compare any two teams based on their ratings.

SOS is just an average of opponents rankings played as of now.

Any computer ranking is worthless right now as many have said too limited data at this point.

http://www.ushsho.com/forums/viewtopic.php?t=29709
Most Recent Ranking Links:
==========================================
OVERALL:
http://www.bgoski.com/b/KRACH_OA.htm
CLASS AA:
http://www.bgoski.com/b/KRACH_AA.htm
CLASS A:
http://www.bgoski.com/b/KRACH_A.htm
SECTION:
http://www.bgoski.com/b/KRACH_SEC.htm
SOS (Strength-Of-Schedule) is at:
http://www.bgoski.com/b/KRACH_SOS.htm
==========================================

Added info. is at:
http://www.bgoski.com
And more info. on the girls forum at:
http://www.ushsho.com/forums/viewtopic.php?t=29597
http://www.ushsho.com/forums/viewtopic.php?t=27320
Here's the real details about KRACH:
http://www.mscs.dal.ca/~butler/krachexp.htm

And, our friends at USCHO speak to KRACH vs other options (yep, 7+ years ago):
http://www.uscho.com/2005/03/17/lets-get-kraching/

In addition to their ongoing rankings for both genders at both levels:
http://www.uscho.com/rankings/
http://www.uscho.com/rankings/krach/d-i-men/
http://www.uscho.com/rankings/krach/d-i-women/
http://www.uscho.com/rankings/krach/d-iii-men/
http://www.uscho.com/rankings/krach/d-ii-iii-men/
http://www.uscho.com/rankings/krach/d-iii-women/
Basic Explanation
KRACH — or “Ken’s Ratings for American College Hockey” — is the implementation for college hockey of a sophisticated mathematical model known as the Bradley-Terry rating system, first applied to college hockey by a statistician named Ken Butler.

This method is based on a statistical technique called logistic regression, in essence meaning that teams’ ratings are determined directly from their won-loss records against one another. A key feature of KRACH is that strength of schedule is calculated directly from the ratings themselves, meaning that KRACH, unlike many ratings (including RPI) cannot easily be distorted by teams with strong records against weak opposition.

The ratings are on an odds scale, so if Team A’s KRACH rating is three times as large as Team B’s, Team A would be expected to amass a winning percentage of .750 and Team B a winning percentage of .250 if it played each other enough times. The correct ratings are defined such that the "expected" winning percentage for a team in the games it’s already played is equal to its "actual" winning percentage.

An alternative definition of a team’s KRACH rating is as the product of its Winning Ratio (winning percentage divided by one minus winning percentage) with the weighted average of its opponents’ KRACH ratings. (The definition of the weighting factor makes this equivalent to the first definition of the KRACH ratings.) In addition to KRACH and RRWP, the table above lists each team’s Winning Percentage, Winning Ratio and Strength of Schedule (the aforementioned weighted average of their opponents’ KRACH ratings).


KRACH is provided for entertainment purposes only and is not used in any official way, nor is it endorsed by USCHO.com.
From the Hub:
http://www.mngirlshockeyhub.com/page/sh ... h-rankings
http://www.bgoski.com
About KRACH:
=================
KRACH is short for “Ken's Rating for American College Hockey.” Ken is Ken Butler, a statistician, and the mathematical model he used is known as the Bradley-Terry Rating System. The system and its details are well documented in great mathematical detail online (see KRACH explanation at Ken Butler's homepage).

..
The KRACH rating system is an attempt to combine the performance of each team with the strength of the opposition against which that performance was achieved, and to summarize the result as one number, a "rating", for each team. The higher the rating, the better the team.

This system accounts for strength-of-schedule (SOS) as it ranks the teams.

Teams with equal records and differing SOS can quickly be ranked as a result (i.e. team with stronger SOS is ranked higher than other teams with same record and weaker SOS).

The ratings are calculated purely from the game results (win, loss or tie), and do not use the goals scored at all. Overtime wins count as wins. Ties, for KRACH's purposes, count as half a win and half a loss.

KRACH only counts games against Minnesota State High School League opponents.
..
Mitch Hawker
Site Admin
Posts: 370
Joined: Mon Feb 25, 2002 12:16 pm

Post by Mitch Hawker »

The KRACH system is preferable to RPI and many have proposed replacing RPI with KRACH as a component of the NCAA's Pairwise Rankings.

KRACH is, by mathematical definition, the best ordering of teams to explain the most past game outcomes.

Straight KRACH does have the disadvantage of not considering goal differential. For predicting future games, KRACH has proven inferior to other systems that do consider goal differential.

Due to KRACH's repeated prediction shortcomings, Jeff Rochon (a math teacher and high school hockey coach in Wisconsin) developed a version of KRACH (he called it STAK) that does consider goal differential. His rankings proved very comparable with USHSHO (and Pagestat) rankings at predicting the future outcomes of high school hockey games. Anyone interested would have to contact him for implementation specifics.
Post Reply