brier

Low Brier score when the information is not already there and it has value would be impressive.

Accurate Brier score when things are often already settled and information has little value is not interesting.

(The Brier score is a mean-squared error between outcome (1 or 0) and the percent certainty of the prediction in that outcome.

This simple lazy model achieves a Brier score of 0.039, beating Intrade and the poll average computed in this article quite badly.

No, squared error (the Brier score) is absolutely a strictly proper scoring rule for classification, which is the claim made in the article.

Having previously spent a reasonable amount of time on Metaculus I’m familiar with Brier scores and rating my confidence.

The Brier score was the standard metric for that research.

Best Brier score so far is 0.007 (nearly perfect calibration).

You can get arbitrarily low Brier score with easy questions.

That happens to get the best Brier score.

If you want to go be pedantic about it and select one metric, markets are evaluated on their Brier Score or some other Proper Scoring Rule, not accuracy.

> The book Superforecasting documented that for their best forecasters, rounding off that last percent would reliably reduce Brier scores.

This article offers up a "coin flip" for each state, computing that such a coin flip would have a Brier score of 0.25.

The Brier score and "diagnosis" are shown immediately, no signup needed.

At first I thought the "equal probability assumption" wasn't too bad an approximation, but it has an average Brier score of 0.217.

The Brier score is pathological when the guess is 0.5: regardless of the outcome, it will be equal to 0.25, so if you define "better than random" as having a score < 0.25, actually acting randomly makes you "overconfident".

Brier score 0.0581 12h before markets resolve is usually not impressive score and information gained has usually little value.

Monte Carlo to roll games forward for distributions on weekly win odds and season outcomes, plus basic reliability/Brier/log-loss tracking.

Yeah I'm not buying the last bit about lower MSE with one term in the model vs two (Brier with one outcome category is MSE of the probabilities).