Roidzilla 2021 Model Performance

I thought this would be a fun exercise and spark some debate. I tried to pull every model's 12z run from Sunday, January 31st and their forecasted snow maps through Tuesday, February 2nd. I realize some areas were already seeing overrunning snow 12z Sunday, but I figured a 12z snapshot on January 31st would still capture majority of the overrunning plus obviously the entire duration of the main event that took place late Sunday all the way into Tuesday.

First up - RGEM



I felt the RGEM was too aggressive. DC to SNJ to CNJ did not get nearly as much as it predicted, and it did not do well in west-central NJ and east-central PA. Its jackpot zone was too expansive, but it got the 'general' jackpot area correct across NW NJ/NEPA. A little underdone in NE NJ. I expected better from a short range model. Grade: C

Second up - 12KM NAM



Impressive. It predicted the lower amounts in the southern portions of the region and was also good in outlining the jackpot area. It did under-call central Long Island, but I am not sure any model predicted the subsidence over that part of the island. Grade: A. Bravo, bravo!

Third up - 3KM NAM



Ehhh it almost looks like it forgot east and west. For real, the small jackpot area it showed in Monmouth County is nearly the same latitude as Allentown, PA which in reality saw close to 2 feet while Monmouth was mostly below 2 feet. NW NJ did better than NE NJ. I don't know, this is some weird ****. Grade: C

Fourth up - HRDPS



Ma che cazz? I know this is 'Crapola' but I checked and the 10:1 map is very similar to this one. This is the saddest performance I've ever seen. Grade: F

Fifth up - CMC



Also on the aggressive side. Especially toward SE NY and SW CT. Also it seemed to under-call north-central NJ area. I don't think it was as bad as the RGEM, though. Grade: B

Sixth up - Icon



Similar to the 3KM NAM, seems this model could not pinpoint the heaviest axis of snow. At least it's not showing bizarre 40" amounts. Also, what's with that hole toward CT/MA? Grade: D

Seventh up - GFS



What an embarrassing performance less than 12 hours before game time. It reminds me of someone who tried to mix hard liquor after drinking wine for dinner all night. It ended up vomiting all over itself. Fa propia schifo. Grade: F

Eighth up - EURO



I think the EURO went out drinking with the GFS the night before. Ok, it was not as bad as the GFS but a superior model like the EURO should demonstrate much better results than this. The northern cut-off of the heaviest axis of snow is surprising. I pulled the Crapola map just to see if that looks any better:



Nope, not really. Disappointed, honestly. I also feel betrayed. Might need therapy. Grade: D

My overall takeaway is the CMC is better than the GFS/EURO in under 12 hours, and the NAM is by far the superior model in the short range. Then again, this is just one storm. Maybe I should do these more often and see if there is a trend.

Frank_Wx wrote:I thought this would be a fun exercise and spark some debate. I tried to pull every model's 12z run from Sunday, January 31st and their forecasted snow maps through Tuesday, February 2nd. I realize some areas were already seeing overrunning snow 12z Sunday, but I figured a 12z snapshot on January 31st would still capture majority of the overrunning plus obviously the entire duration of the main event that took place late Sunday all the way into Tuesday.

First up - RGEM



I felt the RGEM was too aggressive. DC to SNJ to CNJ did not get nearly as much as it predicted, and it did not do well in west-central NJ and east-central PA. Its jackpot zone was too expansive, but it got the 'general' jackpot area correct across NW NJ/NEPA. A little underdone in NE NJ. I expected better from a short range model. Grade: C

Second up - 12KM NAM



Impressive. It predicted the lower amounts in the southern portions of the region and was also good in outlining the jackpot area. It did under-call central Long Island, but I am not sure any model predicted the subsidence over that part of the island. Grade: A. Bravo, bravo!

Third up - 3KM NAM



Ehhh it almost looks like it forgot east and west. For real, the small jackpot area it showed in Monmouth County is nearly the same latitude as Allentown, PA which in reality saw close to 2 feet while Monmouth was mostly below 2 feet. NW NJ did better than NE NJ. I don't know, this is some weird ****. Grade: C

Fourth up - HRDPS



Ma che cazz? I know this is 'Crapola' but I checked and the 10:1 map is very similar to this one. This is the saddest performance I've ever seen. Grade: F

Fifth up - CMC



Also on the aggressive side. Especially toward SE NY and SW CT. Also it seemed to under-call north-central NJ area. I don't think it was as bad as the RGEM, though. Grade: B

Sixth up - Icon



Similar to the 3KM NAM, seems this model could not pinpoint the heaviest axis of snow. At least it's not showing bizarre 40" amounts. Also, what's with that hole toward CT/MA? Grade: D

Seventh up - GFS



What an embarrassing performance less than 12 hours before game time. It reminds me of someone who tried to mix hard liquor after drinking wine for dinner all night. It ended up vomiting all over itself. Fa propia schifo. Grade: F

Eighth up - EURO



I think the EURO went out drinking with the GFS the night before. Ok, it was not as bad as the GFS but a superior model like the EURO should demonstrate much better results than this. The northern cut-off of the heaviest axis of snow is surprising. I pulled the Crapola map just to see if that looks any better:



Nope, not really. Disappointed, honestly. I also feel betrayed. Might need therapy. Grade: D

My overall takeaway is the CMC is better than the GFS/EURO in under 12 hours, and the NAM is by far the superior model in the short range. Then again, this is just one storm. Maybe I should do these more often and see if there is a trend.

Awesome write up Frank. I also feel the NAM has been doing well for many winter storms so far. It was the first model to bring the current storm back.

Secondly, the RPM model has been abysmal. TV mets really shouldnt show it to the public due to how misleading it has been. Had the previous storm and this storm mainly rain for nearly more than half the event which made me hear from people "it looked like mostly rain on the futurecast..." TERRIBLE MODEL

Great job Frank. I didn’t really do an analysis like this but I remember my off the cuff feeling being that the NAM did pretty good - which your analysis confirms. But also is ironic -and cautionary, because the first couple NAM runs when it first came in range were waaaay off. Basically had no significant snow N&W of NJ and the jackpot too far SE. Then it corrected overnight Saturday. I guess the lesson there is, the NAM is good - but give it some time (a couple runs) to sort itself out once in range.

It has been my anecdotal opinion that ever since the NAM’s last upgrade several years ago now, it has dominated every other piece of guidance within it’s temporal range. THAT SAID, and this gets to Bill’s point, THIS SEASON the NAM has consistently been too progressive in its extended range, only to then correct significantly further northwest inside of 48 hours, largely leading the charge when it does so (and in complete contrast to its ‘usual’ bias of over-amplification). WITH THAT SAID, this anecdotal evidence/opinion cannot be taken as invariably true from year to year or even season to season, as often times each season features a series of seemingly different model biases. Why this is or what causes it, I have no clue lol

This is definitely a fun exercise, though, Frank!!