Others have obviously done it. I plan on doing it. There’s no particular reason not to do it, unless it’s raining heavily and you’re not a fan of getting mud all over your jeans.
I feel as if that “no white after Labor Day” rule is there to keep people from wearing traditionally spring/summer fabrics during the fall/winter (ie: linen, seersucker, pincord, etc.), because they’re not really designed for the season. Still, if you live in a climate where it’s sweltering and it’s after Labor Day, I think you’d be foolish to listen to the rule. Dress for the environment, dress for the occasion.