Whisk together first 5 ingredients in a large pitcher or bowl until well blended. Stir in Simple Syrup to taste. Chill the punch well (about 1 hour). Pour into highball glasses, and grate nutmeg on top, if desired.
Leftover punch freezes well. Even better, if you don't thaw it out completely, you get milk punch slushies!