Place milk and evaporated milk in a large saucepan. Bring to a simmer over medium heat.
Combine sugar, cinnamon, nutmeg, and eggs in a large bowl. Gradually add hot milk to egg mixture, stirring constantly with a whisk. Return milk mixture to pan; cook over medium-low heat until thick (about 8 minutes), stirring constantly. Pour into a bowl; stir in brandy and vanilla. Press plastic wrap onto surface of eggnog, and chill 8 hours or overnight.