Preheat oven to 350°.
Place the first 4 ingredients in a large bowl, and beat with a mixer at medium speed until well blended (about 3 minutes). Add eggs, 1 at a time, beating well after each addition.
Lightly spoon flour into dry measuring cups; level with a knife. Combine flour and the next 5 ingredients (flour through nutmeg), stirring with a whisk. Add flour mixture and buttermilk alternately to sugar mixture, beginning and ending with flour mixture; beat well after each addition. Stir in the apple and brandy, if desired. Spoon mixture into an 8-inch square baking pan coated with cooking spray.
Bake at 350° for 30 minutes or until a wooden pick inserted in center comes out clean. Cool in pan on a wire rack for 10 minutes. Remove from pan; cool completely on wire rack. Sprinkle with powdered sugar.