Combine first 11 ingredients in a bowl; stir well. Combine milk, baking soda, and vinegar; add to sweet potato mixture, stirring until dry ingredients are moist.
Press sweet potato mixture into a 6-cup steamed-pudding mold coated with cooking spray. Cover tightly with foil or lid coated with cooking spray.
Place mold on a shallow rack in a stockpot; add boiling water to halfway up sides of mold. Cover and cook in simmering water 3 hours or until knife inserted in center comes out clean, adding water to stockpot as needed. Invert pudding onto a plate; cut into 10 slices. Combine softened ice cream and brandy, stirring with a whisk until blended. Serve with pudding.