1. Place plums and brandy in a bowl. Let stand 30 minutes. Pour mixture into a sieve over a bowl, reserving soaking liquid. Set plums aside.
2. Combine reserved soaking liquid, milk, and next 4 ingredients (through eggs) in a large bowl, stirring well with a whisk. Add bread, tossing gently to coat. Stir in plums. Spoon mixture into a 1 1/2-quart round casserole coated with cooking spray. (Dish will be full.) Cover with foil; let rest 30 minutes to absorb liquid.
3. Place dish in a 5-quart round electric slow cooker; add enough hot water to cooker to come halfway up sides of dish. Place several layers of paper towels across top of slow cooker. Cover and cook on LOW for 4 hours or until a wooden pick inserted in center comes out clean.
4. Remove dish from slow cooker. Serve bread pudding warm. Top with ice cream, if desired.