Combine the first 5 ingredients in a medium saucepan. Bring to a boil, partially cover, and cook 8 minutes or until tender. Mash plums. Bring to a boil; cook 2 minutes. Remove from heat; chill.
Combine chilled plum mixture, Crème Anglaise, and buttermilk. Pour mixture into the freezer can of an ice-cream freezer, and freeze according to the manufacturer's instructions. Spoon ice cream into a freezer-safe container; cover and freeze 1 hour or until firm.