In a saucepan, over medium-high heat, bring the plums, blackberries, blueberries, half the raspberries, the sugar, and 1/4 cup water to a simmer. Cook, stirring occasionally, until the blueberries begin to burst, about 5 minutes. Remove from heat. Stir in the lemon zest and juice and the remaining raspberries. Transfer to a bowl and let cool for 15 minutes.
Line an 8-inch square baking dish with 2 sheets of plastic wrap, allowing at least 6 inches to hang over each side. Place the bottoms of the buns in the dish, cut-side up. Spoon half the fruit mixture over the buns. Top with the remaining buns, cut-side up, and the remaining fruit mixture. Cover with the overhanging plastic wrap, a plate, and two 15-ounce cans. Refrigerate until chilled, at least 1 1/2 hours. Spoon the pudding into bowls and top with whipped cream (if using).