Combine first 4 ingredients in a large bowl; beat at low speed of an electric mixer 2 minutes or until thickened.
Spoon one-third of pudding into a 3-quart trifle bowl. Cut coconut cake in half lengthwise. Cut cake halves crosswise into 1/2" slices. Arrange one-third of cake slices in a single layer over pudding. Arrange one-third of mandarin orange segments over cake. Arrange half of remaining cake slices over orange segments.
Cover cake with half of remaining pudding; top with remaining cake slices and half of remaining orange segments. Top with remaining pudding and orange segments. Mound whipped topping over trifle. Garnish, if desired. Cover and chill several hours.