Combine first 3 ingredients ina large heavy saucepan; whisk in milk. Bring to a boil over medium heat, whisking constantly. Remove mixture from heat.
Beat egg yolks until thick and pale. Gradually stir about one-fourth of hot mixture into yolks; add to the remaining hot mixture, stirring constantly. Cook, stirring constantly, 1 minute. Stir in vanilla.
Layer one-third of wafers in the bottom of a 4-quart baking dish. Stir together bourbon and rum; brush over wafers. Top with one-third of banana. Spoon one-third of custard over banana, and sprinkle with 1/3 cup crushed candy bar. Repeat procedure twice.
Beat whipping cream at medium speed with an electric mixer until foamy; gradually add powdered sugar, beating until soft peaks form. Spread whipped cream over trifle, and sprinkle with remaining crushed candy bar. Cover and chill 3 hours.
Note: We used Skor candy bars.