Combine first 3 ingredients in a saucepan. Gradually whisk in 3 cups milk. Heat to 180° or until tiny bubbles form around edge (do not boil).
Combine remaining 1 cup milk and yolks; stir with a whisk. Gradually add 1 cup hot milk mixture to yolk mixture, whisking constantly. Add egg yolk mixture to pan. Bring to a boil; cook 2 minutes or until thick, stirring constantly. Remove from heat; stir in vanilla. Place pan in a large ice-filled bowl for 20 minutes or until mixture cools, stirring occasionally. Cover surface of pudding with plastic wrap. Refrigerate 2 hours or until chilled. Mash 2 bananas; cut remaining 2 bananas into 1/4-inch-thick slices. Stir mashed and sliced bananas and half of gingersnaps into pudding; chill 30 minutes.
Combine cream and 1 teaspoon sugar in a medium bowl. Beat with a mixer at high speed until stiff peaks form; stir in bourbon, if desired.
Spoon 2/3 cup pudding into each of 8 dessert glasses; divide remaining gingersnaps among servings. Top each serving with 2 tablespoons whipped cream, 1 1/2 teaspoons pecans, and 3/4 teaspoon ginger.