Combine peaches, brown sugar, and cinnamon, tossing well to combine; let stand at room temperature 30 minutes.
Combine granulated sugar, cornstarch, salt, and egg yolks in a medium bowl, stirring with a whisk until smooth.
Heat milks in a medium, heavy saucepan over medium-high heat to 180° or until tiny bubbles form around edge (do not boil). Gradually add hot milk mixture to sugar mixture, stirring constantly with a whisk.
Return the milk mixture to saucepan. Cook mixture over medium heat 2 minutes or until thick and bubbly, stirring constantly. Remove from heat, and stir in vanilla extract.
Spoon the custard into a small bowl. Place the bowl in a large ice-filled bowl for 15 minutes or until the custard is cool, stirring occasionally. Remove bowl from ice.
Arrange 10 of the ladyfinger halves, cut sides up, in a single layer on the bottom of a 1 1/2-quart soufflé or trifle dish. Spoon half of the peach mixture over ladyfingers. Spread half of custard over peach mixture. Arrange 10 ladyfinger halves, standing upright, around side of dish. Arrange remaining 10 ladyfinger halves, cut sides up, on top of custard. Spoon remaining peach mixture over ladyfingers. Spread remaining custard over peach mixture. Cover and refrigerate 4 hours or overnight.