Preheat oven to 375°.
Combine brandy and fruit bits in a small saucepan; bring to a boil. Remove from heat; let stand 1 hour.
Combine milk, butter, vanilla, and nutmeg in a medium saucepan. Heat to 180° or until tiny bubbles form around edge of pan, stirring frequently (do not boil). Remove from heat. Combine 2/3 cup sugar, eggs, and egg yolks in a large bowl. Pour hot milk mixture into egg mixture in a slow, steady stream, stirring constantly with a whisk. Return milk mixture to saucepan over medium-low heat, stirring constantly, until thickened (about 6 minutes).
Combine bread and fruit mixture in a large bowl; pour milk mixture over bread mixture. Transfer bread mixture into an 11 x 7-inch baking dish coated with cooking spray. Place dish in a 13 x 9-inch baking pan; add hot water to larger dish to a depth of 1 inch. Bake at 375° for 45 minutes or until set. Sprinkle with powdered sugar.