Cut potatoes, lemon, and orange into 1/4-inch-thick slices.
Place potato slices in a lightly greased 13- x 9-inch baking dish; arrange lemon and orange slices over potato.
Stir together butter, next 3 ingredients, and, if desired, vanilla. Pour over potato mixture; cover and let stand 1 hour.
Bake at 350° for 1 hour and 30 minutes, basting often with pan juices. Serve with a slotted spoon.