Place sweetbreads in a medium mixing bowl; cover with cold water. Set aside to soak for 1 hour. Drain well; remove and discard white membrane.
Place sweetbreads and water to cover in a small Dutch oven. Bring to a boil. Reduce heat, and simmer 30 minutes. Drain well, and rinse with cold water. Cool completely. Cut into bite-size pieces. Set aside.
Melt butter in a heavy saucepan over low heat; add flour, and stir until smooth. Cook 1 minute, stirring constantly. Gradually add half-and-half; cook over medium heat, stirring constantly, until thickened and bubbly. Stir in salt and pepper.
Roll half of pastry to 1/8-inch thickness, and fit into a 9-inch pie plate. Layer oysters and sweetbreads in pastry shell. Pour sauce over oysters and sweetbreads. Cover with top crust, and slit in several places to allow steam to escape; seal and flute edges. Brush with milk. Bake at 350° for 40 minutes or until golden brown.