Combine sugar, cornstarch, nutmeg, and salt in a saucepan; set aside. Combine milk, vinegar, food coloring, vanilla, and butter; add to sugar mixture, and bring to a boil. Reduce heat, and cook 1 minute, stirring constantly. Add sweet potatoes; cook until thoroughly heated.
Spoon mixture into a lightly greased 10- x 6- x 2-inch baking dish. Roll pastry on a lightly floured surface to fit the top of the baking dish. Place over top of dish, and seal edges. Make slits in top of pastry to allow steam to escape. Bake at 375° for 40 minutes or until golden brown. Cool before serving.