Combine sweet potatoes, 1/2 cup sugar, cinnamon, ginger, salt, and milk; mix well. Cook until thoroughly heated. Gradually stir about one-fourth of hot mixture into eggs; add to remaining hot mixture, stirring constantly. Spoon into pastry shell. Bake at 350° for 20 minutes. Remove from oven.
Cream butter and 1/2 cup sugar; stir in pecans. Sprinkle pecan mixture over pie; continue baking 45 minutes. Cool. Serve with whipped cream, if desired.