Preheat oven to 325°. Whisk together first 3 ingredients in a bowl.
Beat egg yolks with a handheld mixer in a medium bowl at high speed 4 to 5 minutes or until yolks become pale and ribbons form on surface of mixture when beater is lifted. Gradually whisk in sweetened condensed milk mixture, and whisk until thoroughly combined; whisk in buttermilk. Pour mixture into prepared crust.
Bake at 325° for 20 to 25 minutes or until set around edges. (Pie will be slightly jiggly.) Cool on a wire rack 1 hour. Cover pie with lightly greased (with cooking spray) plastic wrap, and freeze 4 to 6 hours.