Combine 1 cup sugar and next 3 ingredients in a large saucepan. Scrape seeds from vanilla bean; add seeds and bean to milk mixture. Bring to a boil over high heat. Immediately remove from heat, and let stand 30 minutes. Remove and discard vanilla bean.
Combine graham cracker crumbs and next 4 ingredients in a medium bowl, tossing until well blended. Press crumbs into a 9-inch glass or ceramic pie plate to form an even crust. Bake at 350° for 8 minutes or until firm to the touch; cool completely, and set aside.
Beat egg yolks in a large bowl. Slowly whisk 1 cup warm cream mixture into egg yolks. Add egg mixture to remaining cream mixture, whisking constantly.
Cook cream mixture over medium heat, whisking constantly, 5 minutes or until mixture reaches 160° and coats the back of a spoon. Remove from heat; pour through a fine wire-mesh strainer into a bowl. Cool custard completely, stirring occasionally.
Combine remaining 2/3 cup sugar and Key lime zest and juice in a bowl, stirring until sugar dissolves. Stir lime juice mixture into cooled custard. Transfer custard to an ice cream maker, and process according to manufacturer's directions.
Mound ice cream into prepared crust. Freeze pie 4 hours or overnight. Garnish, if desired.