Soften gelatin in water; set aside. Combine sugar and egg yolks in a medium saucepan, stirring well. Gradually add milk, stirring until smooth. Cook over medium heat, stirring constantly, until mixture is slightly thickened. Remove from heat, and stir in softened gelatin and sherry. Chill until consistency of unbeaten egg whites.
Fold chestnuts into chilled custard mixture, and pour into a lightly greased 2 1/2-cup mold. Refrigerate overnight.
Unmold onto a small serving platter, and top with whipped cream, if desired. Spoon into individual serving dishes.