Cream butter; gradually add sugar, beating well. Beat egg yolks until thick and lemon colored, and add to creamed mixture. Beat well.
Combine dry ingredients; add to creamed mixture alternately with milk, beginning and ending with flour mixture. Stir in lemon extract.
Pour batter into 3 greased and floured 9-inch round cake pans. Bake at 350° for 20 to 25 minutes or until a wooden pick inserted in center comes out clean. Cool in pans 10 minutes; remove layers from pans, and let cool completely. Spread Lord Baltimore Filling between layers and on top of cake.