Place nuts in an 8- or 9-inch-wide pan. Bake in a 375° oven until golden under skin, about 10 minutes. Pour nuts onto a towel and rub briskly to remove as much skin as possible. Let nuts cool about 5 minutes, then lift from towel; discard skins. In a food processor or blender, whirl nuts until finely ground.
In a bowl, mix nuts, all-purpose flour, whole-wheat flour, brown sugar, baking powder, baking soda, nutmeg, salt, and pepper.
In another bowl, beat egg to blend with buttermilk and 1/4 cup oil.
Add egg mixture to flour mixture. Stir just to evenly moisten. Scrape batter (it is very thick) into an oiled 5- by 9-inch loaf pan and spread level.
Bake in a 375° oven until bread is browned and just begins to pull from pan sides, and a toothpick inserted into center comes out clean, about 45 minutes (about 40 minutes in a convection oven).
Cool in pan on rack about 15 minutes. Invert loaf onto rack and let cool at least 1 hour. If making ahead, store airtight up to 2 days; freeze to store longer.