Make the Cake: Preheat oven to 350°F. Generously coat 2 (8-inch) round cake pans with baking spray. Beat brown sugar, butter, and granulated sugar with a hand mixer on high speed until creamy, about 2 minutes. Add egg and pumpkin; beat well.

Stir together buttermilk and baking soda in a small bowl until dissolved. Stir together baking powder, salt, cinnamon, and 1 3/4 cups of the flour in a separate bowl until combined; add to sugar-butter mixture alternately with buttermilk mixture, beginning and ending with flour mixture, beating on low speed until incorporated after each addition. Toss together pecans and remaining 1/4 cup flour in a separate small bowl; fold into batter.

Pour batter evenly into prepared cake pans. Bake in preheated oven until a wooden pick inserted in centers comes out clean, about 25 minutes. Cool in pans 10 minutes. Remove from pans, and cool completely on a wire rack, about 30 minutes.

Make the Spiced Whipped Cream: Beat cream with an electric mixer fitted with a whisk attachment on high speed cream until foamy, about 1 minute. Gradually add powdered sugar, cinnamon, and ginger, beating on high speed until stiff peaks form, about 1 1/2 minutes. Store in refrigerator up to 30 minutes.

Spread Spiced Whipped Cream between layers and on top of cake. If desired, sprinkle cake with nutmeg.