Preheat oven to 350° F.
Line the bottoms of two 9 inch cake pans with parchment paper and spray lightly with baking spray. Set aside.
Whisk together the self-rising flour, cinnamon, nutmeg, ginger, baking powder, and salt.
In a separate bowl, add the sugar, brown sugar, and the butter. Using a hand mixer, beat until fluffy scraping down sides as necessary. Add the eggs one at at time mixing well between each. Add the sweet potato and vanilla and continue to mix until until smooth.
Add the flour mixture to the butter mixture alternating with the buttermilk. Begin and end with the flour mixture beating at low speed until combined.
Gently stir in the (optional) chopped pecans.
Pour the sweet potato batter (divided evenly) between the two prepared pans.
Bake until a wooden stick comes out clean, 40 to 50 minutes.
Cool the sweet potato cakes in the pans for 10 minutes. Then carefully remove them from the pans onto a wire rack to cool completely on wire racks.