I've long secretly thought that a big ingredient in establishing a career in mathematics is playing a waiting game, especially with the job market as it is. I don't mean a passive waiting game -- if you do not do mathematics, teach, serve, etc you will be pushed out (at least if you don't have a tt job yet!). I mean an active game of patience. Just keep working, being stubborn, contributing in the ways you can.
Plenty of people leave academic math and for many reasons: don't want to do the research to get the PhD, can't manage the persistence or independence to do the PhD, discover they'd rather do something else, manage the PhD and realize academia or math or whatever is not for them, get into the postdoc and realize they'd rather have respect and a nice paycheck or want to live in a particular geographic region or have a sick parent and need to stay someplace or can't find a TT job or ..... This happens at many phases of the career path. I have always known I'm not a stellar mathematician but I'm stubborn, patient, and I love math. I figured I could make it through any of the crap that got thrown my way with patience and stubbornness.
I still think that's true (if I can get a job). I think that I'm growing into my research and making original contributions. I am tackling problems that I'm sure others could think of and solve, but these problems just aren't the next big one on their lists. (I have a list of problems -- a story I want to unravel or a mystery I want to illuminate -- and there are great cool problems I've thought of that don't fit my story that I have not pursued. True for you?) I think I could be a small steady light in the sky of mathematics even though I was never a bright star.
I have to say, too, that many people I knew in grad school and identified as bright stars initially ended up flaming out. How often is that true?
(Some mathematicians have said that math is a young man's game. I think that's true for many bright stars. I certainly don't think it's true for steady lights. There are many men in mathematics who start modestly and continue contributing into old age. I use the gender pronoun intentionally: I have observed as well that many women tend to start more modestly and grow brighter and brighter over time. I think this has to do with American culture in high school and college -- fewer women devote themselves wholeheartedly to mathematics in high school or even college, and thus don't start their cosmic mathematical journeys early enough to burn super-bright in the first month of grad school!)