The answer is simply, “No.” Just listen to TV or radio and you'll hear a wide range of ages. From young children to elderly. Voices come in all shapes, sizes and ages. It really all depends on what market the client is trying to reach.
On almost all audition spec sheets, the agent will indicate an age range. Remember, the age range is not how old you really are, it’s how old you are supposed to sound. Many times women do little kids' voices because it’s easier to direct an adult in the booth than it is a child. Sometimes people in their 20s will play 50-year-olds, and vice versa.
Just remember, don’t force it or change your voice. Think about what makes someone sound younger or older and try to replicate that by slightly changing your tone and your interp. It takes practice, but if you can voice 18-65, you’ll get more work and have a broader range than if you only stuck to your actual age group.
So never second-guess starting a career in voice over due to age. Age is just a number, and your voice can be multi-generational. Get practicing and find your voice.