A hat's job changes with what you're asking of it. Some are there to keep rain off and your ears warm when the wind's coming straight off the sea and no amount of hood will do. Some are there to cover second day hair and let you leave the house ten minutes faster. Some are purely about cutting glare on a bright morning so you're not squinting through the school run or a long drive. Different jobs, but the same test applies to all of them: does it actually stay on, and do you reach for it without thinking.
Colour wise, I'd always steer you towards the ones that behave like neutrals even when they're not strictly neutral. Camel, navy, a proper grey, black that hasn't gone slightly green with age, these sit under or over almost anything without an argument. A soft rust or a deep bottle green earns its place too, because they lift a plain coat without looking like you tried too hard. Where people come unstuck is buying the hat in the exact shade of one specific coat, then wondering why it does nothing for anything else they own. Pair a beanie with a wool coat and jeans and it looks like you dressed on purpose. A cap sits happily on top of a plain jumper and trainers and does most of its work there, not over anything fussy. A visor wants simplicity around it, a vest top, a linen shirt, nothing competing for attention. Autumn through to early spring is beanie and cap weather here, proper cold needs something with actual substance to it, not a thin layer that looks the part for a photograph and nothing else. Visors are fair weather only, and look faintly ridiculous the moment it rains, which it will.
The difference between a hat worth keeping and one that's bin material inside a year is almost always in the detail nobody photographs. A beanie should have a decent weight of yarn, enough that it holds its shape rather than slumping by midday, and a brim that doesn't unravel after one wash. A cap wants a stitched peak that doesn't go soft and floppy, proper eyelets, and a strap or closure that won't snap the first time you adjust it in a hurry. Cheap hardware on any of them, a flimsy clasp, a zip that catches, tells you everything about how the rest was made.
I'd start with navy. It goes with almost everything I already own, and I've never once regretted buying it.