There are two lenses with which to check the visual style of your game. Consider them in this exact order.
- The first is the lens of invitation to play. The marketer calls this user acquisition (horrible naming, as always. :P). The people watch a video or an image showing your game and decide to take a step into your magic circle. Users decide to install your game, using that cold terminology of business. You are investing money to reach your audience so the visual style is very important at this stage. You should consider the devices from which the people will watch your trailers.
- The second step is the realization of the fantasy proposed to the Players. People made the step into the magic circle and became Players. The game makes them a promise and offers a fantasy. If the visuals unmatch their expectation, they can feel something is not OK. For instance, in my case, I have played RPGs my whole life. When I open a modern gacha-based RPG from Asia nowadays, I see boobs and sexy poses everywhere. That hypersexualization makes me step out of the magic circle. Using the boring business language, I will not retain (really business guys? retain? What a terrible word choice, honestly…)
Common visual styles are cartoon, stylized, low poly, and realistic.
Remember: first there is the invitation to play and then the realization of a fantasy. To balance those two things is an art. The art of game design. Especially the marketing and art department are responsible for that. Game designers help their communication.