Most of the times, if your graphics are fine in games, you don't need to update the drivers.
Card manufacturers [ATi, Nvidia] regularly update drivers to fix code issues, but also to fix bugs with games, performance issues, add support for new techniques or technologies, etc.
Unless you have specific issues with drivers it is *ALWAYS* best to update. This is why when someone is having an issue with a game [crashes, poor performance] the very first question is always "What drivers are you using". Typically the number one response from manufacturers is to "make sure you're running the latest drivers.
Not updating drivers is kinda like not changing the oil in your car. Sure, you can *DO* it. . but eventually you're going to start having issues.