Well considering that you have to upgrade to 7.0.3 first before upgrading to 7.0.4, it really depends on the type of mods you have on your site. If all the mods you have were simply installed just by going into the Admin Panel and click install, then it should be a breeze for you.
If some of your mods required you to edit the 'core' files, then more than likely its possible that you will have to re-code those modules if the upgrades overwrite those files you edited.
For me personally, D7.0.3 put a hurting on me (took about 2 1/2 days to get everything squared away). The D7.0.4 was very little impact with the same mods. Only took a couple hours.
Remember, ALWAYS do a complete backup before doing any upgrades or customizations. So if in the case something does go wrong, you will always have a backup to put the site right back to the way it was.. Nothing lost :)