Yes your machine has more than enough power to play UT3.  And UT3 will work with Vista 64.  I've been running UT3 with Vista 64 since the game came out.

I think this may be a patch problem.  Go to http://files.filefront.com/ut3patch3exe/;11484575;/fileinfo.html and download the 1.3 patch.  Run the exe and it should install the patch.  You need this if you want to play custom maps on servers.

What maps are you trying to install?