The way I did it was to:
A: Make a copy of the data base but saved under another data base name eg: Original database name is forum make copied database name forum 2
B: Install 1.3 to a directory called forum2 and do not attempt to do the install script
C: Copy the config.php from the 1.2 forum and put it in the 1.3 forum directory. The config will have to be edited to reflect the data base location
$db_type = 'mysqli';
$db_host = 'localhost';
$db_name = 'change this to the copied database name (forum2)';
$db_username = 'change this (probably wont have to)';
$db_password = 'change this (probably won't have to)';
$db_prefix = '';
$p_connect = false;
$base_url = 'change this to http://etc etc/forum2';
The forum2 database config will have to modyfied as well the base url in config.
D: Then go to http://your forum2 location (where you installed it to) eg http://yoursite/forum2/ and it in theory will tell you your database needs upgrading from 1.2 to 1.3 follow the prompts.
If that goes well for you do what ya gotta do and once your done doing what you need doing you mass message everyone the forums are going to be down for abit.
I would download a backup of the 1.2 database plus the 1.2 forum folder to your computer for safe keeping. Its a matter of then copying avatars etc from the 1.2 directories and putting them in the right spots in the 1.3 forum2 folder, once your sure you have copyed all you need, then delete the 1.2 folder, rename the 1.3 folder to what the 1.2 was. Then delete the config.php from the newly named 1.3 folder plus whatever is in cache folder. Then upload the saved config from the 1.2.
It should if done correctly when you try to go to the forum url tell you that the forum needs to be updated to 1.3 which you let it do.
It will need some more tweaking for example the extensions will have to be reinstalled. They will exist in the 1.3 forums folder but the database won't have them listed as installed unless you copy the database info from the 1.3 you were messing with when I mentioned to make a copy of the 1.2 and convert it etc. I would personally just reinstall them, they won't need to be redownloaded just reinstalled.
If you used a private message system in 1.2 it is not compatable with the private message system in 1.3. I am currently working on an upgrade script. I have had to manually convert from a 1.2 to a the 1.3 private message system.