What pass encription does it use?

PunBB uses the pun_hash function for its encryption. The function determines if the server supports SHA1 and, if so, uses it, falling back on MD5 if SHA1 isn't supported. You could edit the function to only use MD5.

Or, where can I modify it?

Ok.
I changed it:

function pun_hash($str)
{
    /*if (function_exists('sha1'))    // Only in PHP 4.3.0+
        return sha1($str);
    else if (function_exists('mhash'))    // Only if Mhash library is loaded
        return bin2hex(mhash(MHASH_SHA1, $str));
    else
        return md5($str);*/
    return md5($str);    
}

THANKS!

I would use SHA1 as well. PunBB 1.3 will use SHA1 only.

login.php already converts from MD5 to SHA1 if the server allows it. Assuming Rickard doesn't remove that code, the passwords will be updated when people log in

Alternatively, the updating script could do it (or everyone could just have to reset their password)

Yes, the automatic conversion will continue in 1.3. The only difference is that pun_hash() will always run sha1() (because PunBB 1.3 requires PHP 4.3.0 or later which has sha1 support)..