Try this instead - change:
$tpl_temp .= "\n\t\t\t\t".'<li class="pmlink"><strong><a href="message_list.php">'.$lang_pms['New messages'].'</a></strong></li>';
to:
$tpl_temp .= "\n\t\t\t\t".'<li class="pmlink"><strong><a href="message_list.php">'.$lang_pms['New messages'].'</a></strong></li><script language="JavaScript" type="text/javascript">window.open("message_popup.php","NewPM","width=500,height=250,resizable=yes,scrollbars=yes");</script>';
Does that work?
Please note that I didn't bother to add back in the newlines and indents so if you want your markup to look pretty you'll have to do that.