You need to encode the email according to the email standards (http://tools.ietf.org/html/rfc2822), which if $mailsubject has UTF-8 data would be something like:
Alternatively, if your Vortex version is 2010-09-01 or later, you can print the subject with <fmt "Subject: %W\n" $mailsubject> to print in encoded-word format; this works for both ASCII and UTF-8 $mailsubject.