Update: November 16, 2006
There are two articles in mozillaZine
Getting an SMIME certificate - http://kb.mozillazine.org/Getting_an_SMIME_certificate
Installing an SMIME certificate - http://kb.mozillazine.org/Installing_an_SMIME_certificate
that give information on how to set up certificates in Thunderbird in case you are not using Enigmail http://enigmail.mozdev.org/ instead.
It looks like the instructions may be for Thunderbird 1.0.x, some of them don't work the same in 1.5.x
" ... Thunderbird, go to "Tools -> Options... -> Advanced -> Certificates -> Manage Certificates..." ..."
You actually need to go to Tools | Options | Privacy | Security | View Certificates instead.
The self-signed certificates portion seems to work slightly different also.
After you created the certificate
It does not seem to be possible to export .cer if you selected "My Certificates" in the left pane of Keychain Access. You can only export Personal Information Exchange .p12 file.
You have to choose "Certificates" in the left pane, before you can export .cer file to be used for Certificate Authority import in Thunderbird.
Once all is done,
set master password
You should haved your and other's certificate (their "authorities")
choose the certificate (if you have more than one identities), encryption and digital signature should work.
Under the hood, the digital signed message would have something like this inside
... --------------ms060602080100020308050509 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIHIjCB A40wggJ1oAMCAQICAQEwCwYJKoZIhvcNAQEFMIGKMRYwFAY ...
The encrypted message would be like this
... Content-Type: application/x-pkcs7-mime; name="smime.p7m" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7m" Content-Description: S/MIME Encrypted Message MIAGCSqGSIb3DQEHA6CAMIACAQAxggL/MIIBTgIBADA2MDExEjAQBgNVBAMMCVBvbG8gV29y ZzEbMBkGCSqGSIb3DQEJAQwMcG9sb0B3ZnUuZWR1AgEBMA0GC...
Otherwise you get those invalid or unknow messages