The openssl smime command performs various S/MIME operations, though with limited MIME [RFC 2045] support. These notes informed by OpenSSL 0.9.7i as shipped with Mac OS X 10.4.7.
Test or peer-to-peer relationships can use self signed certificates. If many systems need to verify certificates, sign certificates against a Certificate Authority, and ensure client systems possess the root certificate. Client systems managed by other companies or individuals are easier to support with a third-party CA provider, while a in-house CA certificate or one from cacert.org can support systems managed in-house at lower cost.
AS2 [RFC 4130] makes use of S/MIME, though commercial AS2 software should handle the MIME operations transparently, once the keys and certificates have been setup properly.
Always check X.509 certificates for expiration, as expired certificates may prevent communication.
Sign, Encrypt; Decrypt, Verify
openssl smime expects certificates in PEM format, not DER or NET. The sender can sign, encrypt, or sign and then encrypt data. Any number of transport mechanisms (SMTP, HTTP, carrier pigeon [RFC 1149]) may then transfer the S/MIME data to the recipient. The recipient will then verify, decrypt, or decrypt and then verify the S/MIME to restore the raw data and ensure the data has not been corrupted or altered. Decryption is mandatory, while verification is optional (but highly recommended).
The openssl smime has limited support for MIME: “The current code can only handle S/MIME v2 messages, the more complex S/MIME v3 structures may cause parsing errors.” — smime(1). Data may need MIME encoding prior to being signed or encrypted, and MIME structures removed following decryption or verification.
Another problem: older versions of openssl smime -verify corrupt output data, by introducing periodic \r\n linefeeds. This breaks data sensitive to additions of linefeeds, though may not affect e-mail significantly. Be sure to test decryption and verification against very large files, not just short Hello World content.
MIME Handling
Use mime-util to unpack the raw data output from openssl smime -decrypt or openssl smime -verify. This utility strips away any MIME headers, and includes test scripts to generate S/MIME encrypted data.
Verify & Self-Signed Certificates
If self-signed certificates are used, the -noverify option may be necessary. This option has nothing to do with the -verify option; -noverify concerns certificate chain verification, while -verify PKCS signatures. Yes, this is a bad naming convention.
