The Background

The Missus and I flew to Florida a couple of days ago, and as usual we took JetBlue. The only eventful part of the flight was a pleasant arrival 30 minutes ahead of schedule. The flight crew had mentioned that the satellite TV was out of commission, and that all in-flight movies would be free for the duration.

I thought that was a good way of handling the issue, and figured that was the end of that.

However, the next day we both received emails from JetBlue stating that we’d been signed up for their Travel Bank, and that a $15 credit had been applied to both of our Banks in exchange for the inconvenience of the malfunctioning TV service! That kind of proactive customer service is fantastic, and one of the main reasons that we fly JetBlue.

Hello SCOTT

Thank you for choosing JetBlue.

The following credit has been applied to your Travel Bank account number: XXXXXXXXXXXXXXXXXX ^{(Ed. Note: Account # redacted)}

Service Credit: InFlight Entertainment 15.00

We‘re sorry that DIRECTV® service didn’t work during your flight—we know this is one of the many reasons our customers choose to fly with JetBlue. Please accept our sincere apologies and this flight credit for the inconvenience you recently experienced with us.

This credit, which expires 365 days from the date it is issued, is available for use on future travel with JetBlue and is non-transferable.

To book a flight using your Travel Bank credit, visit jetblue.com and choose Travel Bank as your form of payment.

You can check the balance and transactions of your Travel Bank account by clicking here. For more information about Travel Bank and your credits, please visit jetblue.com/help/travelbank. We thank you for your understanding and look forward to a future opportunity to welcome you onboard.

Sincerely,

JetBlue Airways

Plain-Text Passwords

Here’s where the story turns dark, at least from a security perspective: Because this Travel Bank was a new service for both of us, and because JetBlue likewise had to create accounts for us, they sent us a password to get started.

Unlike — oh, I don’t know — every other website in the world, they didn’t send us randomly generated passwords. No.

THEY RE-USED OUR OUR TRUE BLUE ACCOUNT PASSWORDS, AND SENT THEM TO US IN PLAIN TEXT.

Click on this image for the full-size version.

This is a big deal for three reasons, the last of which is maybe a little less than obvious to most:

When emails are transmitted across the internet they are generally not encrypted. This means that your password would be visible to any server or router between JetBlue and your email service. It might be stored, intercepted, or otherwise snooped at any point along the journey.

Perhaps more importantly, anyone that gained access to your email account would know your password. That may seem unlikely (after all, you probably don’t have a crack team of international hackers trailing your every move), but anyone that happened across your phone could see the password.

If that doesn’t sound important, then in my opinion this last point is the worst faux pas of all. Wait, it’s not a faux pas. It’s more an act of pure ignorance and/or negligence: It’s obvious that JetBlue is storing your password in plain text, or at the very least with reversible encryption.

This means that, were hackers to get access to JetBlue’s user account database (unlike you, JetBlue may indeed be targeted by their friendly neighborhood team of international hackers), they could see your password.

And you may store your credit card info in your True Blue account. If so, anyone with access to your account could book flights to your card. Moreover, how secure is your card number if your password isn’t properly secured? That’s the kind of question I shouldn’t have to ask of “New York’s favorite airline”.

And I’m betting that you use that same True Blue password for at least one of your other accounts, perhaps even something critical like your banking or credit card accounts.

Change your Password

The moral of this story is that you should change your password. Not just with JetBlue, but make sure you use a different password for JetBlue than any of your other accounts.

Actually, it’s a best practice to use different passwords for each of your online accounts. Realistically that can be a huge pain in the ass, so even I don’t do it 100% of the time.

The moral of the story for JetBlue: For the love of internet security, use non-reversible encryption to protect our account information! You should never be able to know my password. That principle comes straight out of a community college Web Design 101 course, and you’re an international airline!

Bullhorn vs. Exchange 2003

One of the companies for which I manage IT uses Bullhorn’s applicant tracking software for their recruitment workflow.

That company also uses the now-ancient Exchange 2003 for their email.

But, Bullhorn doesn’t officially support integration with Exchange 2003.

What’s involved?

First off, “integration” is a strong word. It implies that our servers will pass information back and forth and stay in some meaningfully synchronized state. That’s not the goal in this case. The integration simply consists of passing all emails that are sent and received by our recruiters to Bullhorn’s servers.

Once Bullhorn receives the emails, they’re parsed and can be viewed in the Activity Center and/or under the contact record to which they apply (using email address matching).

Exchange journaling

The integration path I wanted to take was to use message journaling in Exchange. In essence this forwards a copy of every single message sent and received on our server to Bullhorn. There are other more convoluted options, but this seemed to be the most straightforward from a sysadmin perspective.

Unlike later versions, Exchange 2003 doesn’t support journaling rules. Those would have allowed me to journal messages that were to/from a particular user or set of users.

Instead Exchange 2003 supports mailbox store-level journaling. In other words, every email to/from a mailbox on a particular store is “forwarded”. (It’s technically different than forwarding, but is conceptually similar enough for this discussion, so I’ll drop the quotes going forward.)

This isn’t a problem in my case; Our organization is relatively small, and so it was trivial to move all of my Bullhorn users to one mailbox store, and move some other users out.

Resources

I’m not going to go into depth regarding setting up journaling because there are quite a few good resources available on the subject:

MSExchange.org – Implementing Exchange 2003 Message Journaling

Microsoft Technet – How to Deploy Exchange Server 2003 Journaling as Part of a Compliance Solution

Bullhorn – Enabling the Bullhorn Email and Calendar Integration with Microsoft Exchange [2007 through 2013]

Google around; You’ll find a lot more information.

Create the contacts

The short of it is that you first need to obtain from Bullhorn your tracker address. It looks something like this:

<corpname><.corpID>@slXtracker.bullhornstaffing.com

You’ll also need tracker addresses for each of your Bullhorn users. Those look something like this:

<username>@slX.bullhornstaffing.com

In Active Directory Users and Computers create a contact for each of your Bullhorn tracker addresses. For sanity’s sake I’ll make up some examples.

Let’s say our corporate tracking address is:

recruitfast.1234@sl1.bullhornstaffing.com

And we have two Bullhorn users, each with these tracker addresses:

Malcolm Reynolds – mreynolds.rec@sl1.bullhornstaffing.com
Zoe Washburne – zwashburne.rec@sl1.bullhornstaffing.com

Here’s what it should look like when you create the corporate tracker contact:

Of course you can give the contact a first and last name that you find to be appropriate. But that works for me!

Modify the “E-mail” field, and add a new “SMTP Address” using your corporate tracker email.

Once the contact is created, go back in and edit it. Under the “Exchange Advanced” tab, check the box next to “Hide from Exchange address lists”. This will help prevent users from sending email directly to this address.

There are more advanced ways to prevent users from emailing an internal recipient. Since the risk of that happening is low and I like to have the ability to run tests later by manually sending emails, I haven’t bothered.

Now create contact records for Malcolm and Zoe. It’s the same process, though I would name them in this fashion:

Malcolm Reynolds (Bullhorn Tracking)
Zoe Washburne (Bullhorn Tracking)

By adding “(Bullhorn Tracking)” to their last name (or just to the display name) there won’t be any confusion in the future as to whether you’re looking at their user account or their contact record.

I’ve been frustrated by this myself: Depending upon your Active Directory and Exchange environment, it may take some time for the contact records to propagate to your Exchange server(s). Be patient if you don’t see them right away.

Enable Journaling

I’m assuming that at this point you have all of your Bullhorn users on one mailbox store. In my example, it’s going to be “Mailbox Store 2″.

In Exchange System Manager (ESM), find the store and open the properties.

Click on the image for a larger version.

Check the box next to “Archive all messages sent or received by mailboxes on this store”. It doesn’t say “journaling” — why make it easy? — but in this context the “archive” is the journal.

Click “Browse” and locate your “Bullhorn Tracker (Corporate)” contact.

Apologies that in this screenshot I left off the “(Corporate)” part. It’s the same contact.

Hit “OK” to set the contact, and then “OK” to save the mailbox store properties.

Congratulations! You’ve now set up journaling! (Pretty easy, huh?)

But what about envelope journaling?

Exchange 2007 and above only do envelope journaling. Exchange 2003 does not have the ability to do envelope journaling by default, and there is no way to enable it in the GUI. So is that why Bullhorn doesn’t work with Exchange 2003?

Apparently not.

The article from MSExchange.org that I posted above talks all about configuring envelope journaling in 2003. I found that it makes no difference to Bullhorn whether it’s enabled or not. Hence I don’t think that there’s a point to configuring it. (Though it couldn’t hurt, and it’s really easy!)

Incidentally, I also found a few articles relating to email journaling to a third-party compliance solution. One of them discussed that Exchange 2003 does not actually forward the full envelope over SMTP even with envelope journaling enabled. Their suggestion was to create a local journaling mailbox, then to enable user-level forwarding for that mailbox to the third-party receiver. I found that did not fix the situation either.

So are we done?

You’d think that would be that. We want to track both incoming and outgoing emails in Bullhorn, and Exchange 2003 is now forwarding both incoming and outgoing emails to your Bullhorn tracking address.

But here’s the rub: It doesn’t work for inbound emails, and since Bullhorn won’t talk to me about my integration because I use Exchange 2003 I have no idea why it doesn’t work. It should work, but it doesn’t.

Email forwarding to the rescue

To get inbound emails to show up in Bullhorn I had to configure email forwarding at the user account level in addition to journaling!

If you’re the usual Exchange administrator then this should be a cakewalk for you. Let’s skip right to the screenshot:

As usual, click on the image for a full-sized version.

Open up the trusty old Active Directory Users and Computers management console.

Locate the Active Directory user account for your Bullhorn user and open its properties.

Click on the “Exchange General” tab, and then click the “Delivery Options..” button.

In the delivery options dialog, click the radio button next to “Forward to:”.

Click the “Modify” button and find the contact for your Bullhorn user. In this case it’s “Malcolm Reynolds (Bullhorn Tracking)”.

Check the box next to “Deliver messages to both forwarding address and mailbox”. (That’s an easy one to forget!)

Click “OK” to save and close the delivery options, and “OK” to save and close the user’s properties.

And that’s it!

You should now see emails flowing into your Bullhorn Activity Center and/or to your candidates’ and contacts’ activity log under “Email”:

Sorry for all the redactions!

Here you can see that both inbound and outbound emails are associated with my candidate record!

The eagle-eyed reader may have noticed that the first email in the list has a timestamp of 1139 (11:39 AM) in the subject field, but the “Date Sent” column says 6:36 PM. This is not an issue with the integration; For some reason the Bullhorn website was extremely slow today to the point of being unusable. I’m assuming that’s the reason that the email didn’t show up in our account for 7 hours.

Testing

I created a candidate record in the system under my name, and gave it two of my personal email addresses.

I’m not a Bullhorn user, but I hopped onto one of our users’ email accounts and sent a test email (or two) to my personal address.

I also sent an email (or eight) to the Bullhorn user’s corporate email address.

Because my personal email address matched an email address on my candidate record, those emails show up in the “Activity” tab under “Email”!

If they don’t show up under a candidate or contact record, they’re probably in the Activity Center (under the main menu).

If you aren’t seeing any messages in Bullhorn, try journaling messages to an external email account that you control.

For example, if your personal email address is butterface1990@gmail.com then set up an Exchange contact just as you did with the Bullhorn tracking contact. Change your mailbox store’s journal settings to use that contact.

You should start seeing messages in that GMail account. If no messages come through then the problem is likely on your server(s) and/or network and not with Bullhorn. (As always, give it some time for that journaling change to propagate through your Active Directory and Exchange servers. Oh, and check your spam folder!)

Caveats

At some point we may start seeing duplicate emails in the system. Bullhorn appears to be ignoring inbound emails that are journaled to the corporate tracking email address. That behavior may later be fixed on Bullhorn’s end, in which case we may see the situation where both the user-level forwarded emails and the inbound journaled emails show up in our account!

No official Bullhorn support. I’ve done my best to get this integration to work. If it breaks down the line then we may just be plain out of luck.

This is not an ideal situation for a large organization. We have 3 Bullhorn users, and likely won’t be increasing that number tremendously over the next few years.

If your organization is large (that’s a subjective term, I know) then you may wind up putting too much load on your server and/or network by journaling and forwarding messages for a lot of users. You may also run out of mailbox stores! (They have size and quantity limits.)

In conclusion

Yes, integration with Exchange should work with just jounaling in place and without the need to forward emails at the user level.

And believe me, I tried every which way to get inbound journaled messages to show up in Bullhorn! Feel free to try some other solutions yourself (and if you succeed let me know), but I’m hoping that this article saves you the time. It cost me a lot of mine!

Of course, we haven’t even talked about calendar integration. That’ll be a post for another day, perhaps.

In most cases, these mail servers are Postfix with MySQL providing virtual alias maps, transport maps, relay domains, and virtual alias domains. Unfortunately the Postfix+MySQL implementation isn’t always 100% great. On very rare occasions the Postfix instance may fail to communicate with the MySQL server, for any number of reasons.

From the perspective of the sender’s MX, this usually results in a 550 status code (often given as “Relay access denied”). This is a hard-fail, in that it tells the upstream MX that the recipient they’re trying to reach is permanently unavailable. The upstream MX then gives up on delivery and sends a bounce notification to the sender. This behavior is highly undesirable for the purposes of redundancy, because the message will never be retried at another one of my MX servers (which is probably working just fine)!

The solution is to tell Postfix to soft-fail in the case of any undeliverable mail. For example, I have the following settings in my main.cf:

access_map_reject_code = 450
maps_rbl_reject_code = 450
reject_code = 450
relay_domains_reject_code = 450
unknown_local_recipient_reject_code = 450
unknown_relay_recipient_reject_code = 450
unknown_virtual_alias_reject_code = 450
unknown_virtual_mailbox_reject_code = 450

A 4xx code is generally interpreted by mail exchangers as a temporary condition, encouraging them to retry delivery at a later time. Most mail exchangers are smart enough to then try other MX records in DNS for delivery.

However, there are downsides to a soft failure (4xx code).

First, you’ll see a whole lot of retries for messages that have truly invalid recipients. In fact, with a 99.999% MX uptime, the vast majority of retries will be for spam messages, mis-typed addresses, spammer bounces, etc. This will result in a lot of extra traffic and a lot of extra load on mail servers. Of course, it will also result in lots of deliveries of valid messages in the case of an outage on one MX!

Second, in the case of a mis-typed address (or other case where the mail is really not deliverable) it could take over a week for the sender to get a bounce back (undeliverable notification). In the mean time they would probably assume that their message went through, and that the recipient was just ignoring them!

So what’s the solution?

First, let’s look at what a properly-configured (read: non-spammer) mail exchanger will do when trying to deliver a message:

1. Obtain from DNS the MX records for the destination domain.
2. Attempt delivery to the MX with the lowest number in its priority field.
3. If delivery doesn’t succeed (generally because the server could not be contacted or the server gave a soft failure) then:
   • Go back to step #2 using the next-highest priority number, or cycle back to the lowest number if there is no higher number.

That’s a simplification, but the idea is that delivery will be tried to each successive MX server in a loop until delivery succeeds or a hard-fail status is received.

The best solution that I’ve found is to set up the last MX server (the one with the highest priority number in DNS) to hard-fail with 5xx status codes, while the others are set to soft-fail with 4xx status codes.

Spammers will generally attempt delivery to any arbitrary MX server and will not obey the priority order in DNS. So there will be some unnecessary retries. But this will cut down on traffic from accidental delivery attempts (e.g. mis-typed addresses), as well as from bounce notifications from valid MX servers (e.g. if a spammer is using one of your users’ email addresses to send mail).

Valid senders with invalid recipients will receive a timely bounce back, though it may take a few minutes depending upon the timeouts in the upstream MX.

Most importantly, delivery of valid and desirable messages will be retried across MX servers until a functional one is found.

Actually, my setup is a bit different: I host incoming mail services for many domains, and want to keep traffic flowing equally across all of my mail exchangers. As such, each domain is assigned three of my MX servers in random order. That means that there is no MX server that will always have the highest priority number in DNS, and so all of those servers are set up to soft-fail.

I think you can see where I’m going: I created another fall-back server which is assigned to users’ domains as a fourth MX record with the highest priority number. This server is set up identically to all the other MX servers, except that it will reply with a 5xx status codes in cases where the other servers would give 4xx codes. For almost all valid (non-spam) email, this server will be a last resort.

I say “almost all” valid email because there is some MX software out there that will try DNS MX records in an arbitrary order. However this behavior is not that common, and the chances that delivery would be first attempted to the hard-fail server while that server was down is slim: It would effect a very low number of messages, and so is an acceptable risk.

If you’ve been working with mail servers and SMTP for any length of time, then you know that mail delivery is never as simple as it seems. There are trade-offs and “gotchas” galore when implementing and administering a mail infrastructure. To my mind, the #1 goal of any admin should be ensuring successful mail delivery. Every trade-off should be in furtherance of that result.

It’s like the old saying: Better that 10 guilty persons escape than one innocent suffer.

Better that 10 spam messages get delivered than one important message bounce.

I administrate roughly fifteen domains that send email on a regular basis.   Outbound email is handled by two corporate (and one personal) email servers running Zimbra and Exchange, as well as a couple of mail exchangers that handle automated email from web servers.

I also don’t send spam.   All automated emails include a clear unsubscribe link, which is a single-click mechanism resulting in an immediate blacklisting of the user’s email address.  Automated emails also include the name and mailing address of the company from which they were sent, as per US federal law.  Corporate and personal emails are used responsibly;  In other words they are not used for blind solicitation nor for any other purposes that would be considered spam.

All email, automated or not, is sent with valid DKIM and SPF headers from static IP addresses registered with ARIN under an appropriate corporate entity.  Reverse DNS matches the sender domain in some cases, but not in others as we have very small IP ranges.

Despite this correct and responsible use of email, customers, vendors and personal contacts still need to be sometimes told to “check your spam folder”.

Of course the big providers like Google, Microsoft and Yahoo!  also do things in a technically correct fashion, but because of their large subscriber bases they receive special treatment by anti-spam filters.  They also have technical teams that can chase down and correct IP and other blacklist issues.

Medium- and large-sized organizations can simply pay another company like Return Path to take care of these issues and improve their reputation with the makers of spam filters.

But what’s the little guy with a limited budget to do, and what to Bitcoins have to do with it?

A while back I was very interested in Bitcoins.   I never really got into mining, and I haven’t done any significant transactions using the digital currency.  However the algorithm of mining Bitcoins and the inherent security is simply elegant, despite its necessarily brute-force nature.

I imagine that such an approach could be used towards email authentication.

I’m not a cryptography expert (nor even nearly so), but I propose that there be a new score for email reputation that’s directly related to the computational power that went into sending the message.   Perhaps, to borrow from Bitcoin mining, the score would be proportional to the value of the SHA256 hash that signed the message.  In other words, the more computational work that went into finding the lowest possible hash value, the better the message would score as ham.

Needless to say, it would be prohibitively expensive (computationally and therefore fiscally) for spammers to send out millions of emails using such a mechanism, and that’s rather the point.

This approach would likewise be impractical for large organizations and legitimate bulk senders, but they can already afford their good reputations.   So I’m not suggesting that we supplant all spam scoring with this approach, but merely that it become one more tool in the box, but one that specifically benefits smaller organizations and individuals with personal email servers.

The implementation should be straight forward, and should not be computationally taxing on the receiver of the message.

My first thought is to leverage DKIM.  Imagine that an email contains the headers shown below, the first being a standard DKIM header (from my personal email), with the second a new header.  I’ve put the initialism MSMR in the header name as a placeholder.  It is short for:  Message Signature Mining Reputation.  Perhaps not the best name in the world, but it will do for now.

DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=rosenber.gs;
    s=default; t=1348858150;
    bh=TUWh0I0EJ9E4V2T3OjQaYKWWoMdOzHN8R8m+SrSZG6M=;
    h=Date:From:To:Message-ID:Subject:MIME-Version:Content-Type;
    b=PCy4cbsWMeTWfcHO2Xk9wkXpvKp6vgXCfbm1dVoopDBOTnybRZL88PqrQ6LISkMDT
        8tmS24ETegBqav81dGAF4YdkzE9DNj623JPB0/lsJsDCPXY8yWTSznG9B0txzs9wJN
        msqKfx5ImU/w5npzv5M28g88H6mYKRzloJlcHpFo=

MSMR-Key: v=1; a=rsa-sha256; n=ba4f33
    h=00000000000f15dbc2960b082dbf914e97198c8b12abc542a15f74a2a0fab1d4

In the MSMR-Key header, key/value pairs are used in the same way as in the DKIM-Signature header.  v and a are likewise similar to those in DKIM to respectively indicate the version of MSMR and the hashing algorithm used.

The tag n holds the nonce value, an integer that is incremented for each attempt to generate a low-valued h, the hash of the DKIM-Signature’s b (base64-encoded signature) field appended to the nonce value.

As such, the receiving server could easily validate the MSMR header by hashing the nonce value + the DKIM signature (assuming the DKIM signature has included the To: header), and comparing that hash to the value of the h tag.  If the hashes match, the MSMR header is valid and can be used for scoring the message.

(Of course, the entirety of the message body and the To: field could be hashed instead of the b tag in the DKIM signature.  This could be an optional approach in the implementation.  I chose to leverage the DKIM header because it’s computationally easier to deal with for the recipient than the body of a very large message, and because it enforces that DKIM be used to authenticate the source of the message).

Determining the spam/ham score from the hash value must be a relative process.

Computational power will continue to become less costly over time (from the perspective of hashes/sec/joule and therefore hashes/sec/monetary unit), and so it will become increasingly easier to generate ever lower value hashes on outbound messages.

Therefore the spam/ham score cannot be derived in direct proportion to the numeric value of the hash;  A scaling factor must be used.  That scaling factor must be updated to reflect the state of the art of computing at any given time.

There are three ways to do this, and I propose that all three be put into place.  It should be up to the recipient email server’s spam filter to decide which source of data to use:

• A central authority decides the current scaling factor and makes it available publicly via API and on their website.
• Large ISPs and/or spam filter vendors determine an appropriate scaling factor using real-world metrics (users reporting spam and ham that was erroneously categorized as spam) and makes it available publicly via API and/or on their websites.
• Individual mail servers at small- and medium-sized organizations determine an appropriate scaling factor using their own data, in the same fashion as the large ISPs.

Ideally, no fee would be charged for either of the first two approaches, but the third is guaranteed to be usable without a fee (using open-source software).

Both senders and recipients would need to obtain the scaling factor so that the senders can meet recpients’ expectations of hash values.

To summarize:

• Small organizations and individuals can gain control over their email reputation with insignificant cost and without the intervention of for-profit businesses.
• A brute-force hashing implementation to establish reputation would be easy to implement.  It would put the computational onus on the sender, but the recipient would require very little CPU time to validate the hash.
• Spammers would have a high, real-world monetary cost for sending bulk email, but low-volume senders would suffer a negligible increase in the cost of running their mail servers.
• As average computing power increases, so too can the difficulty of generating low-value hashes.
• The organization or individual sending mail would not need to affiliate themselves with any central authority, nor pay fees to any third party.

In my defense…

I believe, and have always believed, that email should be free (both of cost and censorship, but this is about cost).  I vehemently oppose any government taxes or ISP fees levied on a per-message basis.  I also oppose the cartels of “legitimate” email senders that bombard my inbox with promotional material that they insist I desire merely because I’ve used their services once or twice in the past.

Though my suggested system of scoring email reputation does, in effect, put into place a fairly definitive per-message cost, I do not believe that it amounts to taxation.  This is because 1) it is not a requirement of sending email, 2) the per-message cost will be controllable by the administrator (to some extent), and 3) the messaging cost is not a direct function of arbitrary government or corporate decisions.

Based upon a paper napkin estimate using figures from Bitcoin mining efficiencies, my own infrastructure’s email volume, and power costs in my area, this system would cost me something on the order of US$2/month on the corporate side, and a matter of pennies/month for my personal email.  This is far cheaper and easier than other solutions and/or fighting each and every recipient ISP personally.

I encourage comments, because this is only a draft of an idea I thought up this afternoon.