Tuesday, December 22, 2009

GMail on Engine Yard

Jump to the end of this if you just want the solution I found, the rest of this includes the trials and errors that didn't work, but may help someone find this page.

We have an application on Engine Yard that is configured to use GMail to send email. Everything works fine in development, but in production, we get the following error:

Net::SMTPUnknownError (530 5.7.0 Must issue a STARTTLS command first. 21sm5842339qyk.4
/usr/lib/ruby/1.8/net/smtp.rb:679:in `check_response'
/usr/lib/ruby/1.8/net/smtp.rb:581:in `auth_login'
/usr/lib/ruby/1.8/net/smtp.rb:685:in `critical'
/usr/lib/ruby/1.8/net/smtp.rb:580:in `auth_login'
/usr/lib/ruby/1.8/net/smtp.rb:570:in `__send__'
/usr/lib/ruby/1.8/net/smtp.rb:570:in `authenticate'
/usr/lib/ruby/1.8/net/smtp.rb:410:in `do_start'
/usr/lib/ruby/1.8/net/smtp.rb:377:in `start'
app/models/invitation.rb:13:in `send_invitation_request'
app/controllers/invitations_controller.rb:50:in `create'
app/controllers/invitations_controller.rb:47:in `create'
Turns out - Engine Yard is still using Ruby 1.8.6 and 1.8.6 doesn't support SMTP TLS that Google uses.

So, I started searching on Engine Yard to figure out what to do. First I tried to use the Engine Yard search engine - I looked for "GMail", and this is what I got.

I tried to make these changes (I'm not a Unix expert, by any means), but when I restarted, I didn't see any change. Same error in the projection.log file.

Doing a search on Google for "GMail" brings back a couple posts that seemed more specific to GMail, but they didn't work either. Each change was a redeployment and this was looking like it could take some time.

I then contacted @engineyard on twitter. I ususally get a nice response, and I was passed on to one of the support guys at EngineYard. This is what I was sent by email after describing what I was trying to do:

"The issue is that Net::SMTP doesn't have proper TLS support, so you
can't use it directly with ActionMailer. You have two options. You can
install a plugin for your app to add TLS support following these

Or you can get ssmtp configured on your instance and just set
actionmailer to use :sendmail to send outgoing emails. The latter
option is what our documentation will walk you through. If you're
going to be sending a small volume of email and just want the simplest
setup, using the above link is probably the easiest option.

If you're going to be sending a larger volume of mail (unlikely due to
Gmail's SMTP limits) then I would definitely see about getting either
ssmtp or even a full blown MTA set up on your instance.

Hope that helps."

This is great - I already tried the suggestion from the document, but that didn't work for me so I'll take EY Support's suggestion and get the plugin. Why wasn't that in the documentation?

Anyway, I go to the recommended page and try to the install the plugin that's referenced, but the plugin doesn't install. The page itself is almost 2 years old and the plugin that's mentioned doesn't exist at all. Ugh, apparently EngineYard didn't take the time to make sure it was a good article to refer a customer to.

Back on my own and realizing the plugin approach is recommended, I try searching on google I find the gem on github. OK - still making progress.

I have the option to install the gem or the plugin. I'll do the gem, as I only need it in production.

To do this on EngineYard, you have to add it through their dashboard UI. But, it turns out they already have the gem hosted on their site. Why didn't support tell me about this? Maybe they didn't know it was there. I assume the gems that are hosted are common and put their intentionally, but now I don't really know for sure.

So, I install the gem, and configure production.rb to require the gem.

I redeploy and am now getting this error when I try to send a password reset email:

ArgumentError (wrong number of arguments (2 for 3)):

/usr/lib/ruby/gems/1.8/gems/action_mailer_tls-1.1.3/lib/smtp_tls.rb:8:in `check_auth_args'
/usr/lib/ruby/gems/1.8/gems/action_mailer_tls-1.1.3/lib/smtp_tls.rb:8:in `do_start'
/usr/lib/ruby/1.8/net/smtp.rb:377:in `start'
app/models/user.rb:123:in `deliver_password_reset_instructions!'
app/controllers/password_resets_controller.rb:27:in `create'

Yuck - I didn't add a lot of code to get this error. Doing a little more Googling, I find a thread discussing a similar problem. I get the feeling the Ruby version on EY and this plugin are incompatible.

But one good thing happened here - I stumbled upon another plugin: action_mailer_optional_tls

I search for it and find it on GitHub.

I tell you, whenever I see Collective Idea's name and Brandon Keeper's face, I have confidence things are going to work. I use many other collective idea gems and read Brandon's blog once in a while, this is a good sign. If this works, I'm going to Holland Michigan to register for a class with Idea Foundary.

And, after yet another deployment to EngineYard ...

It worked! Hard to believe after all that.

And all it ultimately took was the following.
  1. Installed action_mailer_optional_tls plugin
  2. Added ''require 'smtp_tls'" to production.rb
  3. Modified the :smtp_settings => { ..., :tls => true, ... }
I wish Engine Yard could have helped me. This took me 5 hours to finally resolve, probably 25 deployments to test changes in production, and it was a trivial change in the end. This is not an unusual experience for me on Engine Yard. Things are often more complicated, hard to understand, or proprietary, forcing me to learn technologies I only use on their site. I am often wishing the UI was simpler, less configuration was required, and the documentation was easier to understand and access. Give me another frustrating afternoon working around an Engine Yard issue and I may be motivated to post some of these experiences, too.

I really appreciate the position that Engine Yard takes - easy to deploy and manage Rails Applications. I saw Tom Mornini talk in Seattle - the vision is great: Engine Yard handles deployment and we (developers) can focus on development and features. It's clear that's where we're headed as an industry, but Engine Yard has plenty of room to improve before it can really own those claims.