ASP.NET MVC 3 - How to make the Remote validator evaluated last

This is my model.

public class MyModel
{
    [Display(Name = "Email Address")]
    [EmailAddress] // my custom validator to check valid email format
    [Remote("IsEmailAddressUsed", "Validation")]
    public string EmailAddress { get; set; }
}

The problem I have is that the IsEmailAddressUsed action gets called even when I enter an inva开发者_高级运维lid format for email address. The custom EmailAddress validator works fine when I remove the Remote one. I want to check against the DB only when the email address is in a correct format. Is there a way to make the Remote validator evaluated last?

The implementation of the custom validator: EmailAddressAttribute.

public class EmailAddressAttribute : RegularExpressionAttribute, IClientValidatable
{        
    public EmailAddressAttribute() :
        base("^[a-z0-9_\\+-]+(\\.[a-z0-9_\\+-]+)*@[a-z0-9]+(\\.[a-z0-9]+)*\\.([a-z]{2,4})$")
    {
    }

    protected override ValidationResult IsValid(object value,
                                                ValidationContext validationContext)
    {
        if (value != null)
        {
            if (!base.IsValid(value))
            {
                return new ValidationResult(
                    FormatErrorMessage(validationContext.DisplayName));
            }
        }

        return ValidationResult.Success;
    }

    public IEnumerable<ModelClientValidationRule> GetClientValidationRules(ModelMetadata metadata, 
                                                                           ControllerContext context)
    {
        var rule = new ModelClientValidationRule
                       {
                           ErrorMessage = FormatErrorMessage(metadata.GetDisplayName())
                       };
        rule.ValidationParameters.Add("pattern", Pattern);
        rule.ValidationType = "emailaddress";
        yield return rule;
    }
}

And the accompanying script:

if ($.validator && $.validator.unobtrusive) 
{
    $.validator.unobtrusive.adapters.addSingleVal("emailaddress", "pattern");

    $.validator.addMethod("emailaddress", function (value, element, pattern) {
        return value.match(pattern);
    });
}

Thanks!