Why

When a project sends mail through its own SMTP server and that server fails, the Mails worker throws. The queue then puts the job on the failed list (utopia-queue.failed.v1-mails), as if the worker had broken.

The cause is on the customer's side. The SMTP server rejects the login, or the host can't be reached. A retry would hit the same server with the same credentials, and nothing retries this list automatically anyway. The failed list only grows, and it buries real failures. It also keeps the Queue has failed jobs alert firing on fra1 mails around the clock.

Production, last 24h:

A 6h sample of the fra1 project-SMTP errors:

No SMTP host answered: smtp.mailersend.net:587 (Expected 220, the server said: 554 Too many failed …)   132
No SMTP host answered: smtp.mailersend.net:587 (Authentication failed: 535 Too many failed …)          76
No SMTP host answered: smtp.mailersend.net:587 (Authentication failed: 535 Authentication …)           21
530 Error: authentication Required                                                                      10
No SMTP host answered: s102.cyber-folks.pl:465 (Cannot reach …)                                          7
No SMTP host answered: smtp.stackmail.com:587 (Authentication failed: 535 Incorrect authen…)            7
No SMTP host answered: smtp.protonmail.ch:587 (Authentication failed: 535 5.7.8 …)                      5

Only project SMTP is affected

$type is smtp only when the payload carries an smtp config, and every publisher fills that from the project's own settings:

• Account, MFA, membership and session-alert mails build smtp from $project->getAttribute('smtp'), and only when enabled is true. Otherwise it stays [] and the mail goes through _APP_SMTP_HOST, which is our cloud SMTP and still throws.
• Project/SMTP/Tests/Create (the console's "send test email") uses the host and credentials the user entered.
• Cloud's Mails override doesn't touch smtp.

In production, all 649 smtp failures in the last 24h came from 15 customer projects. One project with locked-out mailersend credentials accounted for 557 of them. None came from console. The single cloud failure still throws.

What

The worker already told the two cases apart: project SMTP threw with code 401 and cloud SMTP with 500. Project SMTP failures now record the error on the span and finish the job, the same way #13831 handles mail no provider can deliver. Cloud SMTP failures still throw. Those are ours, and retrying them makes sense.

} catch (\Throwable $error) {
    Span::add('mail.status', 'failure');

    if ($type === 'smtp') {
        Span::add('mail.error.message', $error->getMessage());

        return;
    }

    throw new Exception('Error sending mail: ' . $error->getMessage(), 500);
}

The failure stays visible: the span keeps mail.status: failure, mail.smtp_type: smtp and the error message.

Test

testProjectSmtpFailureIsNotRetried sends through a project SMTP server on a closed local port (127.0.0.1:1). Without the fix it throws Error sending mail: No SMTP host answered: 127.0.0.1:1 (… Connection refused), the same shape as production. With the fix the job finishes, and the cloud adapter is never used. The existing cloud-failure tests still expect a throw.

• vendor/bin/phpunit tests/unit/Platform/Workers/MailsTest.php: 6 tests pass
• composer lint, PHPStan and Rector (tests/tools/rector.php) pass on the changed files

Follow-up

The functions worker does the same thing for customer runtime errors: execution timeouts, startup crashes and memory kills. That needs its own PR.

Related: #14140