Issues with sending high volume outbound emails

[Thunderbottom]

We have been trying out maddy as a replacement for a couple other mailservers for sending newsletters at our organization (nothing really critical). We have around 1.2mn emails that we send out every weekend to our subscribers, and can see that the server OOMs after sending a few mails, and we've had to restart the server a couple times so far. And after the restart, maddy doesn't listen on the submission :587 port at all anymore (until the remote queue is emptied, I guess?), and there are quite a lot of emails still pending to be sent, which maddy processes slowly.

Right now, maddy's the only binary running on the server with 8 cores and 64GB RAM, and we've been unable to really get it to work. While we also understand that maddy isn't really meant for production loads, is there a way we could make it work, or even optimise our configuration for such loads at all? I'm sorry if this is the wrong kind of question to ask since maddy's meant for smaller-scale deployments, but so far maddy looks like a great project and we are willing to contribute in any way possible to make it better.

Here's the current configuration:

# ----------------------------------------------------------------------------
# Base variables

$(hostname) = host.company.com
$(primary_domain) = host.company.com
$(local_domains) = $(primary_domain)

tls file /etc/maddy/certs/$(hostname)/fullchain.pem /etc/maddy/certs/$(hostname)/privkey.pem

# ----------------------------------------------------------------------------
# Local storage & authentication

# pass_table provides local hashed passwords storage for authentication of
# users. It can be configured to use any "table" module, in default
# configuration a table in SQLite DB is used.
# Table can be replaced to use e.g. a file for passwords. Or pass_table module
# can be replaced altogether to use some external source of credentials (e.g.
# PAM, /etc/shadow file).
#
# If table module supports it (sql_table does) - credentials can be managed
# using 'maddy creds' command.

auth.pass_table local_authdb {
    table sql_table {
        driver sqlite3
        dsn credentials.db
        table_name passwords
    }
}

# imapsql module stores all indexes and metadata necessary for IMAP using a
# relational database. It is used by IMAP endpoint for mailbox access and
# also by SMTP & Submission endpoints for delivery of local messages.
#
# IMAP accounts, mailboxes and all message metadata can be inspected using
# imap-* subcommands of maddy.

storage.imapsql local_mailboxes {
    driver sqlite3
    dsn imapsql.db
}

# ----------------------------------------------------------------------------
# SMTP endpoints + message routing

hostname $(hostname)

table.chain local_rewrites {
    optional_step regexp "(.+)\+(.+)@(.+)" "$1@$3"
    optional_step static {
        entry postmaster postmaster@$(primary_domain)
    }
    optional_step file /etc/maddy/aliases
}

msgpipeline local_routing {
    # Insert handling for special-purpose local domains here.
    # e.g.
    # destination lists.example.org {
    #     deliver_to lmtp tcp://127.0.0.1:8024
    # }

    destination postmaster $(local_domains) {
        modify {
            replace_rcpt &local_rewrites
        }

        deliver_to &local_mailboxes
    }

    default_destination {
        reject 550 5.1.1 "User doesn't exist"
    }
}

smtp tcp://0.0.0.0:25 {
    limits {
        # Up to 20 msgs/sec across max. 10 SMTP connections.
        all rate 20 1s
        all concurrency 10
    }

    dmarc yes
    check {
        require_mx_record
        dkim
        spf
    }

    source $(local_domains) {
        reject 501 5.1.8 "Use Submission for outgoing SMTP"
    }
    default_source {
        destination postmaster $(local_domains) {
            deliver_to &local_routing
        }
        default_destination {
            reject 550 5.1.1 "User doesn't exist"
        }
    }
}

submission tcp://0.0.0.0:587 {
    limits {
        # Up to 50 msgs/sec across any amount of SMTP connections.
        all rate 300 1s
    }

    auth &local_authdb

    source $(local_domains) {
        check {
            authorize_sender {
                prepare_email &local_rewrites
                user_to_email identity
            }
        }

        destination postmaster $(local_domains) {
            deliver_to &local_routing
        }
        default_destination {
            modify {
                dkim $(primary_domain) $(local_domains) default
            }
            deliver_to &remote_queue
        }
    }
    default_source {
        reject 501 5.1.8 "Non-local sender domain"
    }
}

target.remote outbound_delivery {
    limits {
        # Up to 20 msgs/sec across max. 10 SMTP connections
        # for each recipient domain.
        destination rate 20 1s
        destination concurrency 10
    }
    mx_auth {
        dane
        mtasts {
            cache fs
            fs_dir mtasts_cache/
        }
        local_policy {
            min_tls_level encrypted
            min_mx_level none
        }
    }
}

target.queue remote_queue {
    target &outbound_delivery

    autogenerated_msg_domain $(primary_domain)
    bounce {
        destination postmaster $(local_domains) {
            deliver_to &local_routing
        }
        default_destination {
            reject 550 5.0.0 "Refusing to send DSNs to non-local addresses"
        }
    }
}

# ----------------------------------------------------------------------------
# IMAP endpoints

imap tls://0.0.0.0:993 tcp://0.0.0.0:143 {
    auth &local_authdb
    storage &local_mailboxes
}

[foxcpp]

1.2mn looks like a lot of emails. maddy currently has a fairly simple queue implementation - it keeps a small in-memory record for each queued email. The reason the started server is not listening on the submission port is that it needs to initialize the queue first, this includes reading all records into memory.

This issue discusses potential improvements: #262

[cuu508]

As a guess, perhaps the issue is the max_parallelism parameter?

If it is at the default value, perhaps what is happening is that the queue fills up much quicker than it empties out, so the memory usage goes up. I assume queue management operations probably also get slower as the queue size grows.

[Thunderbottom]

This seems to have helped a bit. Although the memory still fills up since the outgoing volume isn't as high as the queuing rate. Will try tweaking the outbound connection rate as well and see if that helps. Thanks a lot!