[FIX] mass_mailing: prevent duplicates and sending to opted-out

The mass-mailing App suffered from severe issues due to its inability
to detect and handle duplicates "at the email level", and the absence
of any global blacklist system, leading to lack of user trust.

Mailing-lists typically include multiple records with the same email,
and it is critical to avoid sending them the same email several times.
A related problem is the unsubscription of an email that is present
in other records (duplicates). Opting out the first email should
automatically blacklist it for other records as well.

Ideally we should have a global blacklist table in order to
share the unsubscription requests globally across models (Leads,
Partners, Mailing-list contacts). It would also allow importing
it from other blacklist systems. (TODO for master)

This commit introduces a partial solution, made of several
small changes:

- In mail.mail: double-check that an outgoing email has the
correct status (`outgoing`) before sending it. This allows
adding emails in the queue and cancelling them before they
actually get sent.
- In crm.lead: force predictable recipients for mass-mailing,
by always using the email of the lead rather than the email
of the linked partner when there is one. This simplifies the
computation of the blacklist and seen list. Other areas in
the codebase already assume as much.
- In mass.mailing:

 + Before sending out a mailing-list batch, compute the
   blacklist (all opted-out emails) and the seen_list
   (emails who previously received this mail) to make
   sure we only ever target valid recipients.
 + While delivering the mass-mailing, any email targeting
   an address that is in the blacklist or "seen list" is
   canceled before being sent. The corresponding statistics
   entry is considered "not delivered".
   Also updates the "seen list" continuously.
 + When a mass-mail belongs to a campaign with the
   "unique AB/B testing" flag, the "seen list" is
   common to the whole campaign, as an extra safety.
 + Auto-delete mass-mailing test messages sent with the
   test wizard, to avoid polluting the mail_mail table

Note: this fix uses a simple regex for efficiently extracting
the blacklist in pure SQL from different models, and doing so,
assumes that each record only holds a single email
(no comma-separated adresses). This should be sufficient for
most cases. The regex:

                   ([^ ,;<@]+@[^> ,;]+)

1 files changed, 21 insertions, 7 deletions

diff --git a/addons/mass_mailing/wizard/mail_compose_message.py b/addons/mass_mailing/wizard/mail_compose_message.py
index 82566623d19..c8b1e54f316 100644
--- a/addons/mass_mailing/wizard/mail_compose_message.py
+++ b/addons/mass_mailing/wizard/mail_compose_message.py
@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 # Part of Odoo. See LICENSE file for full copyright and licensing details.
-from odoo import api, fields, models
+from odoo import api, fields, models, tools
 
 
 class MailComposeMessage(models.TransientModel):
@@ -40,14 +40,28 @@ class MailComposeMessage(models.TransientModel):
                         'mailing_model': self.model,
                         'mailing_domain': self.active_domain,
                 })
+            blacklist = self._context.get('mass_mailing_blacklist')
+            seen_list = self._context.get('mass_mailing_seen_list')
             for res_id in res_ids:
-                res[res_id].update({
+                mail_values = res[res_id]
+                recips = tools.email_split(mail_values.get('email_to'))
+                mail_to = recips[0].lower() if recips else False
+                if (blacklist and mail_to in blacklist) or (seen_list and mail_to in seen_list):
+                    # prevent sending to blocked addresses that were included by mistake
+                    mail_values['state'] = 'cancel'
+                elif seen_list is not None:
+                    seen_list.add(mail_to)
+                stat_vals = {
+                    'model': self.model,
+                    'res_id': res_id,
+                    'mass_mailing_id': mass_mailing.id
+                }
+                # propagate exception state to stat when still-born
+                if mail_values.get('state') == 'cancel':
+                    stat_vals['exception'] = fields.Datetime.now()
+                mail_values.update({
                     'mailing_id': mass_mailing.id,
-                    'statistics_ids': [(0, 0, {
-                        'model': self.model,
-                        'res_id': res_id,
-                        'mass_mailing_id': mass_mailing.id,
-                    })],
+                    'statistics_ids': [(0, 0, stat_vals)],
                     # email-mode: keep original message for routing
                     'notification': mass_mailing.reply_to_mode == 'thread',
                     'auto_delete': not mass_mailing.keep_archives,