[IMP] phone_validation: add blacklist mechanism

PURPOSE

SMS are a powerful marketing tool. For instance it is perfect to announce a
sale or to communicate a coupon code, to welcome a new customer in a fidelity
program, ...

Purpose of this task is to integrate SMS sending in batch in mass mailing. It
will use same mailing objects but sending SMS instead of emails. Some metrics
and flows will have to be slightly updated at the same time.

SPECIFICATIONS

Purpose of this commit is to add a blacklist mechanism for phone numbers
used to send SMS like what already exists for email addresses when sending
emails.

Define a new phone.blacklist model, holding a number and the state of the
blacklist (active field), as well as tools methods to access it. Make it
as private as possible, accessing it in sudo once access are granted.

Also clean phone validation tools: lessen number of tool functions and update
caller to simplify code readability. Some fixes are also included in this
commit, notably blank spaces cleaning in phone numbers.

Improve phone.validation.mixin to add a tool method computing a sanitized
number, in addition to formatting it to national / international.

Define a new mail.thread.phone mixin computing the blacklist status of a
record. This mixin

  * inherit from phone.validation.mixin in order to have access to some
    base phone number parsing capabilities;
  * computes a sanitized phone number based on ´´_phone_get_number_fields´´.
    It takes first sanitized value, trying each field returned by the
    method. That means one sanitized phone number is available per record
    even if several fields are available;
  * compute blacklist state of records. It is based on phone.blacklist
    model and give an easy-to-use field and API to manipulate blacklisted
    records;
  * give some API methods :

    * ``_phone_set_blacklisted``: set recordset as blacklisted;
    * ``_phone_reset_blacklisted``: reactivate recordset (even if not blacklisted
        this method can be called safely);

Put menus in technical in order to have access to it. Add a Phone / SMS
menu below "Email" and use it to store SMS / Phone actions.

Finally prepare tests addition by performing some light cleaning while adding
blacklist tests. Purpose is to ease future tests related to SMS.

LINKS

Task 1997464
PR #34424
Original SMS addition: Task 1922163 (4287481)

8 files changed, 344 insertions, 27 deletions

diff --git a/addons/phone_validation/__manifest__.py b/addons/phone_validation/__manifest__.py
index 2ebc99f3ead..9afac809ea8 100644
--- a/addons/phone_validation/__manifest__.py
+++ b/addons/phone_validation/__manifest__.py
@@ -3,6 +3,7 @@
 
 {
     'name': 'Phone Numbers Validation',
+    'version': '2.0',
     'summary': 'Validate and format phone numbers',
     'sequence': '9999',
     'category': 'Hidden',
@@ -11,12 +12,24 @@ Phone Numbers Validation
 ========================
 
 This module adds the feature of validation and formatting phone numbers
-according to a destination country. It also handles national and international
-formatting.
+according to a destination country.
 
-This module applies this feature to Leads and Contacts.""",
+It also adds phone blacklist management through a specific model storing
+blacklisted phone numbers.
+
+It adds two mixins :
+
+  * phone.validation.mixin: parsing / formatting helpers on records, to be
+    used for example in number fields onchange;
+  * mail.thread.phone: handle sanitation and blacklist of records numbers;
+""",
     'data': [
-        'views/res_company_view.xml'
+        'security/ir.model.access.csv',
+        'views/res_company_view.xml',
+        'views/phone_blacklist_views.xml',
+    ],
+    'depends': [
+        'base',
+        'mail',
     ],
-    'depends': ['base'],
 }
diff --git a/addons/phone_validation/models/__init__.py b/addons/phone_validation/models/__init__.py
index 85ea0597483..dfe42e557e3 100644
--- a/addons/phone_validation/models/__init__.py
+++ b/addons/phone_validation/models/__init__.py
@@ -1,5 +1,7 @@
 # -*- coding: utf-8 -*-
 # Part of Odoo. See LICENSE file for full copyright and licensing details.
 
-from . import res_company
+from . import phone_blacklist
 from . import phone_validation_mixin
+from . import mail_thread_phone
+from . import res_company
diff --git a/addons/phone_validation/models/mail_thread_phone.py b/addons/phone_validation/models/mail_thread_phone.py
new file mode 100644
index 00000000000..3e166985f8e
--- /dev/null
+++ b/addons/phone_validation/models/mail_thread_phone.py
@@ -0,0 +1,103 @@
+# -*- coding: utf-8 -*-
+# Part of Odoo. See LICENSE file for full copyright and licensing details.
+
+from odoo import api, fields, models, _
+from odoo.exceptions import UserError
+
+
+class PhoneMixin(models.AbstractModel):
+    """ Purpose of this mixin is to offer two services
+
+      * compute a sanitized phone number based on ´´_sms_get_number_fields´´.
+        It takes first sanitized value, trying each field returned by the
+        method (see ``MailThread._sms_get_number_fields()´´ for more details
+        about the usage of this method);
+      * compute blacklist state of records. It is based on phone.blacklist
+        model and give an easy-to-use field and API to manipulate blacklisted
+        records;
+
+    Main API methods
+
+      * ``_phone_set_blacklisted``: set recordset as blacklisted;
+      * ``_phone_reset_blacklisted``: reactivate recordset (even if not blacklisted
+        this method can be called safely);
+    """
+    _name = 'mail.thread.phone'
+    _description = 'Phone Blacklist Mixin'
+    _inherit = ['mail.thread', 'phone.validation.mixin']
+
+    phone_sanitized = fields.Char(
+        string='Sanitized Number', compute="_compute_phone_sanitized", compute_sudo=True, store=True,
+        help="Field used to store sanitized phone number. Helps speeding up searches and comparisons.")
+    phone_blacklisted = fields.Boolean(
+        string='Phone Blacklisted', compute="_compute_phone_blacklisted", compute_sudo=True, store=False,
+        search="_search_phone_blacklisted", groups="base.group_user",
+        help="If the email address is on the blacklist, the contact won't receive mass mailing anymore, from any list")
+
+    @api.depends(lambda self: self._phone_get_number_fields())
+    def _compute_phone_sanitized(self):
+        self._assert_phone_field()
+        number_fields = self._phone_get_number_fields()
+        for record in self:
+            for fname in number_fields:
+                sanitized = record.phone_get_sanitized_number(number_fname=fname)
+                if sanitized:
+                    break
+            record.phone_sanitized = sanitized
+
+    @api.depends('phone_sanitized')
+    def _compute_phone_blacklisted(self):
+        # TODO : Should remove the sudo as compute_sudo defined on methods.
+        # But if user doesn't have access to mail.blacklist, doen't work without sudo().
+        blacklist = set(self.env['phone.blacklist'].sudo().search([
+            ('number', 'in', self.mapped('phone_sanitized'))]).mapped('number'))
+        for record in self:
+            record.phone_blacklisted = record.phone_sanitized in blacklist
+
+    @api.model
+    def _search_phone_blacklisted(self, operator, value):
+        # Assumes operator is '=' or '!=' and value is True or False
+        self._assert_phone_field()
+        if operator != '=':
+            if operator == '!=' and isinstance(value, bool):
+                value = not value
+            else:
+                raise NotImplementedError()
+
+        if value:
+            query = """
+                SELECT m.id
+                    FROM phone_blacklist bl
+                    JOIN %s m
+                    ON m.phone_sanitized = bl.number AND bl.active
+            """
+        else:
+            query = """
+                SELECT m.id
+                    FROM %s m
+                    LEFT JOIN phone_blacklist bl
+                    ON m.phone_sanitized = bl.number AND bl.active
+                    WHERE bl.id IS NULL
+            """
+        self._cr.execute(query % self._table)
+        res = self._cr.fetchall()
+        if not res:
+            return [(0, '=', 1)]
+        return [('id', 'in', [r[0] for r in res])]
+
+    def _assert_phone_field(self):
+        if not hasattr(self, "_phone_get_number_fields"):
+            raise UserError(_('Invalid primary phone field on model %s') % self._name)
+        if not any(fname in self and self._fields[fname].type == 'char' for fname in self._phone_get_number_fields()):
+            raise UserError(_('Invalid primary phone field on model %s') % self._name)
+
+    def _phone_get_number_fields(self):
+        """ This method returns the fields to use to find the number to use to
+        send an SMS on a record. """
+        return []
+
+    def _phone_set_blacklisted(self):
+        return self.env['phone.blacklist'].sudo()._add([r.phone_sanitized for r in self])
+
+    def _phone_reset_blacklisted(self):
+        return self.env['phone.blacklist'].sudo()._remove([r.phone_sanitized for r in self])
diff --git a/addons/phone_validation/models/phone_blacklist.py b/addons/phone_validation/models/phone_blacklist.py
new file mode 100644
index 00000000000..2a6e90b6cb5
--- /dev/null
+++ b/addons/phone_validation/models/phone_blacklist.py
@@ -0,0 +1,109 @@
+# -*- coding: utf-8 -*-
+# Part of Odoo. See LICENSE file for full copyright and licensing details.
+
+import logging
+
+from odoo import api, fields, models, _
+from odoo.addons.phone_validation.tools import phone_validation
+from odoo.exceptions import UserError
+
+_logger = logging.getLogger(__name__)
+
+
+class PhoneBlackList(models.Model):
+    """ Blacklist of phone numbers. Used to avoid sending unwanted messages to people. """
+    _name = 'phone.blacklist'
+    _inherit = ['mail.thread']
+    _description = 'Phone Blacklist'
+    _rec_name = 'number'
+
+    number = fields.Char(string='Phone Number', required=True, index=True, tracking=True, help='Number should be E164 formatted')
+    active = fields.Boolean(default=True, tracking=True)
+
+    _sql_constraints = [
+        ('unique_number', 'unique (number)', 'Number already exists')
+    ]
+
+    @api.model_create_multi
+    def create(self, values):
+        # First of all, extract values to ensure emails are really unique (and don't modify values in place)
+        to_create = []
+        done = set()
+        for value in values:
+            number = value['number']
+            sanitized = phone_validation.phone_sanitize_numbers_w_record([number], self.env.user)[number]['sanitized']
+            if not sanitized:
+                raise UserError(_('Invalid number %s') % number)
+            if sanitized in done:
+                continue
+            done.add(sanitized)
+            to_create.append(dict(value, number=sanitized))
+
+        """ To avoid crash during import due to unique email, return the existing records if any """
+        sql = '''SELECT number, id FROM phone_blacklist WHERE number = ANY(%s)'''
+        numbers = [v['number'] for v in to_create]
+        self._cr.execute(sql, (numbers,))
+        bl_entries = dict(self._cr.fetchall())
+        to_create = [v for v in to_create if v['number'] not in bl_entries]
+
+        results = super(PhoneBlackList, self).create(to_create)
+        return self.env['phone.blacklist'].browse(bl_entries.values()) | results
+
+    def write(self, values):
+        if 'number' in values:
+            number = values['number']
+            sanitized = phone_validation.phone_sanitize_numbers_w_record([number], self.env.user)[number]['sanitized']
+            if not sanitized:
+                raise UserError(_('Invalid number %s') % number)
+            values['number'] = sanitized
+        return super(PhoneBlackList, self).write(values)
+
+    def _search(self, args, offset=0, limit=None, order=None, count=False, access_rights_uid=None):
+        """ Override _search in order to grep search on sanitized number field """
+        if args:
+            new_args = []
+            for arg in args:
+                if isinstance(arg, (list, tuple)) and arg[0] == 'number' and isinstance(arg[2], str):
+                    number = arg[2]
+                    sanitized = phone_validation.phone_sanitize_numbers_w_record([number], self.env.user)[number]['sanitized']
+                    if sanitized:
+                        new_args.append([arg[0], arg[1], sanitized])
+                    else:
+                        new_args.append(arg)
+                else:
+                    new_args.append(arg)
+        else:
+            new_args = args
+        return super(PhoneBlackList, self)._search(new_args, offset=offset, limit=limit, order=order, count=count, access_rights_uid=access_rights_uid)
+
+    def add(self, number):
+        sanitized = phone_validation.phone_sanitize_numbers_w_record([number], self.env.user)[number]['sanitized']
+        return self._add([sanitized])
+
+    def _add(self, numbers):
+        """ Add or re activate a phone blacklist entry.
+
+        :param numbers: list of sanitized numbers """
+        records = self.env["phone.blacklist"].with_context(active_test=False).search([('number', 'in', numbers)])
+        todo = [n for n in numbers if n not in records.mapped('number')]
+        if records:
+            records.write({'active': True})
+        if todo:
+            records += self.create([{'number': n} for n in todo])
+        return records
+
+    def remove(self, number):
+        sanitized = phone_validation.phone_sanitize_numbers_w_record([number], self.env.user)[number]['sanitized']
+        return self._remove([sanitized])
+
+    def _remove(self, numbers):
+        """ Add de-activated or de-activate a phone blacklist entry.
+
+        :param numbers: list of sanitized numbers """
+        records = self.env["phone.blacklist"].with_context(active_test=False).search([('number', 'in', numbers)])
+        todo = [n for n in numbers if n not in records.mapped('number')]
+        if records:
+            records.write({'active': False})
+        if todo:
+            records += self.create([{'number': n, 'active': False} for n in todo])
+        return records
diff --git a/addons/phone_validation/models/phone_validation_mixin.py b/addons/phone_validation/models/phone_validation_mixin.py
index f062586c7c3..0e7ab1c2aaa 100644
--- a/addons/phone_validation/models/phone_validation_mixin.py
+++ b/addons/phone_validation/models/phone_validation_mixin.py
@@ -9,6 +9,11 @@ class PhoneValidationMixin(models.AbstractModel):
     _name = 'phone.validation.mixin'
     _description = 'Phone Validation Mixin'
 
+    def _phone_get_country_field(self):
+        if 'country_id' in self:
+            return 'country_id'
+        return False
+
     def _phone_get_country(self):
         if 'country_id' in self and self.country_id:
             return self.country_id
@@ -31,3 +36,17 @@ class PhoneValidationMixin(models.AbstractModel):
             force_format='INTERNATIONAL' if always_international else 'NATIONAL',
             raise_exception=False
         )
+
+    def phone_get_sanitized_numbers(self, number_fname='mobile', force_format='E164'):
+        res = dict.fromkeys(self.ids, False)
+        country_fname = self._phone_get_country_field()
+        for record in self:
+            number = record[number_fname]
+            res[record.id] = phone_validation.phone_sanitize_numbers_w_record([number], record, record_country_fname=country_fname, force_format=force_format)[number]['sanitized']
+        return res
+
+    def phone_get_sanitized_number(self, number_fname='mobile', force_format='E164'):
+        self.ensure_one()
+        country_fname = self._phone_get_country_field()
+        number = self[number_fname]
+        return phone_validation.phone_sanitize_numbers_w_record([number], self, record_country_fname=country_fname, force_format=force_format)[number]['sanitized']
diff --git a/addons/phone_validation/security/ir.model.access.csv b/addons/phone_validation/security/ir.model.access.csv
new file mode 100644
index 00000000000..53a27a4e6b2
--- /dev/null
+++ b/addons/phone_validation/security/ir.model.access.csv
@@ -0,0 +1,3 @@
+id,name,model_id:id,group_id:id,perm_read,perm_write,perm_create,perm_unlink
+access_phone_blacklist_all,access.phone.blacklist.all,model_phone_blacklist,,0,0,0,0
+access_phone_blacklist_system,access.phone.blacklist.system,model_phone_blacklist,base.group_system,1,1,1,1
diff --git a/addons/phone_validation/tools/phone_validation.py b/addons/phone_validation/tools/phone_validation.py
index e1df861ac8c..91ee9ba6b37 100644
--- a/addons/phone_validation/tools/phone_validation.py
+++ b/addons/phone_validation/tools/phone_validation.py
@@ -17,7 +17,7 @@ try:
         try:
             phone_nbr = phonenumbers.parse(number, region=country_code, keep_raw_input=True)
         except phonenumbers.phonenumberutil.NumberParseException as e:
-            raise UserError(_('Unable to parse %s:\n%s') % (number, e))
+            raise UserError(_('Unable to parse %s: %s') % (number, str(e)))
 
         if not phonenumbers.is_possible_number(phone_nbr):
             raise UserError(_('Impossible number %s: probably invalid number of digits') % number)
@@ -77,45 +77,41 @@ except ImportError:
 
 
 def phone_sanitize_numbers(numbers, country_code, country_phone_code, force_format='E164'):
+    """ Given a list of numbers, return parsezd and sanitized information
+
+    :return dict: {number: {
+        'sanitized': sanitized and formated number or False (if cannot format)
+        'code': 'empty' (number was a void string), 'invalid' (error) or False (sanitize ok)
+        'msg': error message when 'invalid'
+    }}
+    """
+    if not isinstance(numbers, (list)):
+        raise NotImplementedError()
     result = dict.fromkeys(numbers, False)
     for number in numbers:
         if not number:
             result[number] = {'sanitized': False, 'code': 'empty', 'msg': False}
             continue
         try:
+            stripped = number.strip()
             sanitized = phone_format(
-                number, country_code, country_phone_code,
+                stripped, country_code, country_phone_code,
                 force_format=force_format, raise_exception=True)
         except Exception as e:
-            result[number] = {'sanitized': False, 'code': 'invalid', 'msg': e}
+            result[number] = {'sanitized': False, 'code': 'invalid', 'msg': str(e)}
         else:
             result[number] = {'sanitized': sanitized, 'code': False, 'msg': False}
     return result
 
 
 def phone_sanitize_numbers_w_record(numbers, record, country=False, record_country_fname='country_id', force_format='E164'):
+    if not isinstance(numbers, (list)):
+        raise NotImplementedError()
     if not country:
-        if record and hasattr(record, record_country_fname) and record[record_country_fname]:
+        if record and record_country_fname and hasattr(record, record_country_fname) and record[record_country_fname]:
             country = record[record_country_fname]
         elif record:
             country = record.env.company.country_id
     country_code = country.code if country else None
     country_phone_code = country.phone_code if country else None
     return phone_sanitize_numbers(numbers, country_code, country_phone_code, force_format=force_format)
-
-
-def phone_sanitize_numbers_string_w_record(numbers_str, record, country=False, record_country_fname='country_id', force_format='E164'):
-    found_numbers = [number.strip() for number in numbers_str.split(',')]
-    return phone_sanitize_numbers_w_record(found_numbers, record, country=country, record_country_fname=record_country_fname, force_format=force_format)
-
-
-def phone_get_sanitized_records_number(records, number_fname='mobile', country_fname='country_id', force_format='E164'):
-    res = dict.fromkeys(records.ids, False)
-    for record in records:
-        number = record[number_fname]
-        res[record.id] = phone_sanitize_numbers_w_record([number], records, record_country_fname=country_fname,force_format=force_format)[number]['sanitized']
-    return res
-
-
-def phone_get_sanitized_record_number(record, number_fname='mobile', country_fname='country_id', force_format='E164'):
-    return phone_get_sanitized_records_number(record, number_fname, country_fname, force_format=force_format)[record.id]
diff --git a/addons/phone_validation/views/phone_blacklist_views.xml b/addons/phone_validation/views/phone_blacklist_views.xml
new file mode 100644
index 00000000000..4d76576cefa
--- /dev/null
+++ b/addons/phone_validation/views/phone_blacklist_views.xml
@@ -0,0 +1,72 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<odoo>
+    <record id="phone_blacklist_view_tree" model="ir.ui.view">
+        <field name="name">phone.blacklist.view.tree</field>
+        <field name="model">phone.blacklist</field>
+        <field name="arch" type="xml">
+            <tree string="Phone Blacklist">
+                <field name="create_date" string="Blacklist Date"/>
+                <field name="number"/>
+            </tree>
+        </field>
+    </record>
+
+    <record id="phone_blacklist_view_form" model="ir.ui.view">
+        <field name="name">phone.blacklist.view.form</field>
+        <field name="model">phone.blacklist</field>
+        <field name="arch" type="xml">
+            <form string="Phone Blacklist" duplicate="false" edit="false">
+                <sheet>
+                    <group>
+                        <group>
+                            <field name="number"/>
+                            <field name="active"/>
+                        </group>
+                    </group>
+                </sheet>
+                <div class="oe_chatter">
+                    <field name="message_follower_ids" widget="mail_followers"/>
+                    <field name="message_ids" widget="mail_thread"/>
+                </div>
+            </form>
+        </field>
+    </record>
+
+    <record id="phone_blacklist_view_search" model="ir.ui.view">
+        <field name="name">phone.blacklist.view.search</field>
+        <field name="model">phone.blacklist</field>
+        <field name="arch" type="xml">
+            <search>
+                <field name="number"/>
+                <filter string="Archived" name="inactive" domain="[('active','=',False)]"/>
+            </search>
+        </field>
+    </record>
+
+    <record id="phone_blacklist_action" model="ir.actions.act_window">
+        <field name="name">Phone Blacklist</field>
+        <field name="res_model">phone.blacklist</field>
+        <field name="view_id" ref="phone_blacklist_view_tree"/>
+        <field name="search_view_id" ref="phone_blacklist_view_search"/>
+        <field name="help" type="html">
+            <p class="o_view_nocontent_smiling_face">
+                Add a phone number in the blacklist
+            </p><p>
+                Blacklisted phone numbers means that the recipient won't receive mass SMS anymore.
+            </p>
+        </field>
+    </record>
+
+    <!-- Technical Menu -->
+    <menuitem id="phone_menu_main"
+        name="Phone / SMS"
+        parent="base.menu_custom"
+        sequence="2"/>
+
+    <menuitem id="phone_blacklist_menu"
+        name="Phone Blacklist"
+        parent="phone_validation.phone_menu_main"
+        sequence="3"
+        action="phone_blacklist_action"/>
+
+</odoo>