[IMP] crm : add and apply Predictive Lead Scoring

This new Predictive Lead Scoring (aka PLS) replaces the stage based probability mechanism.
It uses Naive Bayes probability computation theorem.

The lead probability is now computated automatically based on multiple criteria (= frequency fields).
Some are mandatory and will always be used, some are optional and can be activated
in the settings.
- Mandatory fields :
    - team_id
    - stage_id
    - tag_ids

- Optional fields :
    - email_state (validity of the email_from field: correct / incorrect or empty if email_from is empty)
    - phone_state (validity of the phone field !! not the mobile one !!: correct / incorrect or empty if phone is empty)
    - country_id
    - state_id
    - source_id (UTM Source)

PLS uses a start date to consider only the lead created after that date to generate the frequency table
and to recompute the lead probability. This date is also configurable in the CRM settings.
Default start date is date.todays() (at module install).

The settings for PLS are NOT company related :
PLS Fields = Many2many to a model that store only the fields allowed for PLS computation.
PLS Start Date = Date
As Date and Many2many fields are not allowed for config_parameters,
Char fields are used to store the configuration
and Date and m2m fields are used to ease the configuration by the user in the config panel.
Date and m2m fields are both computed fields based on their corresponding Char fields.
Char field for Date store a strigified date.
char field for Many2many store a comma separated string which is a list of activated fields.

Each mandatory field is used in a specific way
- Team_id : the frequency table is split for every team_id, plus once for leads that don't have a team_id.
Considering Lead A from Team 1 and Lead B from Team 2.
Even if Lead A and Lead B have the exact same attributes (country, phone, email,stage, ...),
their probability will most likely not be the same.
That's because each teams compute their leads' probability based on their own won/lost leads.

- Stage_id : if no optional fields are activated and if there is no tag on the lead,
this will make PLS work quite the same way as it was working with removed stage based probability,
as only the stage will count in the computation.
But here, instead of fixing manually the probability for each stage,
this stage related probability is automatically computed based on past experience (won and lost leads).

- Tag_ids : each tag is considered separatelly, as if the lead had only one tag.
(no link between tag on same lead is made). To avoid that a tag take to much importance if his subset is too
small, we include the tag frequencies in the frequency table only if at least 50 won or lost leads had this tag.

The frequency table is a table where all won and lost leads are aggregated by fields
(mandatory or optional if activated). For each team_id / field couple, we store the number of
won and lost leads that has that fields values.
For example: Lead A is assigned to team 1 and client comes from France and has been won.
If we consider only this lead A, the frequency table will looks like :
   id   |  variable   | value | won_count | lost_count | team_id
--------+-------------+-------+-----------+------------+---------
    1     country_id     FR         1           0            1

To computed the probability of a lead, we get all the records for the frequency table
that match all fields (per team_id) and we compute a won score and a lost score.
The probability is computed and normalized based on those scores
- P = S(Won) / (S(Won) + S(Lost))

Considering two variables A and B :
- S(Won) ∝ P(A∩B | Won)*P(Won) = P(A|Won) * P(B|Won) * P(Won)
- S(Lost) ∝ P(A∩B | Lost)*P(Lost) = P(A|Lost) * P(B|Lost) * P(Lost)

To overcome the 'zero frequency problem',
we do not start at 0 for the won or lost count for each variable / value combination.
It suggested to start at 1, but to avoid that a small subset take to much weight compared to
a larger one, we start at 0.1.

For example :
If we start at 1 : If team 1 has few records from France,
it can get high probability to win even if every lead where lost.

  variable   | value | won_count | lost_count | team_id | Probability
-------------+-------+-----------+------------+---------+-------------
 country_id     FR         1           7            1         0.125
 country_id     FR        198         7306          2         0.0263

If we start at 0.1 :

  variable   | value | won_count | lost_count | team_id | Probability
-------------+-------+-----------+------------+---------+-------------
 country_id     FR         0.1         6.1          1         0.0161
 country_id     FR        197.1       7305.1        2         0.0263

The more we have records for each couple, the more the computation become precise.

The allow the user to decide himself which probability should be set on the lead,
a manual probabiltiy system has been implemented using 2 stored fields + 1 computed field :
- probability (stored): Probability set on the lead, can be set manually in the lead form.
- automated_probability (stored): Probability computed by PLS.
- is_automated_probability (computed): If both previous fields are equal, the probability is considered
as automatic. and each time the automated_probability is updated, the probability is aligned.
If both are not equal, the probability is considered as manual. The automated_probability is still
updated but the probability is not aligned. The user can reset the probability in automatic mode
by clicking on the estimated probability displayed in edit mode in the lead form.

probability and automated_probability are not computed fields as they are manually computed at write and create.
The computation is triggered if one of the activated frequency fields is modified or set.
To trigger also the computation when modifying the lead values on the lead views (form and kanban),
_onchange methods have been added for each frequency field.

Also, a cron has been added to run every day to rebuild the frequency table and to recompute all active
and pending leads (not won nor lost) probability.

Task ID : 1925439
PR #33589

remove rerun cron on action lost and won

10 files changed, 663 insertions, 10 deletions

diff --git a/addons/crm/__manifest__.py b/addons/crm/__manifest__.py
index 4b25fb7538e..1100d1f4214 100644
--- a/addons/crm/__manifest__.py
+++ b/addons/crm/__manifest__.py
@@ -20,6 +20,7 @@
         'web_tour',
         'contacts',
         'digest',
+        'phone_validation',
     ],
     'data': [
         'security/crm_security.xml',
@@ -29,6 +30,7 @@
         'data/crm_stage_data.xml',
         'data/crm_lead_data.xml',
         'data/digest_data.xml',
+        'data/crm_lead_prediction_data.xml',
 
         'wizard/crm_lead_lost_views.xml',
         'wizard/crm_lead_to_opportunity_views.xml',
diff --git a/addons/crm/data/crm_lead_prediction_data.xml b/addons/crm/data/crm_lead_prediction_data.xml
new file mode 100644
index 00000000000..20a432a11f1
--- /dev/null
+++ b/addons/crm/data/crm_lead_prediction_data.xml
@@ -0,0 +1,33 @@
+<?xml version="1.0" encoding='UTF-8'?>
+<odoo>
+    <data noupdate="1">
+        <!-- Lead scoring frequency fields -->
+        <record id="frequency_field_state_id" model="crm.lead.scoring.frequency.field">
+            <field name="field_id" eval="ref('crm.field_crm_lead__state_id')"/>
+        </record>
+        <record id="frequency_field_country_id" model="crm.lead.scoring.frequency.field">
+            <field name="field_id" eval="ref('crm.field_crm_lead__country_id')"/>
+        </record>
+        <record id="frequency_field_phone_state" model="crm.lead.scoring.frequency.field">
+            <field name="field_id" eval="ref('crm.field_crm_lead__phone_state')"/>
+        </record>
+        <record id="frequency_field_email_state" model="crm.lead.scoring.frequency.field">
+            <field name="field_id" eval="ref('crm.field_crm_lead__email_state')"/>
+        </record>
+        <record id="frequency_field_source_id" model="crm.lead.scoring.frequency.field">
+            <field name="field_id" eval="ref('crm.field_crm_lead__source_id')"/>
+        </record>
+    </data>
+
+	<record id="website_crm_score_cron" model="ir.cron">
+        <field name="name">Predictive Lead Scoring: Rebuild Frequencies table</field>
+        <field name="model_id" ref="model_crm_lead"/>
+        <field name="state">code</field>
+        <field name="code">model._cron_update_automated_probabilities()</field>
+        <field name="interval_number">1</field>
+        <field name="interval_type">days</field>
+        <field name="numbercall">-1</field>
+        <field name="active" eval="True"/>
+        <field name="doall" eval="False"/>
+    </record>
+</odoo>
diff --git a/addons/crm/models/__init__.py b/addons/crm/models/__init__.py
index c51fbebbb55..0a441965aa8 100644
--- a/addons/crm/models/__init__.py
+++ b/addons/crm/models/__init__.py
@@ -9,3 +9,4 @@ from . import crm_team
 from . import res_config_settings
 from . import res_partner
 from . import digest
+from . import crm_lead_scoring_frequency
diff --git a/addons/crm/models/crm_lead.py b/addons/crm/models/crm_lead.py
index 4b869d65ca7..8b3b45f79ff 100644
--- a/addons/crm/models/crm_lead.py
+++ b/addons/crm/models/crm_lead.py
@@ -2,6 +2,7 @@
 # Part of Odoo. See LICENSE file for full copyright and licensing details.
 
 import logging
+from psycopg2 import sql, extras
 from datetime import datetime, timedelta, date
 from dateutil.relativedelta import relativedelta
 
@@ -9,6 +10,8 @@ from odoo import api, fields, models, tools, SUPERUSER_ID
 from odoo.tools.translate import _
 from odoo.tools import email_re, email_split
 from odoo.exceptions import UserError, AccessError
+from odoo.addons.phone_validation.tools import phone_validation
+from collections import OrderedDict
 
 from . import crm_stage
 
@@ -50,9 +53,17 @@ class Lead(models.Model):
     _name = "crm.lead"
     _description = "Lead/Opportunity"
     _order = "priority desc,activity_date_deadline,id desc"
-    _inherit = ['mail.thread.cc', 'mail.thread.blacklist', 'mail.activity.mixin', 'utm.mixin', 'format.address.mixin']
+    _inherit = ['mail.thread.cc', 'mail.thread.blacklist', 'mail.activity.mixin',
+                'utm.mixin', 'format.address.mixin', 'phone.validation.mixin']
     _primary_email = 'email_from'
 
+    @api.model_cr_context
+    def _auto_init(self):
+        res = super(Lead, self)._auto_init()
+        tools.create_index(self._cr, 'crm_lead_create_date_team_id_idx',
+                           self._table, ['create_date', 'team_id'])
+        return res
+
     def _default_stage_id(self):
         team = self.env['crm.team'].sudo()._get_default_team_id(user_id=self.env.uid)
         return self._stage_find(team_id=team.id, domain=[('fold', '=', False)]).id
@@ -90,8 +101,20 @@ class Lead(models.Model):
     date_last_stage_update = fields.Datetime(string='Last Stage Update', index=True, default=fields.Datetime.now)
     date_conversion = fields.Datetime('Conversion Date', readonly=True)
 
-    # Only used for type opportunity
+    # Probability - Only used for type opportunity
     probability = fields.Float('Probability', group_operator="avg", default=10.0)
+    automated_probability = fields.Float('Automated Probability', readonly=True)
+    is_automated_probability = fields.Boolean('Is automated probability?', compute="_compute_is_automated_probability")
+    phone_state = fields.Selection([
+        ('correct', 'Correct'),
+        ('incorrect', 'Incorrect'),
+        ('empty', 'Empty')], string='Phone Quality', default="empty", compute="_compute_phone_state", store=True)
+    email_state = fields.Selection([
+        ('correct', 'Correct'),
+        ('incorrect', 'Incorrect'),
+        ('empty', 'Empty')], string='Email quality', default="empty", compute="_compute_email_state", store=True)
+
+    # Only used for type opportunity
     planned_revenue = fields.Monetary('Expected Revenue', currency_field='company_currency', tracking=True)
     expected_revenue = fields.Monetary('Prorated Revenue', currency_field='company_currency', store=True, compute="_compute_expected_revenue")
     date_deadline = fields.Date('Expected Closing', help="Estimate of the date on which the opportunity will be won.")
@@ -180,6 +203,39 @@ class Lead(models.Model):
             date_close = fields.Datetime.from_string(lead.date_closed)
             lead.day_close = abs((date_close - date_create).days)
 
+    @api.depends('probability', 'automated_probability')
+    def _compute_is_automated_probability(self):
+        """ If probability and automated_probability are equal, the probability is considered as automatic.
+         when the probability is in automatic mode, the field probability will be aligned
+         when automated_probability is modified. """
+        for lead in self:
+            lead.is_automated_probability = tools.float_compare(lead.probability, lead.automated_probability, 2) == 0
+
+    @api.depends('phone', 'country_id.code')
+    def _compute_phone_state(self):
+        for lead in self:
+            phone_status = 'empty'
+            if lead.phone:
+                country_code = lead.country_id.code if lead.country_id and lead.country_id.code else None
+                try:
+                    if phone_validation.phone_parse(lead.phone, country_code):  # otherwise library not installed
+                        phone_status = 'correct'
+                except UserError:
+                    phone_status = 'incorrect'
+            lead.phone_state = phone_status
+
+    @api.depends('email_from')
+    def _compute_email_state(self):
+        for lead in self:
+            email_state = 'empty'
+            if lead.email_from:
+                email_state = 'incorrect'
+                for email in email_split(lead.email_from):
+                    if tools.email_validate(email):
+                        email_state = 'correct'
+                        break
+            lead.email_state = email_state
+
     @api.multi
     def _compute_meeting_count(self):
         meeting_data = self.env['calendar.event'].read_group([('opportunity_id', 'in', self.ids)], ['opportunity_id'], ['opportunity_id'])
@@ -238,6 +294,48 @@ class Lead(models.Model):
             values = self._onchange_user_values(self.user_id.id)
             self.update(values)
 
+    def _onchange_compute_probability(self, optional_field_name=None):
+        """Recompute probability on onchange methods of :
+            'stage_id', 'team_id', 'tag_ids'
+            'country_id', 'state_id', 'phone_state', 'email_state', 'source_id' """
+        if optional_field_name and optional_field_name not in self._pls_get_safe_fields():
+            return
+        lead_probabilities = self._pls_get_naive_bayes_probabilities()
+        if self.id in lead_probabilities:
+            self.automated_probability = lead_probabilities[self.id]
+            if self._origin.is_automated_probability:
+                self.probability = self.automated_probability
+
+    @api.onchange('stage_id')
+    def _onchange_stage_id(self):
+        self._onchange_compute_probability()
+
+    @api.onchange('source_id')
+    def _onchange_source_id(self):
+        self._onchange_compute_probability(optional_field_name='source_id')
+
+    @api.onchange('automated_probability')
+    def _onchange_automated_probability(self):
+        """ If was in auto mode before the change, align both fields so that it will stay in auto mode. """
+        if self._origin.is_automated_probability:
+            self.probability = self.automated_probability
+
+    @api.onchange('team_id')
+    def _onchange_team_id(self):
+        self._onchange_compute_probability()
+
+    @api.onchange('tag_ids')
+    def _onchange_tag_ids(self):
+        self._onchange_compute_probability()
+
+    @api.onchange('phone_state')
+    def _onchange_phone_state(self):
+        self._onchange_compute_probability(optional_field_name='phone_state')
+
+    @api.onchange('email_state')
+    def _onchange_email_state(self):
+        self._onchange_compute_probability(optional_field_name='email_state')
+
     @api.constrains('user_id')
     @api.multi
     def _valid_team(self):
@@ -249,11 +347,13 @@ class Lead(models.Model):
 
     @api.onchange('state_id')
     def _onchange_state(self):
+        self._onchange_compute_probability(optional_field_name='state_id')
         if self.state_id:
             self.country_id = self.state_id.country_id.id
 
     @api.onchange('country_id')
     def _onchange_country_id(self):
+        self._onchange_compute_probability(optional_field_name='country_id')
         res = {'domain': {'state_id': []}}
         if self.country_id:
             res['domain']['state_id'] = [('country_id', '=', self.country_id.id)]
@@ -293,7 +393,10 @@ class Lead(models.Model):
         onchange_values.update(vals)  # we don't want to overwrite any existing key
         vals = onchange_values
 
-        return super(Lead, self.with_context(context)).create(vals)
+        result = super(Lead, self.with_context(context)).create(vals)
+        # Compute new probability for each lead separately
+        result._write_probability(vals)
+        return result
 
     @api.multi
     def write(self, vals):
@@ -313,7 +416,32 @@ class Lead(models.Model):
             vals['date_closed'] = False
         if vals.get('user_id') and 'date_open' not in vals:
             vals['date_open'] = fields.Datetime.now()
-        return super(Lead, self).write(vals)
+
+        write_result = super(Lead, self).write(vals)
+        # Compute new automated_probability (and, eventually, probability) for each lead separately
+        self._write_probability(vals)
+
+        return write_result
+
+    def _write_probability(self, vals):
+        compute = False
+        fields_to_check = ['tag_ids', 'stage_id', 'team_id'] + self._pls_get_safe_fields()
+        for field, value in vals.items():
+            if field in fields_to_check:
+                compute = True
+                break
+
+        if not compute:
+            return
+        lead_probabilities = self._pls_get_naive_bayes_probabilities()
+        for lead in self:
+            if lead.id in lead_probabilities:
+                lead_proba = lead_probabilities[lead.id]
+                proba_vals = {'automated_probability': lead_proba}
+                if tools.float_compare(lead.probability, lead.automated_probability, 2) == 0:
+                    proba_vals['probability'] = lead_proba
+                super(Lead, lead).write(proba_vals)
+        return
 
     @api.multi
     @api.returns('self', lambda value: value.id)
@@ -349,8 +477,9 @@ class Lead(models.Model):
 
     @api.multi
     def action_set_lost(self):
-        """ Lost semantic: probability = 0, active = False """
-        return self.write({'probability': 0, 'active': False})
+        """ Lost semantic: probability = 0 or active = False """
+        result = self.write({'active': False, 'probability': 0})
+        return result
 
     @api.multi
     def action_set_won(self):
@@ -358,9 +487,11 @@ class Lead(models.Model):
         for lead in self:
             stage_id = lead._stage_find(domain=[('is_won', '=', True)])
             lead.write({'stage_id': stage_id.id, 'probability': 100})
-
         return True
 
+    def action_set_automated_probability(self):
+        self.write({'probability': self.automated_probability})
+
     @api.multi
     def action_set_won_rainbowman(self):
         self.ensure_one()
@@ -1218,6 +1349,376 @@ class Lead(models.Model):
             'template': '/crm/static/xls/crm_lead.xls'
         }]
 
+    # ----------------------------------------
+    # Predictive Lead Scoring
+    # ----------------------------------------
+    def _pls_get_naive_bayes_probabilities(self, batch_mode=False):
+        """
+        In machine learning, naive Bayes classifiers (NBC) are a family of simple "probabilistic classifiers" based on
+        applying Bayes theorem with strong (naive) independence assumptions between the variables taken into account.
+        E.g: will TDE eat m&m's depending on his sleep status, the amount of work he has and the fullness of his stomach?
+        As we use experience to compute the statistics, every day, we will register the variables state + the result.
+        As the days pass, we will be able to determine, with more and more precision, if TDE will eat m&m's
+        for a specific combination :
+            - did sleep very well, a lot of work and stomach full > Will never happen !
+            - didn't sleep at all, no work at all and empty stomach > for sure !
+        Following Bayes' Theorem: the probability that an event occurs (to win) under certain conditions is proportional
+        to the probability to win under each condition separately and the probability to win. We compute a 'Win score'
+        -> P(Won | A∩B) ∝ P(A∩B | Won)*P(Won) OR S(Won | A∩B) = P(A∩B | Won)*P(Won)
+        To compute a percentage of probability to win, we also compute the 'Lost score' that is proportional to the
+        probability to lose under each condition separately and the probability to lose.
+        -> Probability =  S(Won | A∩B) / ( S(Won | A∩B) + S(Lost | A∩B) )
+        See https://www.youtube.com/watch?v=CPqOCI0ahss can help to get a quick and simple example.
+        One issue about NBC is when a event occurence is never observed.
+        E.g: if when TDE has an empty stomach, he always eat m&m's, than the "not eating m&m's when empty stomach' event
+        will never be observed.
+        This is called 'zero frequency' and that leads to division (or at least multiplication) by zero.
+        To avoid this, we add 0.1 in each frequency. With few data, the computation is than not really realistic.
+        The more we have records to analyse, the more the estimation will be precise.
+        :return: probability in percent (and integer rounded) that the lead will be won at the current stage.
+        """
+        lead_probabilities = {}
+        if len(self) == 0:
+            return lead_probabilities
+
+        LeadScoringFrequency = self.env['crm.lead.scoring.frequency']
+
+        # get stages
+        first_stage_id = self.env['crm.stage'].search([], order='sequence', limit=1)
+
+        # Get all leads values, no matter the team_id
+        leads_values_dict = self._pls_get_lead_pls_values(batch_mode=batch_mode)
+        if not leads_values_dict:
+            return lead_probabilities
+
+        # Get unique couples to search in frequency table
+        leads_values = set()
+        for values in leads_values_dict.values():
+            for couple in values['values']:
+                leads_values.add(couple)
+
+        # get all variable related records from frequency table, no matter the team_id
+        fields = list(set([lead_value[0] for lead_value in leads_values]))
+        frequencies = LeadScoringFrequency.search([('variable', 'in', fields)], order="team_id asc")
+
+        # get all team_ids from frequencies
+        frequency_teams = frequencies.mapped('team_id')
+        frequency_team_ids = [0] + [team.id for team in frequency_teams]
+
+        # 1. Compute each variable value count individually
+        # regroup each variable to be able to compute their own probabilities
+        # As all the variable does not enter into account (as we reject unset values in the process)
+        # each value probability must be computed only with their own variable related total count
+        # special case: for lead for which team_id is not in frequency table,
+        # we consider all the records, independently from team_id (this is why we add a result[-1])
+        result = dict((team_id, dict((field, dict(won_total=0, lost_total=0)) for field in fields)) for team_id in frequency_team_ids)
+        result[-1] = dict((field, dict(won_total=0, lost_total=0)) for field in fields)
+        for frequency in frequencies:
+            team_result = result[frequency.team_id.id if frequency.team_id else 0]
+
+            field = frequency['variable']
+            value = frequency['value']
+
+            team_result[field][value] = {'won': frequency['won_count'], 'lost': frequency['lost_count']}
+            team_result[field]['won_total'] += frequency['won_count']
+            team_result[field]['lost_total'] += frequency['lost_count']
+
+            if value not in result[-1][field]:
+                result[-1][field][value] = {'won': 0, 'lost': 0}
+            result[-1][field][value]['won'] += frequency['won_count']
+            result[-1][field][value]['lost'] += frequency['won_count']
+            result[-1][field]['won_total'] += frequency['won_count']
+            result[-1][field]['lost_total'] += frequency['won_count']
+
+        # Get all won, lost and total count for all records in frequencies per team_id
+        for team_id in result:
+            result[team_id]['team_won'], \
+            result[team_id]['team_lost'], \
+            result[team_id]['team_total'] = self._pls_get_won_lost_total_count(result[team_id], first_stage_id)
+
+        save_team_id = None
+        p_won, p_lost = 1, 1
+        for lead_id, lead_values in leads_values_dict.items():
+            lead_team_id = lead_values['team_id'] if lead_values['team_id'] else 0  # team_id = None -> Convert to 0
+            lead_team_id = lead_team_id if lead_team_id in result else -1  # team_id not in frequency Table -> convert to -1
+            if lead_team_id != save_team_id:
+                save_team_id = lead_team_id
+                team_won = result[save_team_id]['team_won']
+                team_lost = result[save_team_id]['team_lost']
+                team_total = result[save_team_id]['team_total']
+                # if one count = 0, we cannot compute lead probability
+                if not team_won or not team_lost:
+                    continue
+                p_won = team_won / team_total
+                p_lost = team_lost / team_total
+
+            # 2. Compute won and lost score using each variable's individual probability
+            s_lead_won, s_lead_lost = p_won, p_lost
+            for field, value in lead_values['values']:
+                field_result = result.get(save_team_id, {}).get(field)
+                value_result = field_result.get(str(value)) if field_result else False
+                if value_result:
+                    total_won = team_won if field == 'stage_id' else field_result['won_total']
+                    total_lost = team_lost if field == 'stage_id' else field_result['lost_total']
+
+                    s_lead_won *= value_result['won'] / total_won
+                    s_lead_lost *= value_result['lost'] / total_lost
+
+            # 3. Compute Probability to win
+            lead_probabilities[lead_id] = 100 * s_lead_won / (s_lead_won + s_lead_lost)
+        return lead_probabilities
+
+    def _cron_update_automated_probabilities(self):
+        # Clear the frequencies table (in sql to speed up the cron)
+        try:
+            self.check_access_rights('unlink')
+        except AccessError:
+            raise UserError(_("You don't have the access needed to run this cron."))
+        else:
+            self._cr.execute('TRUNCATE TABLE crm_lead_scoring_frequency')
+
+        # get stages by sequence
+        stage_ids = self.env['crm.stage'].search_read([], ['sequence', 'name', 'id'], order='sequence')
+        stage_sequences = {stage['id']: stage['sequence'] for stage in stage_ids}
+
+        values_to_create = []
+        # Compute stat individually for each team
+        for team in self.env['crm.team'].with_context(active_test=False).search([]):
+            values_to_create = self._pls_update_frequency_table(values_to_create, stage_ids, stage_sequences, team_id=team.id)
+        values_to_create = self._pls_update_frequency_table(values_to_create, stage_ids, stage_sequences)
+
+        # create all frequencies from all company and team in batch
+        self.env['crm.lead.scoring.frequency'].create(values_to_create)
+        _logger.info("Predictive Lead Scoring : crm.lead.scoring.frequency table rebuilt")
+
+        # Recompute all the leads. Won : probability = 100 | Lost : probability = 0 or inactive
+        # Here, inactive won't be returned anyway
+        pls_start_date = self._pls_get_safe_start_date()
+        if pls_start_date:
+            pending_lead_domain = [('probability', '<', 100), ('probability', '>', 0), ('create_date', '>', pls_start_date)]
+            leads_to_update = self.env['crm.lead'].search(pending_lead_domain)
+            lead_probabilities = leads_to_update._pls_get_naive_bayes_probabilities(batch_mode=True)
+
+            # Update in execute batch to avoid server roundtrips + page_size to 10000 to avoid memory errors
+            sql = "update crm_lead set automated_probability = %s where id = %s"
+            batch_params = [(lead_probabilities[lead.id], lead.id) for lead in leads_to_update if lead.id in lead_probabilities]
+            extras.execute_batch(self._cr, sql, batch_params, page_size=10000)
+            _logger.info("Predictive Lead Scoring : all automated probability updated (count: %d)" % (len(leads_to_update)))
+
+    # ----------------------------
+    # Utility Tools for PLS
+    # ----------------------------
+
+    # PLS Config Parameters
+    # ---------------------
+    def _pls_get_safe_start_date(self):
+        """ As config_parameters does not accept Date field,
+            we get directly the date formated string stored into the Char config field,
+            as we directly use this string in the sql queries.
+            To avoid sql injections when using this config param,
+            we ensure the date string can be effectively a date."""
+        str_date = self.env['ir.config_parameter'].sudo().get_param('crm.pls_start_date')
+        if not fields.Date.to_date(str_date):
+            return False
+        return str_date
+
+    def _pls_get_safe_fields(self):
+        """ As config_parameters does not accept M2M field,
+            we the fields from the formated string stored into the Char config field.
+            To avoid sql injections when using that list, we return only the fields
+            that are defined on the model. """
+        pls_fields_config = self.env['ir.config_parameter'].sudo().get_param('crm.pls_fields')
+        pls_fields = pls_fields_config.split(',') if pls_fields_config else []
+        pls_safe_fields = [field for field in pls_fields if field in self._fields]
+        return pls_safe_fields
+
+    # Rebuild Frequency Table Tools
+    # -----------------------------
+    def _pls_update_frequency_table(self, values_to_create, stage_ids, stage_sequences, team_id=None):
+        """ Create / update the frequency table in a cross company way, per team_id"""
+        pls_start_date = self._pls_get_safe_start_date()
+        if not pls_start_date:
+            return values_to_create
+
+        fields = ['stage_id', 'team_id'] + self._pls_get_safe_fields()
+        frequencies = dict((field, {}) for field in (fields + ['tag_id']))
+
+        frequencies = self._pls_update_frequency_table_fields(frequencies, stage_ids, stage_sequences, fields, team_id, pls_start_date)
+        frequencies = self._pls_update_frequency_table_tag(frequencies, team_id, pls_start_date)
+
+        # build the create multi
+        for field, value in frequencies.items():
+            for param, result in value.items():
+                # To avoid that a tag take to much importance if his subset is too small,
+                # we include the tag frequencies in the frequency table only if at least 50 won or lost leads had this tag.
+                if field != 'tag_id' or (result['won'] + result['lost']) >= 50:
+                    # We add + 0.1 in won and lost counts to avoid zero frequency issues
+                    # should be +1 but it weights too much on small recordset.
+                    values_to_create.append({
+                        'variable': field,
+                        'value': param,
+                        'won_count': result['won'] + 0.1,
+                        'lost_count': result['lost'] + 0.1,
+                        'team_id': team_id
+                    })
+        return values_to_create
+
+    def _pls_update_frequency_table_fields(self, frequencies, stage_ids, stage_sequences, fields, team_id, pls_start_date):
+        # get all lead fields combination aggregated by won / lost count
+        #   Prepare fields injection
+        team_condition = 'and l.team_id = %s' if team_id else 'and l.team_id is null'
+        str_fields = ", ".join(["{}"] * len(fields))
+        args = [sql.Identifier(field) for field in fields] * 2
+
+        #   Build sql query in safe mode
+        query = """select probability, active, %s, count(probability) as count
+                    from crm_lead l
+                    where (probability = 0 or probability >= 100)
+                    and create_date > %%s
+                    %s
+                    group by probability, active, %s """
+        query = sql.SQL(query % (str_fields, team_condition, str_fields)).format(*args)
+
+        query_params = [pls_start_date] + ([int(team_id)] if team_id else [])
+        self._cr.execute(query, query_params)
+        results = self._cr.dictfetchall()
+
+        # Increment won / lost frequencies by criteria (field / value couple)
+        for result in results:
+            won = result['count'] if result['probability'] == 100 else 0
+            lost = result['count'] if result['probability'] == 0 else 0
+            for field in fields:
+                value = result[field]
+                if value:
+                    if field == 'stage_id':
+                        if won:  # increment all stages if won
+                            stages_to_increment = [stage['id'] for stage in stage_ids]
+                        else:  # increment only current + previous stages if lost
+                            current_stage_sequence = stage_sequences[value]
+                            stages_to_increment = [stage['id'] for stage in stage_ids if
+                                          stage['sequence'] <= current_stage_sequence]
+                        for stage in stages_to_increment:
+                            frequencies = self._pls_increment_frequency(frequencies, field, stage, won, lost)
+                    else:
+                        frequencies = self._pls_increment_frequency(frequencies, field, value, won, lost)
+
+        return frequencies
+
+    def _pls_update_frequency_table_tag(self, frequencies, team_id, pls_start_date):
+        # get all tag_ids won / lost count
+        query = """select l.probability, l.active, t.id, count(l.probability) as count
+                    from crm_lead_tag_rel rel
+                    inner join crm_lead_tag t on rel.tag_id = t.id
+                    inner join crm_lead l on l.id = rel.lead_id
+                    where (l.probability = 0 or l.probability >= 100) 
+                    and l.create_date > %%s
+                    %s
+                    group by l.probability, l.active, t.id"""
+        team_condition = 'and l.team_id = %s' if team_id else 'and l.team_id is null'
+        query_params = [pls_start_date] + ([int(team_id)] if team_id else [])
+        self._cr.execute(query % team_condition, query_params)
+        tag_results = self._cr.dictfetchall()
+
+        for result in tag_results:
+            won = result['count'] if result['probability'] == 100 else 0
+            lost = result['count'] if result['probability'] == 0 else 0
+            value = result['id']
+            frequencies = self._pls_increment_frequency(frequencies, 'tag_id', value, won, lost)
+
+        return frequencies
+
+    def _pls_increment_frequency(self, frequencies, field, value, won, lost):
+        if value not in frequencies[field]:
+            frequencies[field][value] = {'won': won, 'lost': lost}
+        else:
+            frequencies[field][value]['won'] += won
+            frequencies[field][value]['lost'] += lost
+        return frequencies
+
+    # Compute Automated Probability Tools
+    # -----------------------------------
+    def _pls_get_lead_pls_values(self, batch_mode=False):
+        """
+        Due to onchange, we don't have always the id of the lead to recompute.
+        When we update few records (one, typically) with onchanges, we build the
+        lead_values (= couple field/value) using the ORM.
+        To speed up the computation and avoid making too much DB read inside loops,
+        we can activate the batch_mode, that needs the id of each lead to recompute to be known.
+        That batch mode is directly making sql queries to bypass the ORM,
+        so we can get everything we need for all leads to recompute in a minimum number of queries.
+        This batch mode is currently called when the computation is triggered by
+        crm.lead._cron_update_automated_probabilities().
+        :param team_id: (int) team_id to search on
+        :param batch_mode: (bool) batch mode
+        :return: dict of list of tuple of field - value by lead ({lead_id: [(field1: value1), (field2: value2), ...], ...})
+        """
+        leads_values_dict = OrderedDict()
+        fields = ["stage_id", "team_id"] + self._pls_get_safe_fields()
+        if batch_mode:
+            # get all info on leads
+            #   Prepare fields injection
+            str_fields = ", ".join(["{}"] * len(fields))
+            args = [sql.Identifier(field) for field in fields]
+            #   Build sql query in safe mode
+            query = """SELECT id, %s
+                        FROM crm_lead l
+                        WHERE probability > 0 AND probability < 100 AND active = True AND id in %%s order by team_id asc"""
+            query = sql.SQL(query % str_fields).format(*args)
+
+            self._cr.execute(query, [tuple(self.ids)])
+            lead_results = self._cr.dictfetchall()
+
+            query = """SELECT l.id as lead_id, t.id as tag_id
+                        FROM crm_lead l
+                        LEFT JOIN crm_lead_tag_rel rel ON l.id = rel.lead_id
+                        LEFT JOIN crm_lead_tag t ON rel.tag_id = t.id
+                        WHERE l.probability > 0 AND l.probability < 100 AND l.active = True AND l.id in %s order by l.team_id asc"""
+            self._cr.execute(query, [tuple(self.ids)])
+            tag_results = self._cr.dictfetchall()
+
+            # get all (variable, value) couple for all in self
+            for lead in lead_results:
+                lead_values = []
+                for field in fields:
+                    if field == 'team_id':  # ignore team_id as stored separately in leads_values_dict[lead_id][team_id]
+                        continue
+                    value = lead[field]
+                    if value:
+                        lead_values.append((field, value))
+                leads_values_dict[lead['id']] = {'values': lead_values, 'team_id': lead['team_id']}
+
+            for tag in tag_results:
+                if tag['tag_id']:
+                    leads_values_dict[tag['lead_id']]['values'].append(('tag_id', tag['tag_id']))
+            return leads_values_dict
+        else:
+            for lead in self:
+                lead_values = []
+                for field in fields:
+                    if field == 'team_id':  # ignore team_id as stored separately in leads_values_dict[lead_id][team_id]
+                        continue
+                    value = lead[field].id if isinstance(lead[field], models.BaseModel) else lead[field]
+                    if value:
+                        lead_values.append((field, value))
+                for tag in lead.tag_ids:
+                    lead_values.append(('tag_id', tag.id))
+                leads_values_dict[lead.id] = {'values': lead_values, 'team_id': lead['team_id'].id}
+            return leads_values_dict
+
+    def _pls_get_won_lost_total_count(self, team_results, first_stage_id):
+        """ Get all won and all lost + total :
+               first stage can be used to know how many lost and won there is
+               as won count are equals for all stage
+               and first stage is always incremented in lost_count
+        :param frequencies: lead_scoring_frequencies
+        :param first_stage_id: stage with smallest sequence
+        :return: won count, lost count and total count for all records in frequencies
+        """
+        if str(first_stage_id.id) not in team_results['stage_id']:
+            return 0, 0, 0
+        stage_result = team_results['stage_id'][str(first_stage_id.id)]
+        return stage_result['won'], stage_result['lost'], stage_result['won'] + stage_result['lost']
+
 
 class Tag(models.Model):
 
diff --git a/addons/crm/models/crm_lead_scoring_frequency.py b/addons/crm/models/crm_lead_scoring_frequency.py
new file mode 100644
index 00000000000..fc6d0bdbbbe
--- /dev/null
+++ b/addons/crm/models/crm_lead_scoring_frequency.py
@@ -0,0 +1,21 @@
+# -*- coding: utf-8 -*-
+from odoo import fields, models
+
+
+class LeadScoringFrequency(models.Model):
+    _name = 'crm.lead.scoring.frequency'
+    _description = 'Lead Scoring Frequency'
+
+    variable = fields.Char('Variable', index=True)
+    value = fields.Char('Value')
+    won_count = fields.Float('Won Count', digits=(16, 1))  # Float because we add 0.1 to avoid zero Frequency issue
+    lost_count = fields.Float('Lost Count', digits=(16, 1))  # Float because we add 0.1 to avoid zero Frequency issue
+    team_id = fields.Many2one('crm.team', 'Sales Team')
+
+
+class FrequencyField(models.Model):
+    _name = 'crm.lead.scoring.frequency.field'
+    _description = 'Fields that can be used for predictive lead scoring computation'
+
+    name = fields.Char(related="field_id.field_description")
+    field_id = fields.Many2one('ir.model.fields', domain=[('model_id.model', '=', 'crm.lead')], required=True)
diff --git a/addons/crm/models/res_config_settings.py b/addons/crm/models/res_config_settings.py
index d364d1815fe..83ae62ff50e 100644
--- a/addons/crm/models/res_config_settings.py
+++ b/addons/crm/models/res_config_settings.py
@@ -2,6 +2,7 @@
 # Part of Odoo. See LICENSE file for full copyright and licensing details.
 
 from odoo import api, fields, models
+from datetime import datetime, date
 
 
 class ResConfigSettings(models.TransientModel):
@@ -15,6 +16,10 @@ class ResConfigSettings(models.TransientModel):
     module_crm_iap_lead_website = fields.Boolean("Create Leads/Opportunities from your website's traffic")
     lead_mining_in_pipeline = fields.Boolean("Create a lead mining request directly from the opportunity pipeline.", config_parameter='crm.lead_mining_in_pipeline')
     crm_phone_valid_method = fields.Selection(related="company_id.phone_international_format", required=True, readonly=False)
+    predictive_lead_scoring_start_date = fields.Date(string='Lead Scoring Starting Date', compute="_compute_pls_start_date", inverse="_inverse_pls_start_date_str")
+    predictive_lead_scoring_start_date_str = fields.Char(string='Lead Scoring Starting Date in String', default=date.today().strftime('%Y-%m-%d'), config_parameter='crm.pls_start_date')
+    predictive_lead_scoring_fields = fields.Many2many('crm.lead.scoring.frequency.field', string='Lead Scoring Frequency Fields', compute="_compute_pls_fields", inverse="_inverse_pls_fields_str")
+    predictive_lead_scoring_fields_str = fields.Char(string='Lead Scoring Frequency Fields in String', config_parameter='crm.pls_fields')
 
     def _find_default_lead_alias_id(self):
         alias = self.env.ref('crm.mail_alias_lead_info', False)
@@ -28,6 +33,38 @@ class ResConfigSettings(models.TransientModel):
             ], limit=1)
         return alias
 
+    @api.depends('predictive_lead_scoring_fields_str')
+    def _compute_pls_fields(self):
+        """ As config_parameters does not accept m2m field,
+            we get the fields back from the Char config field, to ease the configuration in config panel """
+        for setting in self:
+            if setting.predictive_lead_scoring_fields_str:
+                names = setting.predictive_lead_scoring_fields_str.split(',')
+                setting.predictive_lead_scoring_fields = self.env['crm.lead.scoring.frequency.field'].search([('name', 'in', names)])
+            else:
+                setting.predictive_lead_scoring_fields = None
+
+    def _inverse_pls_fields_str(self):
+        """ As config_parameters does not accept m2m field,
+            we store the fields with a comma separated string into a Char config field """
+        for setting in self:
+            if setting.predictive_lead_scoring_fields:
+                setting.predictive_lead_scoring_fields_str = ','.join(setting.predictive_lead_scoring_fields.mapped('name'))
+
+    @api.depends('predictive_lead_scoring_start_date_str')
+    def _compute_pls_start_date(self):
+        """ As config_parameters does not accept Date field,
+            we get the date back from the Char config field, to ease the configuration in config panel """
+        for setting in self:
+            setting.predictive_lead_scoring_start_date = fields.Date.to_date(setting.predictive_lead_scoring_start_date_str)
+
+    def _inverse_pls_start_date_str(self):
+        """ As config_parameters does not accept Date field,
+            we store the date formated string into a Char config field """
+        for setting in self:
+            if setting.predictive_lead_scoring_start_date:
+                setting.predictive_lead_scoring_start_date_str = fields.Date.to_string(setting.predictive_lead_scoring_start_date)
+
     @api.onchange('group_use_lead')
     def _onchange_group_use_lead(self):
         """ Reset alias / leads configuration if leads are not used """
diff --git a/addons/crm/security/ir.model.access.csv b/addons/crm/security/ir.model.access.csv
index a5da8f8fa02..7269054d25b 100644
--- a/addons/crm/security/ir.model.access.csv
+++ b/addons/crm/security/ir.model.access.csv
@@ -21,3 +21,5 @@ access_calendar_event_type_sale_manager,calendar.event.type.manager,calendar.mod
 access_calendar_event_type_sale_user,calendar.event.type.user,calendar.model_calendar_event_type,base.group_user,1,0,0,0

 access_calendar_event_type_sale_salesman,calendar.event.type.salesman,calendar.model_calendar_event_type,sales_team.group_sale_salesman,1,0,0,0

 access_mail_activity_type_sale_manager,mail.activity.type.sale.manager,mail.model_mail_activity_type,sales_team.group_sale_manager,1,1,1,1

+access_crm_lead_scoring_frequency,access_crm_lead_scoring_frequency,model_crm_lead_scoring_frequency,sales_team.group_sale_salesman,1,0,0,0

+access_crm_lead_scoring_frequency_field,access_crm_lead_scoring_frequency_field,model_crm_lead_scoring_frequency_field,sales_team.group_sale_salesman,1,0,0,0

diff --git a/addons/crm/static/src/css/crm.css b/addons/crm/static/src/css/crm.css
index f9cbe46b289..0cd13b24b70 100644
--- a/addons/crm/static/src/css/crm.css
+++ b/addons/crm/static/src/css/crm.css
@@ -2,3 +2,11 @@
 .openerp .oe_margin_top_8 {
 	margin-top: 8px;
 }
+
+.o_pls_probability button {
+    font-size: 1.2rem;
+}
+
+.o_pls_probability small {
+    font-size: 1rem;
+}
diff --git a/addons/crm/views/crm_lead_views.xml b/addons/crm/views/crm_lead_views.xml
index 65372748bcf..0384fe3b40e 100644
--- a/addons/crm/views/crm_lead_views.xml
+++ b/addons/crm/views/crm_lead_views.xml
@@ -74,6 +74,23 @@
                     <div class="oe_title">
                         <label for="name" class="oe_edit_only" string="Lead"/>
                         <h1><field name="name" placeholder="Describe the lead..."/></h1>
+                        <h2 class="o_row row no-gutters d-flex">
+                            <div class="col">
+                                <span class="oe_edit_only o_pls_probability">
+                                    <label for="probability"/>
+                                    <button class="btn btn-link" name="action_set_automated_probability" type="object"
+                                            attrs="{'invisible': [('is_automated_probability', '=', True)]}">
+                                        <i class="fa fa-gear" role="img" title="Switch to automatic probability" aria-label="Switch to automatic probability"></i>
+                                    </button>
+                                    <small class="oe_grey" attrs="{'invisible': [('is_automated_probability', '=', True)]}"><field name="automated_probability"/> %%</small>
+                                </span>
+                                <div id="probability" class="o_row d-flex">
+                                    <field name="is_automated_probability" invisible="1"/>
+                                    <field name="probability" widget="float" class="oe_inline"/>
+                                    <span class="oe_grey"> %%</span>
+                                </div>
+                            </div>
+                        </h2>
                     </div>
                     <group>
                         <group>
@@ -101,6 +118,8 @@
                                 <field name="title" placeholder="Title" domain="[]" options='{"no_open": True}'/>
                             </div>
                             <field name="is_blacklisted" invisible="1"/>
+                            <field name="email_state" invisible="1"/>
+                            <field name="phone_state" invisible="1"/>
                             <label for="email_from" class="oe_inline"/>
                             <div class="o_row o_row_readonly">
                                 <i class="fa fa-ban" style="color: red;" role="img" title="This email is blacklisted for mass mailing"
@@ -519,9 +538,17 @@
                                     </div>
                                 </div>
                                 <div class="col">
-                                    <label for="probability" class="oe_edit_only"/>
-                                    <div class="o_row d-flex">
-                                        <field name="probability" widget="integer" class="oe_inline"/>
+                                    <span class="oe_edit_only o_pls_probability">
+                                        <label for="probability"/>
+                                        <button class="btn btn-link" name="action_set_automated_probability" type="object"
+                                                attrs="{'invisible': [('is_automated_probability', '=', True)]}">
+                                            <i class="fa fa-gear" role="img" title="Switch to automatic probability" aria-label="Switch to automatic probability"></i>
+                                        </button>
+                                        <small class="oe_grey" attrs="{'invisible': [('is_automated_probability', '=', True)]}"><field name="automated_probability"/> %%</small>
+                                    </span>
+                                    <div id="probability" class="o_row d-flex">
+                                        <field name="is_automated_probability" invisible="1"/>
+                                        <field name="probability" widget="float" class="oe_inline"/>
                                         <span class="oe_grey"> %%</span>
                                     </div>
                                 </div>
@@ -555,6 +582,8 @@
                                 />
                                 <field name="is_blacklisted" invisible="1"/>
                                 <field name="partner_is_blacklisted" invisible="1"/>
+                                <field name="email_state" invisible="1"/>
+                                <field name="phone_state" invisible="1"/>
                                 <label for="email_from" class="oe_inline"/>
                                 <div class="o_row o_row_readonly">
                                     <i class="fa fa-ban" style="color: red;" role="img" title="This email is blacklisted for mass mailing"
diff --git a/addons/crm/views/res_config_settings_views.xml b/addons/crm/views/res_config_settings_views.xml
index 408efb08bf3..ed20f377978 100644
--- a/addons/crm/views/res_config_settings_views.xml
+++ b/addons/crm/views/res_config_settings_views.xml
@@ -51,6 +51,25 @@
                                 </div>
                             </div>
                         </div>
+                        <div class="col-12 col-lg-6 o_setting_box" title="This can be used to compute statistical probability to close a lead">
+                            <div class="o_setting_left_pane"></div>
+                            <div class="o_setting_right_pane">
+                                <b>Predictive Lead Scoring</b>
+                                <div class="text-muted">
+                                    Let statistical analysis determine the probability to close a lead
+                                </div>
+                                <div>
+                                    Consider leads created as of the
+                                    <field name="predictive_lead_scoring_start_date_str" invisible="1"/>
+                                    <field name="predictive_lead_scoring_start_date"/>
+                                </div>
+                                <div>
+                                    Consider following fields in computation
+                                    <field name="predictive_lead_scoring_fields" widget="many2many_checkboxes"/>
+                                    <field name="predictive_lead_scoring_fields_str" invisible="1"/>
+                                </div>
+                            </div>
+                        </div>
                     </div>
                     <h2>Lead Generation</h2>
                     <div class="row mt16 o_settings_container">