[IMP] core, base: add tooling to populate database

Some use case like testing performance or upgrade scripts required a database
with prefilled data, covering basic corner cases. A solution can be to
create data a procedural way.

This commit proposes an API to easily populate a model, usually by giving a
list of possible values for each field or by giving a compute method that will
be based on raw values of other fields.

The basic way to define how to populate a new field is to override `_populate_factories`,
a method that returns a sequence of pairs `(field_name, factory)`.

The definition of a field is a "factory", a function that returns a neverending iterator
combining its value(s) with the values of the iterator given in parameter.
Some factory helpers are given in `tools.populate.py`:
- `iterate(vals, weighs)` ensures that one record is created for each value
by iterating on them, then resumes as `random.choice` on those vals following weights
once the first iteration is finished.
- `cartesian(vals, weights)` makes a cartesian product of its own values with the values
of its input iterator, then resumes as a randomized generator.
- `compute(function)` calls the given function with the current values dict and a random object,
and assigns the current field to the returned value.
- ...

Each iterator yields dictionaries of field values, and the factory should add a
value for the current field(s).  The yielded dictionaries also contain a pseudo_field
`"__complete"`, that indicates whether this step is some randomized data
to reach the expected count of records.  A falsy value indicates that the iterator
is still covering mandatory cases.  This indicates whether a cartesian product is
finished, or an `iterate` has consumed all its values.

The order of the factories is quite important, since some computed fields may need
other fields to be defined, and `cartesian` factories should always be at the beginning
to avoid having too many combination.  That is why the factories are given as a list of
pairs instead of a dictionary; this makes it easier to insert elements at any place.

Example:
field A: cartesian([T, F])
field B: cartesian([0, 1])
field C: iterate([a, b, c, d, e])
field D: compute(1-B)

_c is shortcut for __complete
_ is a random value, or result of a random value

```
iter | root  | field A  | field B | field C     | field D   | result
0     {_c:F}  {... A:T}  {... B:0} {...C:a}      {...D:1}    T,0,a,1 complete:False
                         {... B:1} {...C:b}      {...D:0}    T,1,b,0 complete:False
              {... A:F}  {... B:0} {...C:c}      {...D:1}    F,0,c,1 complete:False
                         {... B:1} {...C:d}      {...D:0}    F,1,d,0 complete:False
1     {_c:T}  {... A:_}  {... B:_} {...C:e,_c:F} {...D:_}    _,_,e,_ complete:False
2     {_c:T}  {... A:_}  {... B:_} {...C:_}      {...D:_}    _,_,_,_ complete:True
```

X-original-commit: 4c0182dafa584853ed83a166096f45c33c06a245

6 files changed, 275 insertions, 0 deletions

diff --git a/odoo/addons/test_populate/__init__.py b/odoo/addons/test_populate/__init__.py
new file mode 100644
index 00000000000..a0fdc10fe11
--- /dev/null
+++ b/odoo/addons/test_populate/__init__.py
@@ -0,0 +1,2 @@
+# -*- coding: utf-8 -*-
+from . import models
diff --git a/odoo/addons/test_populate/__manifest__.py b/odoo/addons/test_populate/__manifest__.py
new file mode 100644
index 00000000000..a19d973a142
--- /dev/null
+++ b/odoo/addons/test_populate/__manifest__.py
@@ -0,0 +1,13 @@
+# -*- coding: utf-8 -*-
+{
+    'name': 'test-populate',
+    'version': '0.1',
+    'category': 'Tests',
+    'description': """A module to test populate.""",
+    'depends': ['base'],
+    'data': [
+        'ir.model.access.csv',
+    ],
+    'installable': True,
+    'auto_install': False,
+}
diff --git a/odoo/addons/test_populate/ir.model.access.csv b/odoo/addons/test_populate/ir.model.access.csv
new file mode 100644
index 00000000000..4be68aa1702
--- /dev/null
+++ b/odoo/addons/test_populate/ir.model.access.csv
@@ -0,0 +1,5 @@
+"id","name","model_id:id","group_id:id","perm_read","perm_write","perm_create","perm_unlink"
+access_test_populate,access_test_populate,model_test_populate,,1,1,1,1
+access_test_populate_category,access_test_populate_category,model_test_populate_category,,1,1,1,1
+access_test_no_populat,access_test_no_populat,model_test_no_populate,,1,1,1,1
+access_test_populate_inherit,access_test_populate_inherit,model_test_populate_inherit,,1,1,1,1
diff --git a/odoo/addons/test_populate/models.py b/odoo/addons/test_populate/models.py
new file mode 100644
index 00000000000..d59524d185b
--- /dev/null
+++ b/odoo/addons/test_populate/models.py
@@ -0,0 +1,94 @@
+# -*- coding: utf-8 -*-
+# Part of Odoo. See LICENSE file for full copyright and licensing details.
+
+from odoo import models, fields, api
+
+from odoo.tools import populate, pycompat
+
+
+class TestPopulateModel(models.Model):
+    _name = 'test.populate'
+    _description = 'Test Populate'
+
+    name = fields.Char(default='Foo')
+    state = fields.Selection([('a', 'A'), ('b', 'B')], default='a')
+    active = fields.Boolean('Active', default=True)
+    category_id = fields.Many2one('test.populate.category', 'Category')
+    some_ref = fields.Integer('Reference')
+    dependant_field_1 = fields.Char('Dependant 1')
+    dependant_field_2 = fields.Char('Dependant 2')
+
+    _populate_dependencies = ['test.populate.category']
+
+    _populate_sizes = {
+        'small': 20,
+        'medium': 30,
+        'large': 100,
+    }
+
+    def _populate_factories(self):
+
+        # cross dependant field in a sub generator, cartesian product of two fields
+        dependant_factories = [
+            ('dependant_field_1', populate.cartesian(['d1_1', 'd1_2'])),
+            ('dependant_field_2', populate.cartesian(['d2_1', 'd2_2', 'd2_3_{counter}'])),
+        ]
+        def generate_dependant(iterator, *args):
+            dependants_generator = populate.chain_factories(dependant_factories, self._name)
+            for values in dependants_generator:
+                dependant_values = next(iterator)
+                yield {**values, **dependant_values, '__complete': values['__complete'] and dependant_values['__complete']}
+
+        def get_name(values=None, counter=0, **kwargs):
+            active = 'active' if values['active'] else 'inactive'
+            cat = 'filling' if values['__complete'] else 'corner'
+            return '%s_%s_%s' % (active, cat, counter)
+
+        category_ids = self.env.registry.populated_models['test.populate.category']
+
+        return [
+            ('active', populate.cartesian([True, False], [3, 1])),
+            ('state', populate.cartesian([False] + self.env['test.populate']._fields['state'].get_values(self.env))),
+            ('some_ref', populate.iterate([False, 1, 2, 3, 4])),
+            ('_dependant', generate_dependant),
+            ('name', populate.compute(get_name)),
+            ('category_id', populate.randomize([False] + category_ids)),
+        ]
+
+class TestPopulateDependencyModel(models.Model):
+    _name = 'test.populate.category'
+    _description = 'Test Populate Category'
+
+    _populate_sizes = {
+        'small': 3,
+        'medium': 10,
+        'large': 20,
+    }
+    name = fields.Char('Name', required=True, default='Cat1')
+    active = fields.Boolean('Active', default=True)
+
+    def _populate_factories(self):
+        return [
+            ('active', populate.cartesian([True, False], [9, 1])),
+            ('name', populate.cartesian(['Cat1', 'Cat2', 'Cat3'])),
+        ]
+
+class TestNoPopulateModelInherit(models.Model):
+    _name = 'test.populate.inherit'
+    _inherit = 'test.populate'
+
+    _description = 'Test populate inherit'
+
+    additionnal_field = fields.Char(required=True)
+
+    def _populate_factories(self):
+        return super()._populate_factories() + [
+            ('additionnal_field', populate.iterate(['V1', 'V2', 'V3'])),
+        ]
+
+
+class TestNoPopulateModel(models.Model):
+    _name = 'test.no.populate'
+    _description = 'A model with no populate method and a required field, should not crash'
+
+    name = fields.Char(required=True)
diff --git a/odoo/addons/test_populate/tests/__init__.py b/odoo/addons/test_populate/tests/__init__.py
new file mode 100644
index 00000000000..a77cffb103d
--- /dev/null
+++ b/odoo/addons/test_populate/tests/__init__.py
@@ -0,0 +1,4 @@
+# -*- coding: utf-8 -*-
+
+from . import test_populate
+
diff --git a/odoo/addons/test_populate/tests/test_populate.py b/odoo/addons/test_populate/tests/test_populate.py
new file mode 100644
index 00000000000..09243fefdea
--- /dev/null
+++ b/odoo/addons/test_populate/tests/test_populate.py
@@ -0,0 +1,157 @@
+# -*- coding: utf-8 -*-
+# Part of Odoo. See LICENSE file for full copyright and licensing details.
+import collections
+import logging
+from odoo.tests import common
+from odoo.cli.populate import Populate
+from odoo.tools import mute_logger, populate
+try:
+    from unittest.mock import patch
+except ImportError:
+    from mock import patch
+
+
+_logger = logging.getLogger(__name__)
+
+# todo patch cursor commit
+class TestPopulate(common.TransactionCase):
+    def setUp(self):
+        super(TestPopulate, self).setUp()
+        patcher = patch.object(self.cr, 'commit')
+        patcher.start()
+        self.addCleanup(patcher.stop)
+
+    def test_dependency(self):
+        ordered_models = Populate._get_ordered_models(self.env, ['test.populate'])
+        ordered_models_names = [model._name for model in ordered_models]
+        self.assertEqual(ordered_models_names, ['test.populate.category', 'test.populate'])
+
+    @mute_logger('odoo.cli.populate')
+    def test_no_populate(self):
+        """ Check that model with no populate method are not populated"""
+        model = 'test.no.populate'
+        populated = Populate.populate(self.env, 'small', [model])
+        new = populated[model]
+        self.assertFalse(new)
+
+    @mute_logger('odoo.cli.populate')
+    def test_populate(self):
+        """ Check that model with populate methods are correctly populated"""
+        model = 'test.populate'
+        populated = Populate.populate(self.env, 'small', [model])
+        records = self.check_test_populate_values(populated, model)
+
+        # pseudo random after cartesian with ~ 1/4 False, 3/4 True
+        # seed is model dependant
+        self.assertEqual(records.mapped('active')[6:20], [
+            True, True, True, True, True, True, False, False, True, True, True, True, False, True
+        ])
+        # pseudo random after iterate
+        self.assertEqual(records.mapped('some_ref')[5:20], [
+            1, 0, 2, 4, 4, 3, 4, 1, 2, 2, 2, 4, 4, 1, 2
+        ])
+
+    @mute_logger('odoo.cli.populate')
+    def test_populate_inherit(self):
+        """ Check that model with populate methods are correctly populated"""
+        model = 'test.populate.inherit'
+        populated = Populate.populate(self.env, 'small', [model])
+        records = self.check_test_populate_values(populated, model)  # should be same values as base class
+        # and additionnal_field has own values set
+        # iterate then pseudo random
+        self.assertEqual(records.mapped('additionnal_field')[:20], [
+            'V1', 'V2', 'V3',  # iterate
+            'V3', 'V1', 'V2', 'V1', 'V2', 'V1', 'V2', 'V2', 'V2', 'V1', 'V1', 'V3', 'V1', 'V2', 'V2', 'V3', 'V2'  # pseudorandom
+        ])
+
+    def check_test_populate_values(self, populated, model):
+        new = populated[model]
+        self.assertTrue(new)
+        records = self.env[model].browse(new)
+            # main cartesian product
+        self.assertEqual(records.mapped('active')[:6], [
+            True, True, True,
+            False, False, False,
+        ])
+        self.assertEqual(records.mapped('state')[:6], [
+            False, 'a', 'b',
+            False, 'a', 'b',
+        ])
+
+        # custom name call
+        self.assertEqual(records.mapped('name')[:6], [
+            'active_corner_0', 'active_corner_1', 'active_corner_2',
+            'inactive_corner_3', 'inactive_corner_4', 'inactive_corner_5',
+        ])
+        self.assertIn('filling', records.mapped('name')[6]) # filling when cartesian and iterate are done
+        # iterate then pseudo random
+        self.assertEqual(records.mapped('some_ref')[:5], [
+            0, 1, 2, 3, 4 # iterate
+        ])
+
+        # some custom multi field generator (as cartesian product in this example)
+        self.assertEqual(records.mapped('dependant_field_1')[:6], [
+            'd1_1', 'd1_1', 'd1_1',
+            'd1_2', 'd1_2', 'd1_2'
+        ])
+        self.assertEqual(records.mapped('dependant_field_2')[:6], [
+            'd2_1', 'd2_2', 'd2_3_0',
+            'd2_1', 'd2_2', 'd2_3_1'
+        ])
+        used_category_ids = set(records.mapped('category_id').ids[:20])
+        self.assertEqual(len(used_category_ids), 6) # event if id may change, with given seed, the 6 category are used
+        generated_category_ids = set(populated['test.populate.category'])
+        self.assertFalse(used_category_ids-generated_category_ids) # all category are the generated one
+        self.assertFalse(hasattr(self.env.registry, 'populated_models'), 'populated_models flag has been removed from registry')
+
+        return records
+
+
+@common.tagged('-at_install', 'post_install')
+class TestPopulateValidation(common.TransactionCase):
+    """ check that all fields in _populate_factories exists """
+    def setUp(self):
+        super(TestPopulateValidation, self).setUp()
+        self.env.registry.populated_models = collections.defaultdict(list)
+        self.addCleanup(delattr, self.env.registry, 'populated_models')
+
+    def test_populate_factories(self):
+        for model in self.env.values():
+            factories = model._populate_factories() or []
+            factories_fields = set([field_name for field_name, factory in factories if not field_name.startswith('_')])
+            missing = factories_fields - model._fields.keys()
+            self.assertFalse(missing, 'Fields %s not found in model %s' % (missing, model._name))
+
+
+@common.tagged('-standard', '-at_install', 'post_install', 'missing_populate')
+class TestPopulateMissing(common.TransactionCase):
+    """ check that all fields in _populate_factories exists """
+    def setUp(self):
+        super(TestPopulateMissing, self).setUp()
+        self.env.registry.populated_models = collections.defaultdict(list)
+        self.addCleanup(delattr, self.env.registry, 'populated_models')
+
+    def test_populate_missing_factories(self):
+        no_factory_models = []
+        for model in self.env.values():
+            factories = model._populate_factories()
+            if not factories:
+                if model._transient or model._abstract:
+                    continue
+                ir_model = self.env['ir.model'].search([('model', '=', model._name)])
+                if all(module.startswith('test_') for module in ir_model.modules.split(',')):
+                    continue
+                no_factory_models.append(model._name)
+            else:
+                factories_fields = next(populate.chain_factories(factories, model._name)).keys()
+                def is_electable(field):
+                    return not field.compute \
+                        and field.store \
+                        and field.name not in ('create_uid', 'write_uid', 'write_date', 'create_date', 'id') \
+                        and field.type not in ('many2many', 'one2many')
+                electable_fields = set([key for key, field in model._fields.items() if is_electable(field)])
+                no_factory_fields = set(electable_fields - factories_fields)
+                if no_factory_fields:
+                    _logger.info('Model %s has some undefined field: %s', model._name, no_factory_fields)
+
+        _logger.info('No populate factories defiend for %s', no_factory_models)
\ No newline at end of file