[IMP] doc: add information on the new populate feature.

* Cmdline interface: how to trigger database population
* Testing: how to implement database population on a given model.
  * autodocumentation of the population methods

+ improve population methods docstrings.

closes odoo/odoo#50596

Signed-off-by: Victor Feyens (vfe) <vfe@odoo.com>

4 files changed, 205 insertions, 16 deletions

diff --git a/doc/reference/cmdline.rst b/doc/reference/cmdline.rst
index e55b9147d92..71b52bce8bb 100644
--- a/doc/reference/cmdline.rst
+++ b/doc/reference/cmdline.rst
@@ -536,6 +536,10 @@ starting requirements are.
 
 Scaffolding is available via the :command:`odoo-bin scaffold` subcommand.
 
+.. code-block:: console
+
+    $ odoo_bin scaffold my_module /addons/
+
 .. option:: name (required)
 
     the name of the module to create, may munged in various manners to
@@ -551,8 +555,34 @@ Scaffolding is available via the :command:`odoo-bin scaffold` subcommand.
     a template directory, files are passed through jinja2_ then copied to
     the ``destination`` directory
 
+
+This will create module *my_module* in directory */addons/*.
+
+.. _reference/cmdline/populate:
+
+Database Population
+===================
+
+.. program:: odoo-bin populate
+
+Odoo CLI supports database population features. If the feature is
+:ref:`implemented on a given model<reference/testing/populate/methods>`, it allows automatic data
+generation of the model's records to test your modules in databases containing non-trivial amounts of records.
+
 .. code-block:: console
 
-    $ odoo_bin scaffold my_module /addons/
+    $ odoo_bin populate
 
-This will create module *my_module* in directory */addons/*.
+.. option:: --models
+
+    list of models for which the database should be filled
+
+.. option:: --size (small|medium|large)
+
+    population size, the actual records number depends on the model's `_populate_sizes` attribute.
+    The generated records content is specified by the :meth:`~odoo.models._populate_factories` method
+    of a given model (cf. the :file:`populate` folder of modules for further details).
+
+.. seealso::
+
+    :ref:`reference/testing/populate`
diff --git a/doc/reference/testing.rst b/doc/reference/testing.rst
index 55c229fd041..346f369a828 100644
--- a/doc/reference/testing.rst
+++ b/doc/reference/testing.rst
@@ -526,6 +526,105 @@ Two new command line arguments were added since Odoo 13.0 to control this behavi
 :option:`--screenshots <odoo-bin --screenshots>` and :option:`--screencasts <odoo-bin --screencasts>`
 
 
+Performance Testing
+===================
+
+Query counts
+------------
+
+One of the ways to test performance is to measure database queries. Manually, this can be tested with the
+`--log-sql` CLI parameter. If you want to establish the maximum number of queries for an operation,
+you can use the :meth:`~odoo.tests.common.BaseCase.assertQueryCount` method, integrated in Odoo test classes.
+
+.. code-block:: python
+
+    with self.assertQueryCount(11):
+        do_something()
+
+.. _reference/testing/populate:
+
+Database population
+-------------------
+
+Odoo CLI offers a :ref:`database population<reference/cmdline/populate>` feature.
+
+.. code-block:: console
+
+    odoo-bin populate
+
+Instead of the tedious manual, or programmatic, specification of test data,
+one can use this feature to fill a database on demand with the desired number of test data.
+This can be used to detect diverse bugs or performance issues in tested flows.
+
+To specify this feature for a given model, the following methods and attributes can be defined.
+
+.. _reference/testing/populate/methods:
+
+.. currentmodule:: odoo.models
+
+.. autoattribute:: Model._populate_sizes
+.. autoattribute:: Model._populate_dependencies
+.. automethod:: Model._populate
+.. automethod:: Model._populate_factories
+
+.. note::
+
+    You have to define at least :meth:`~odoo.models.Model._populate` or :meth:`~odoo.models.Model._populate_factories`
+    on the model to enable database population.
+
+Example model
+^^^^^^^^^^^^^
+
+.. code-block:: python
+
+    from odoo.tools import populate
+
+    class CustomModel(models.Model)
+        _inherit = "custom.some_model"
+        _populate_sizes = {"small": 100, "medium": 2000, "large": 10000}
+        _populate_dependencies = ["custom.some_other_model"]
+
+        def _populate_factories(self):
+            # Record ids of previously populated models are accessible in the registry
+            some_other_ids = self.env.registry.populated_models["custom.some_other_model"]
+
+            def get_some_field(values=None, random=None, **kwargs):
+                """ Choose a value for some_field depending on other fields values.
+
+                    :param dict values:
+                    :param random: seeded :class:`random.Random` object
+                """
+                field_1 = values['field_1']
+                if field_1 in [value2, value3]:
+                    return random.choice(some_field_values)
+                return False
+
+            return [
+                ("field_1", populate.randomize([value1, value2, value3])),
+                ("field_2", populate.randomize([value_a, value_b], [0.5, 0.5])),
+                ("some_other_id", populate.randomize(some_other_ids)),
+                ("some_field", populate.compute(get_some_field, seed="some_field")),
+                ('active', populate.cartesian([True, False])),
+            ]
+
+        def _populate(self, size):
+            records = super()._populate(size)
+
+            # If you want to update the generated records
+            # E.g setting the parent-child relationships
+            records.do_something()
+
+            return records
+
+Population tools
+^^^^^^^^^^^^^^^^
+
+Multiple population tools are available to easily create
+the needed data generators.
+
+.. automodule:: odoo.tools.populate
+    :members: cartesian, compute, constant, iterate, randint, randomize
+
 .. _qunit: https://qunitjs.com/
 .. _qunit_config.js: https://github.com/odoo/odoo/blob/51ee0c3cb59810449a60dae0b086b49b1ed6f946/addons/web/static/tests/helpers/qunit_config.js#L49
 .. _web.tests_assets: https://github.com/odoo/odoo/blob/51ee0c3cb59810449a60dae0b086b49b1ed6f946/addons/web/views/webclient_templates.xml#L594
diff --git a/odoo/models.py b/odoo/models.py
index 82c5d278bc9..c6b757bd304 100644
--- a/odoo/models.py
+++ b/odoo/models.py
@@ -6150,22 +6150,41 @@ Record ids: %(records)s
         return 'placeholder.png'
 
     def _populate_factories(self):
-        """ Return a list of pairs ``(field_name, factory)``, ``factory`` is a function that creates
-        a generator of values (dict of field values).  Its API suggested by::
+        """ Generates a factory for the different fields of the model.
 
-            # generator yield values by iterating on base_generator and adding a value
-            # for the given field in the dicts yielded by base_generator
-            generator = factory(base_generator, field_name, model_name)
-            # suggested usage
-            for values in generator:
-                model.create(values)
+        ``factory`` is a generator of values (dict of field values).
+
+        Factory skeleton::
+
+            def generator(iterator, field_name, model_name):
+                for counter, values in enumerate(iterator):
+                    # values.update(dict())
+                    yield values
+
+        See :mod:`odoo.tools.populate` for population tools and applications.
+
+        :returns: list of pairs(field_name, factory) where `factory` is a generator function.
+        :rtype: list(tuple(str, generator))
+
+        .. note::
+
+            It is the responsibility of the generator to handle the field_name correctly.
+            The generator could generate values for multiple fields together. In this case,
+            the field_name should be more a "field_group", covering the different fields
+            updated by the generator (e.g. "_address" for a generator updating multiple address fields).
         """
         return []
 
     @property
     def _populate_sizes(self):
         """ Return a dict mapping symbolic sizes (``'small'``, ``'medium'``, ``'large'``) to integers,
-        giving the minimal number of records that method ``_populate`` should create.
+        giving the minimal number of records that :meth:`_populate` should create.
+
+        The default population sizes are:
+
+        * ``small`` : 10
+        * ``medium`` : 100
+        * ``large`` : 1000
         """
         return {
             'small': 10,  # minimal representative set
@@ -6175,12 +6194,16 @@ Record ids: %(records)s
 
     @property
     def _populate_dependencies(self):
+        """ Return the list of models which have to be populated before the current one.
+
+        :rtype: list
+        """
         return []
 
     def _populate(self, size):
         """ Create records to populate this model.
-        
-        :param size: symbolic size for the number of records: ``'small'``, ``'medium'`` or ``'large'``
+
+        :param str size: symbolic size for the number of records: ``'small'``, ``'medium'`` or ``'large'``
         """
         batch_size = 1000
         min_size = self._populate_sizes[size]
diff --git a/odoo/tools/populate.py b/odoo/tools/populate.py
index 2b1ae521696..d44092a9378 100644
--- a/odoo/tools/populate.py
+++ b/odoo/tools/populate.py
@@ -17,7 +17,7 @@ def format_str(val, counter, values):
     return val
 
 
-def chain_factories(field_factories, model_name): 
+def chain_factories(field_factories, model_name):
     """ Instanciate a generator by calling all the field factories. """
     generator = root_factory()
     for (fname, field_factory) in field_factories:
@@ -35,6 +35,14 @@ def root_factory():
 def randomize(vals, weights=None, seed=False, formatter=format_str, counter_offset=0):
     """ Return a factory for an iterator of values dicts with pseudo-randomly
     chosen values (among ``vals``) for a field.
+
+    :param list vals: list in which a value will be chosen, depending on `weights`
+    :param list weights: list of probabilistic weights
+    :param seed: optional initialization of the random number generator
+    :param function formatter: (val, counter, values) --> formatted_value
+    :param int counter_offset:
+    :returns: function of the form (iterator, field_name, model_name) -> values
+    :rtype: function (iterator, str, str) -> dict
     """
     def generate(iterator, field_name, model_name):
         r = Random('%s+field+%s' % (model_name, seed or field_name))
@@ -48,6 +56,14 @@ def randomize(vals, weights=None, seed=False, formatter=format_str, counter_offs
 def cartesian(vals, weights=None, seed=False, formatter=format_str, then=None):
     """ Return a factory for an iterator of values dicts that combines all ``vals`` for
     the field with the other field values in input.
+
+    :param list vals: list in which a value will be chosen, depending on `weights`
+    :param list weights: list of probabilistic weights
+    :param seed: optional initialization of the random number generator
+    :param function formatter: (val, counter, values) --> formatted_value
+    :param function then: if defined, factory used when vals has been consumed.
+    :returns: function of the form (iterator, field_name, model_name) -> values
+    :rtype: function (iterator, str, str) -> dict
     """
     def generate(iterator, field_name, model_name):
         counter = 0
@@ -64,8 +80,16 @@ def cartesian(vals, weights=None, seed=False, formatter=format_str, then=None):
 
 def iterate(vals, weights=None, seed=False, formatter=format_str, then=None):
     """ Return a factory for an iterator of values dicts that picks a value among ``vals``
-    for each input.  Once all ``vals`` have been used once, resume as a ``randomize``
-    generator.
+    for each input.  Once all ``vals`` have been used once, resume as ``then`` or as a
+    ``randomize`` generator.
+
+    :param list vals: list in which a value will be chosen, depending on `weights`
+    :param list weights: list of probabilistic weights
+    :param seed: optional initialization of the random number generator
+    :param function formatter: (val, counter, values) --> formatted_value
+    :param function then: if defined, factory used when vals has been consumed.
+    :returns: function of the form (iterator, field_name, model_name) -> values
+    :rtype: function (iterator, str, str) -> dict
     """
     def generate(iterator, field_name, model_name):
         counter = 0
@@ -83,6 +107,9 @@ def iterate(vals, weights=None, seed=False, formatter=format_str, then=None):
 def constant(val, formatter=format_str):
     """ Return a factory for an iterator of values dicts that sets the field
     to the given value in each input dict.
+
+    :returns: function of the form (iterator, field_name, model_name) -> values
+    :rtype: function (iterator, str, str) -> dict
     """
     def generate(iterator, field_name, _):
         for counter, values in enumerate(iterator):
@@ -95,6 +122,11 @@ def compute(function, seed=None):
     """ Return a factory for an iterator of values dicts that computes the field value
     as ``function(values, counter, random)``, where ``values`` is the other field values,
     ``counter`` is an integer, and ``random`` is a pseudo-random number generator.
+
+    :param function function: (values, counter, random) --> field_values
+    :param seed: optional initialization of the random number generator
+    :returns: function of the form (iterator, field_name, model_name) -> values
+    :rtype: function (iterator, str, str) -> dict
     """
     def generate(iterator, field_name, model_name):
         r = Random('%s+field+%s' % (model_name, seed or field_name))
@@ -107,6 +139,11 @@ def compute(function, seed=None):
 def randint(a, b, seed=None):
     """ Return a factory for an iterator of values dicts that sets the field
     to the random integer between a and b included in each input dict.
+
+    :param int a: minimal random value
+    :param int b: maximal random value
+    :returns: function of the form (iterator, field_name, model_name) -> values
+    :rtype: function (iterator, str, str) -> dict
     """
     def get_rand_int(random=None, **kwargs):
         random.randint(a, b)