[IMP] core, base: add tooling to populate database

Some use case like testing performance or upgrade scripts required a database
with prefilled data, covering basic corner cases. A solution can be to
create data a procedural way.

This commit proposes an API to easily populate a model, usually by giving a
list of possible values for each field or by giving a compute method that will
be based on raw values of other fields.

The basic way to define how to populate a new field is to override `_populate_factories`,
a method that returns a sequence of pairs `(field_name, factory)`.

The definition of a field is a "factory", a function that returns a neverending iterator
combining its value(s) with the values of the iterator given in parameter.
Some factory helpers are given in `tools.populate.py`:
- `iterate(vals, weighs)` ensures that one record is created for each value
by iterating on them, then resumes as `random.choice` on those vals following weights
once the first iteration is finished.
- `cartesian(vals, weights)` makes a cartesian product of its own values with the values
of its input iterator, then resumes as a randomized generator.
- `compute(function)` calls the given function with the current values dict and a random object,
and assigns the current field to the returned value.
- ...

Each iterator yields dictionaries of field values, and the factory should add a
value for the current field(s).  The yielded dictionaries also contain a pseudo_field
`"__complete"`, that indicates whether this step is some randomized data
to reach the expected count of records.  A falsy value indicates that the iterator
is still covering mandatory cases.  This indicates whether a cartesian product is
finished, or an `iterate` has consumed all its values.

The order of the factories is quite important, since some computed fields may need
other fields to be defined, and `cartesian` factories should always be at the beginning
to avoid having too many combination.  That is why the factories are given as a list of
pairs instead of a dictionary; this makes it easier to insert elements at any place.

Example:
field A: cartesian([T, F])
field B: cartesian([0, 1])
field C: iterate([a, b, c, d, e])
field D: compute(1-B)

_c is shortcut for __complete
_ is a random value, or result of a random value

```
iter | root  | field A  | field B | field C     | field D   | result
0     {_c:F}  {... A:T}  {... B:0} {...C:a}      {...D:1}    T,0,a,1 complete:False
                         {... B:1} {...C:b}      {...D:0}    T,1,b,0 complete:False
              {... A:F}  {... B:0} {...C:c}      {...D:1}    F,0,c,1 complete:False
                         {... B:1} {...C:d}      {...D:0}    F,1,d,0 complete:False
1     {_c:T}  {... A:_}  {... B:_} {...C:e,_c:F} {...D:_}    _,_,e,_ complete:False
2     {_c:T}  {... A:_}  {... B:_} {...C:_}      {...D:_}    _,_,_,_ complete:True
```

X-original-commit: 4c0182dafa584853ed83a166096f45c33c06a245

2 files changed, 117 insertions, 1 deletions

diff --git a/odoo/tools/config.py b/odoo/tools/config.py
index efcd71ccf57..65e22cc37af 100644
--- a/odoo/tools/config.py
+++ b/odoo/tools/config.py
@@ -352,9 +352,10 @@ class configmanager(object):
 
             odoo.tools.config.parse_config(sys.argv[1:])
         """
-        self._parse_config(args)
+        opt = self._parse_config(args)
         odoo.netsvc.init_logger()
         odoo.modules.module.initialize_sys_path()
+        return opt
 
     def _parse_config(self, args=None):
         if args is None:
@@ -529,6 +530,8 @@ class configmanager(object):
             m.strip() for m in self.options['server_wide_modules'].split(',') if m.strip()
         ]
 
+        return opt
+
     def _is_addons_path(self, path):
         from odoo.modules.module import MANIFEST_NAMES
         for f in os.listdir(path):
diff --git a/odoo/tools/populate.py b/odoo/tools/populate.py
new file mode 100644
index 00000000000..ddac293eabf
--- /dev/null
+++ b/odoo/tools/populate.py
@@ -0,0 +1,113 @@
+import random
+from odoo.tools import pycompat
+
+
+def Random(seed):
+    """ Return a random number generator object with the given seed. """
+    r = random.Random()
+    r.seed(seed, version=2)
+    return r
+
+
+def format_str(val, counter, values):
+    """ Format the given value (with method ``format``) when it is a string. """
+    if isinstance(val, str):
+        return val.format(counter=counter, values=values)
+    return val
+
+
+def chain_factories(field_factories, model_name): 
+    """ Instanciate a generator by calling all the field factories. """
+    generator = root_factory()
+    for (fname, field_factory) in field_factories:
+        generator = field_factory(generator, fname, model_name)
+    return generator
+
+
+def root_factory():
+    """ Return a generator with empty values dictionaries (except for the flag ``__complete``). """
+    yield {'__complete': False}
+    while True:
+        yield {'__complete': True}
+
+
+def randomize(vals, weights=None, seed=False, formatter=format_str, counter_offset=0):
+    """ Return a factory for an iterator of values dicts with pseudo-randomly
+    chosen values (among ``vals``) for a field.
+    """
+    def generate(iterator, field_name, model_name):
+        r = Random('%s+field+%s' % (model_name, seed or field_name))
+        for counter, values in enumerate(iterator):
+            val = r.choices(vals, weights)[0]
+            values[field_name] = formatter(val, counter + counter_offset, values)
+            yield values
+    return generate
+
+
+def cartesian(vals, weights=None, seed=False, formatter=format_str, then=None):
+    """ Return a factory for an iterator of values dicts that combines all ``vals`` for
+    the field with the other field values in input.
+    """
+    def generate(iterator, field_name, model_name):
+        counter = 0
+        for values in iterator:
+            if values['__complete']:
+                break  # will consume and lose an element, (complete so a filling element). If it is a problem, use peekable instead.
+            for val in vals:
+                yield {**values, field_name: formatter(val, counter, values)}
+            counter += 1
+        factory = then or randomize(vals, weights, seed, formatter, counter)
+        yield from factory(iterator, field_name, model_name)
+    return generate
+
+
+def iterate(vals, weights=None, seed=False, formatter=format_str, then=None):
+    """ Return a factory for an iterator of values dicts that picks a value among ``vals``
+    for each input.  Once all ``vals`` have been used once, resume as a ``randomize``
+    generator.
+    """
+    def generate(iterator, field_name, model_name):
+        counter = 0
+        for val in vals: # iteratable order is important, shortest first
+            values = next(iterator)
+            values[field_name] = formatter(val, counter, values)
+            values['__complete'] = False
+            yield values
+            counter += 1
+        factory = then or randomize(vals, weights, seed, formatter, counter)
+        yield from factory(iterator, field_name, model_name)
+    return generate
+
+
+def constant(val, formatter=format_str):
+    """ Return a factory for an iterator of values dicts that sets the field
+    to the given value in each input dict.
+    """
+    def generate(iterator, field_name, _):
+        for counter, values in enumerate(iterator):
+            values[field_name] = formatter(val, counter, values)
+            yield values
+    return generate
+
+
+def compute(function, seed=None):
+    """ Return a factory for an iterator of values dicts that computes the field value
+    as ``function(values, counter, random)``, where ``values`` is the other field values,
+    ``counter`` is an integer, and ``random`` is a pseudo-random number generator.
+    """
+    def generate(iterator, field_name, model_name):
+        r = Random('%s+field+%s' % (model_name, seed or field_name))
+        for counter, values in enumerate(iterator):
+            val = function(values=values, counter=counter, random=r)
+            values[field_name] = val
+            yield values
+    return generate
+
+
+def randint(a, b, seed=None):
+    """ Return a factory for an iterator of values dicts that sets the field
+    to the random integer between a and b included in each input dict.
+    """
+    def get_rand_int(random=None, **kwargs):
+        random.randint(a, b)
+    return compute(get_rand_int, seed=seed)