Odoo/odoo/api.py, branch 14.0 The current checkout of upstream Odoo [IMP] core: smaller and faster field cache 2020-07-24T08:30:00+00:00 Raphael Collet rco@odoo.com 2020-06-09T17:51:55+00:00 9b7a4e1815de580575bed6657f9f8915907ce534 Consider a context-dependent field and two possible cache structures for storing the data of that field for N records and K context keys. The common use-case is to assume that N is much larger than K. We look at how many dicts are used in each cache structure, where we label dicts with N entries as "large", and dicts with K entries as "small". Structure | Number of dicts ---------------------------+------------------- cache[field][id][key] | 1 large + N small cache[field][key][id] | 1 small + K large It is known that "small" dicts have a large relative overhead when compared to "large" dicts. Given this fact, the second cache structure will generate less memory overhead. Here are numbers for N=16, showing the size in bytes taken by the cache structure. Larger values of N lead to larger size differences, always in favor of the second structure. Structure | K=1 | K=2 | K=3 | K=4 | K=5 | K=6 -----------------------+------+------+------+------+------+------ cache[field][id][key] | 4488 | 4488 | 4488 | 4488 | 4488 | 6536 cache[field][key][id] | 888 | 1536 | 2184 | 2832 | 3480 | 4256 Moreover, some memory operations are performed on many records and a single context key. Those operations are clearly simpler and faster with the second cache structure. 
Consider a context-dependent field and two possible cache structures for
storing the data of that field for N records and K context keys.  The
common use-case is to assume that N is much larger than K.  We look at
how many dicts are used in each cache structure, where we label dicts
with N entries as "large", and dicts with K entries as "small".

     Structure                 | Number of dicts
    ---------------------------+-------------------
     cache[field][id][key]     | 1 large + N small
     cache[field][key][id]     | 1 small + K large

It is known that "small" dicts have a large relative overhead when
compared to "large" dicts.  Given this fact, the second cache structure
will generate less memory overhead.  Here are numbers for N=16, showing
the size in bytes taken by the cache structure.  Larger values of N lead
to larger size differences, always in favor of the second structure.

     Structure             |  K=1 |  K=2 |  K=3 |  K=4 |  K=5 |  K=6
    -----------------------+------+------+------+------+------+------
     cache[field][id][key] | 4488 | 4488 | 4488 | 4488 | 4488 | 6536
     cache[field][key][id] |  888 | 1536 | 2184 | 2832 | 3480 | 4256

Moreover, some memory operations are performed on many records and a
single context key.  Those operations are clearly simpler and faster
with the second cache structure.
 [FIX] core: prefetching of context-dependent fields 2020-06-03T13:00:36+00:00 Raphael Collet rco@odoo.com 2020-06-02T14:24:01+00:00 6d5da2d2d3f0630b507a37f0b282cfa4f3f86bfd Consider a context-dependent field, and successively access it on a recordset with different contexts. On the first context, the field is correctly computed in batch. After that, the field is always computed one by one. The bug is in the method that determines which records in a given set have no value in cache. On the first context, the cache is empty for the field, so all records are returned. After that, the method considers that all records have a value in cache: they do, but for another context key! Simply using the context key when looking up the cache fixes the issue. closes odoo/odoo#52360 X-original-commit: 35d69589d9b43afe6c6fc9779458323f7180153e Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com> Co-authored-by: Xavier-Do <xdo@odoo.com> 
Consider a context-dependent field, and successively access it on a
recordset with different contexts.  On the first context, the field is
correctly computed in batch.  After that, the field is always computed
one by one.

The bug is in the method that determines which records in a given set
have no value in cache.  On the first context, the cache is empty for
the field, so all records are returned.  After that, the method
considers that all records have a value in cache: they do, but for
another context key!  Simply using the context key when looking up the
cache fixes the issue.

closes odoo/odoo#52360

X-original-commit: 35d69589d9b43afe6c6fc9779458323f7180153e
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
Co-authored-by: Xavier-Do <xdo@odoo.com>
[REF] ir.autovacuum: declarative garbage collector registration 2020-05-19T13:38:19+00:00 Julien Castiaux juc@odoo.com 2020-03-17T11:58:14+00:00 c5d3a109f530cd0763a1c3860086b5b7abef1b64 The ir.autovacuum model purpose is to run several garbage collecting operations like removing files from the filestore when no attachment references them anymore. The precedent strategy to register new garbage collection tasks was to override the `power_on` method and to imperatively execute a vacuum cleaning method on a given model. All calls were executed in a single SQL transaction without any error handling, meaning a single fail during any call resulted in a complete failure of the entire vacuum cleaning chain. We introduce a new `@autovacuum` api decorator, its purpose it to register garbage collecting methods that will be safely executed in their own transaction by the vacuum cleaner. In order to ensure this new strategy is used, we deprecate `power_on` extensions. By the way, garbage-collecting methods can be quite heavy and we don't want users to directly call them. We now ensure they are private. closes odoo/odoo#47842 Task: 2154079 Signed-off-by: Raphael Collet (rco) <rco@openerp.com> Co-authored-by: Raphael Collet <rco@odoo.com> Co-authored-by: Olivier Dony <odo@odoo.com> 
The ir.autovacuum model purpose is to run several garbage collecting
operations like removing files from the filestore when no attachment
references them anymore.

The precedent strategy to register new garbage collection tasks was to
override the `power_on` method and to imperatively execute a vacuum
cleaning method on a given model. All calls were executed in a single
SQL transaction without any error handling, meaning a single fail during
any call resulted in a complete failure of the entire vacuum cleaning
chain.

We introduce a new `@autovacuum` api decorator, its purpose it to
register garbage collecting methods that will be safely executed in
their own transaction by the vacuum cleaner. In order to ensure this
new strategy is used, we deprecate `power_on` extensions.

By the way, garbage-collecting methods can be quite heavy and we don't
want users to directly call them. We now ensure they are private.

closes odoo/odoo#47842

Task: 2154079
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
Co-authored-by: Olivier Dony <odo@odoo.com>
[FIX] core, base: leftover deprecation warnings 2020-03-19T09:32:21+00:00 Xavier Morel xmo@odoo.com 2020-03-13T11:05:57+00:00 fe376d50d0c6da332a5ea2540c95583931dd2203 * from collections import <ABC> is deprecated, unclear why the deprecation warning didn't appear before (possibly only appears in 3.7/3.8?) either way `collections.abc` should be 3.3+ so switch everything to it. * add some more ignores on third-party packages deprecation warnings (meh) * while at it, mitigate generation of non-breaking space on some versions of Babel (in the french locale used by our tests anyway) closes odoo/odoo#47581 Related: odoo/enterprise#9214 Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com> 
* from collections import <ABC> is deprecated, unclear why the
  deprecation warning didn't appear before (possibly only appears in
  3.7/3.8?) either way `collections.abc` should be 3.3+ so switch
  everything to it.
* add some more ignores on third-party packages deprecation
  warnings (meh)
* while at it, mitigate generation of non-breaking space on some
  versions of Babel (in the french locale used by our tests anyway)

closes odoo/odoo#47581

Related: odoo/enterprise#9214
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
[FIX] doc: api.model no longer accept traditional style call 2020-02-25T12:06:06+00:00 Denis Ledoux dle@odoo.com 2020-02-25T09:51:59+00:00 c5b9bccf6b1854b3d04f6415ecfc3755ed8ab959 Since we dropped the old-api style compatibility layer, some years ago. This docstring was added to the 13.0 documentation thanks to odoo/odoo#44838 So it's best not to mention this old-api style in our latest documentation closes odoo/odoo#46223 X-original-commit: 63691c78006fbf7077eb13f3045be21a18bbe753 Signed-off-by: Denis Ledoux (dle) <dle@odoo.com> 
Since we dropped the old-api style compatibility layer,
some years ago.

This docstring was added to the 13.0 documentation
thanks to odoo/odoo#44838
So it's best not to mention this old-api style
in our latest documentation

closes odoo/odoo#46223

X-original-commit: 63691c78006fbf7077eb13f3045be21a18bbe753
Signed-off-by: Denis Ledoux (dle) <dle@odoo.com>
[FIX] api, product: search by pricelist 2020-02-11T11:05:09+00:00 Nicolas Martinelli nim@odoo.com 2020-02-07T09:55:23+00:00 12af185961de047f7f9399826f393aff2019c73b - Activate variants and pricelists - Go to Product > Products Variants - Search for anything on the 'Pricelist' filter A traceback is raised: 'Can only create cache keys from hashable values...'. This comes from the following change: https://github.com/odoo/odoo/blob/4b06fe19fa68255b7982d15e5847da2f6d6209fd/addons/web/static/src/js/views/control_panel/control_panel_model.js#L962 It returns a list instead of a string. Since a list is not hashable, it causes the issue. There are not much solutions since the context key `pricelist` can be a `list` or an `int` (an ID). We force the cache key conversion to a tuple to avoid the `TypeError` and handle the `list` use case (which was broken on top of crashing). opw-2187757 closes odoo/odoo#45064 X-original-commit: ec0a8767143b4c4818418281803c4f52fefa8d2a Signed-off-by: Nicolas Martinelli (nim) <nim@odoo.com> 
- Activate variants and pricelists
- Go to Product > Products Variants
- Search for anything on the 'Pricelist' filter

A traceback is raised: 'Can only create cache keys from hashable
values...'.

This comes from the following change:

https://github.com/odoo/odoo/blob/4b06fe19fa68255b7982d15e5847da2f6d6209fd/addons/web/static/src/js/views/control_panel/control_panel_model.js#L962

It returns a list instead of a string. Since a list is not hashable, it
causes the issue.

There are not much solutions since the context key `pricelist` can be a
`list` or an `int` (an ID). We force the cache key conversion to a tuple
to avoid the `TypeError` and handle the `list` use case (which was
broken on top of crashing).

opw-2187757

closes odoo/odoo#45064

X-original-commit: ec0a8767143b4c4818418281803c4f52fefa8d2a
Signed-off-by: Nicolas Martinelli (nim) <nim@odoo.com>
[IMP] core: improve mapped and filtered performance 2020-01-21T14:06:50+00:00 Adrian Torres adt@odoo.com 2019-12-11T10:09:13+00:00 1d13928764defd91d4ec2b2ce9af238a591c7c30 Previously, mapped was following a very naive approach, which was simply calling the field name passed as input for every record in a recordset, sequentially. The problem with this approach is that we will potentially recompute the same fields multiple times for differents records, when this could be done once per field for ALL records, and store this value in cache for further access. Another potential problem is that we don't take advantage of the ORM's prefetching to fetch all the records that are not in cache at once, instead of doing the same query for every record in the recordset. Yet another problem is the conversion of each cache value to a record format and then combining all of the individual records into a single recordset, which, depending on the size of the recordset, can take an unbelievable amount of CPU time. With this new implementation of `mapped()` we take care of all of these problems: This is done by first delegating `mapped()` from the model to the field, this mapped takes a recordset as input and it will try to batch compute and prefetch as much as possible for the entire recordset, but it will not keep these values for the actual output, it just stores everything in cache and then at the end, retrieves everything from the cache to guarantee the same order. After the mapped, the conversion from cache format to record format is delegated to the new `convert_to_record_multi` which will fetch all the ids and then perform a single browse to encapsulate all of the records into a single recordset with the least amount of overhead possible. Part of Task 2170344 closes odoo/odoo#42611 Signed-off-by: Raphael Collet (rco) <rco@openerp.com> 
Previously, mapped was following a very naive approach, which was simply
calling the field name passed as input for every record in a recordset,
sequentially.

The problem with this approach is that we will potentially recompute the
same fields multiple times for differents records, when this could be
done once per field for ALL records, and store this value in cache for
further access.

Another potential problem is that we don't take advantage of the ORM's
prefetching to fetch all the records that are not in cache at once,
instead of doing the same query for every record in the recordset.

Yet another problem is the conversion of each cache value to a record
format and then combining all of the individual records into a single
recordset, which, depending on the size of the recordset, can take an
unbelievable amount of CPU time.

With this new implementation of `mapped()` we take care of all of these
problems:

This is done by first delegating `mapped()` from the model to the field,
this mapped takes a recordset as input and it will try to batch compute
and prefetch as much as possible for the entire recordset, but it will
not keep these values for the actual output, it just stores everything
in cache and then at the end, retrieves everything from the cache to
guarantee the same order.

After the mapped, the conversion from cache format to record format is
delegated to the new `convert_to_record_multi` which will fetch all the
ids and then perform a single browse to encapsulate all of the records
into a single recordset with the least amount of overhead possible.

Part of Task 2170344

closes odoo/odoo#42611

Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
[IMP] api: move cache_key to the Environment and cache it 2020-01-21T08:28:36+00:00 Adrian Torres adt@odoo.com 2020-01-03T08:12:03+00:00 be01db2f712952a6c1d3688c8587d9624ef5dcc5 Computing the `cache_key` turned out to be a big factor during the lifespan of a `BaseModel.mapped` call and a lot of this time is spent computing the same `cache_key` over and over. These unnecessary computations can be easily reduced to a couple by moving the `cache_key` method on the environment (instead of the field) and by implementing a memo for that method. The rationale is that the `cache_key` of a field does not change for a given environment. The result of this patch is up to 50% faster `Field.__get__` which in turn means a GLOBAL gain in performance, especially for methods / functions that rely heavily on `__get__` such as `BaseModel.mapped`. closes odoo/odoo#42674 Signed-off-by: Adrian Torres (adt) <adt@odoo.com> 
Computing the `cache_key` turned out to be a big factor during the
lifespan of a `BaseModel.mapped` call and a lot of this time is spent
computing the same `cache_key` over and over.

These unnecessary computations can be easily reduced to a couple by
moving the `cache_key` method on the environment (instead of the field)
and by implementing a memo for that method.  The rationale is that the
`cache_key` of a field does not change for a given environment.

The result of this patch is up to 50% faster `Field.__get__` which in
turn means a GLOBAL gain in performance, especially for methods /
functions that rely heavily on `__get__` such as `BaseModel.mapped`.

closes odoo/odoo#42674

Signed-off-by: Adrian Torres (adt) <adt@odoo.com>
[FIX] base: returning a domain from an onchange 2019-11-27T09:59:51+00:00 Xavier Morel xmo@odoo.com 2019-10-28T15:04:18+00:00 17ad6be46aaec5fc4ed03b4f0f0df1a6755ef1b3 Turns out we've got an operator just for the pattern of "match value if there's one, otherwise match everything". Also remove an example of domain in onchange doc. closes odoo/odoo#40938 Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com> 
Turns out we've got an operator just for the pattern of "match value
if there's one, otherwise match everything".

Also remove an example of domain in onchange doc.

closes odoo/odoo#40938

Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
[IMP] base: remove force_company 2019-11-18T12:25:05+00:00 Victor Feyens vfe@odoo.com 2019-09-12T11:43:09+00:00 3455d02189ef8879c8e7f331d159bbd3c1c7b949 From now on, if one wants to force following operations to happen in a given company, use with_company(company) or with_company(cid) to update the environment. 
From now on, if one wants to force following operations to happen in a given company,
use with_company(company) or with_company(cid) to update the environment.