<feed xmlns='http://www.w3.org/2005/Atom'>
<title>Odoo/odoo/addons/base/tests/test_cache.py, branch 14.0</title>
<subtitle>The current checkout of upstream Odoo</subtitle>
<link rel='alternate' type='text/html' href='http://erp.ceuconsult.com:9000/Odoo/'/>
<entry>
<title>[IMP] base: Adapt tests to work with/without demo data</title>
<updated>2019-11-05T12:08:02+00:00</updated>
<author>
<name>Yannick Tivisse</name>
<email>yti@odoo.com</email>
</author>
<published>2019-11-04T15:20:09+00:00</published>
<link rel='alternate' type='text/html' href='http://erp.ceuconsult.com:9000/Odoo/commit/?id=04dc074804f78f764ed35e01d30cbfd9aca4564c'/>
<id>04dc074804f78f764ed35e01d30cbfd9aca4564c</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>[IMP] models: ORM speedup</title>
<updated>2019-08-20T12:43:59+00:00</updated>
<author>
<name>Raphael Collet</name>
<email>rco@odoo.com</email>
</author>
<published>2019-08-14T07:35:49+00:00</published>
<link rel='alternate' type='text/html' href='http://erp.ceuconsult.com:9000/Odoo/commit/?id=9920f20e4c7753bc17bea71dea3a90f7de687196'/>
<id>9920f20e4c7753bc17bea71dea3a90f7de687196</id>
<content type='text'>
This branch is the combination of several optimizations in the ORM:

* store field values once in the cache: the cache reflects more
faithfully the database, only fields that explicitly depend on the
context have an extra indirection in the cache;

* delay recomputations by default: use method `recompute` to explicitly
flush out pending recomputations;

* delay updates in method `write`: updates are stored in a data
structure that can be flushed efficiently to the database with method
`flush` (which also flush out recomputations);

* make method `modified` take advantage of inverse fields to inverse
dependencies;

* filter records by evaluating a domain on records in Python;

* a computed field with `readonly=False` behaves like a normal field
with an onchange method;

* computed fields are computed in superuser mode by default.

Work done by Toufik Ben Jaa, Raphael Collet, Denis Ledoux and Fabien
Pinckaers.

closes odoo/odoo#35659

Signed-off-by: Denis Ledoux &lt;beledouxdenis@users.noreply.github.com&gt;</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This branch is the combination of several optimizations in the ORM:

* store field values once in the cache: the cache reflects more
faithfully the database, only fields that explicitly depend on the
context have an extra indirection in the cache;

* delay recomputations by default: use method `recompute` to explicitly
flush out pending recomputations;

* delay updates in method `write`: updates are stored in a data
structure that can be flushed efficiently to the database with method
`flush` (which also flush out recomputations);

* make method `modified` take advantage of inverse fields to inverse
dependencies;

* filter records by evaluating a domain on records in Python;

* a computed field with `readonly=False` behaves like a normal field
with an onchange method;

* computed fields are computed in superuser mode by default.

Work done by Toufik Ben Jaa, Raphael Collet, Denis Ledoux and Fabien
Pinckaers.

closes odoo/odoo#35659

Signed-off-by: Denis Ledoux &lt;beledouxdenis@users.noreply.github.com&gt;</pre>
</div>
</content>
</entry>
<entry>
<title>[FIX] *: sudo() -&gt; with_user()</title>
<updated>2019-07-04T11:32:22+00:00</updated>
<author>
<name>Raphael Collet</name>
<email>rco@odoo.com</email>
</author>
<published>2019-06-17T09:45:09+00:00</published>
<link rel='alternate' type='text/html' href='http://erp.ceuconsult.com:9000/Odoo/commit/?id=b7fd679a6c41dbb694c6fb2ce4911798de0424b1'/>
<id>b7fd679a6c41dbb694c6fb2ce4911798de0424b1</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>[FIX] base: make cache tests more specific</title>
<updated>2019-01-08T15:47:55+00:00</updated>
<author>
<name>Raphael Collet</name>
<email>rco@odoo.com</email>
</author>
<published>2019-01-07T13:09:29+00:00</published>
<link rel='alternate' type='text/html' href='http://erp.ceuconsult.com:9000/Odoo/commit/?id=f1df5f352d4e39c0d1fc35930e75079236e36bd5'/>
<id>f1df5f352d4e39c0d1fc35930e75079236e36bd5</id>
<content type='text'>
closes odoo/odoo#29998</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
closes odoo/odoo#29998</pre>
</div>
</content>
</entry>
<entry>
<title>[IMP] api: improve storing performance of the api cache</title>
<updated>2019-01-03T12:39:04+00:00</updated>
<author>
<name>Denis Ledoux</name>
<email>dle@odoo.com</email>
</author>
<published>2018-12-20T12:27:15+00:00</published>
<link rel='alternate' type='text/html' href='http://erp.ceuconsult.com:9000/Odoo/commit/?id=189cc4405cbf4a745989dec6c0b2c41d07e42789'/>
<id>189cc4405cbf4a745989dec6c0b2c41d07e42789</id>
<content type='text'>
This revision moves the `cache_key` to the first level
dict of the cache, instead of the last one.

Doing so, we reduce the number of times the reference
to the cache key is stored in the dict.

For instance,
for 100.000 records, 20 fields and 2 env (e.g. with and without sudo)
formerly, there were 100.000 * 20 * 2 occurences of cache key references
now, there is only 2 references.

Storing references to an object consumes memory.
Therefore, by reducing the number of object references
in the cache, we reduce the memory consumed by the cache.
Also, we reduce the time to access a value in the cache
as the cache size is smaller.

The time and memory consumption are therefore improved,
while keeping the advantages of revision
d7190a3fd043639ec21e91ddb30481b2989842c5
which was about sharing the cache of fields
which do not depends on the context, but
only on the cursor and user id.

This revision relies on the fact there are less different references
to the cache key then references to fields/records.
Indeed, this is more likely to have 100.000 different records stored
in the cache rather than 100.000 different environments.

Here is the Python proof of concept that was used
to make the conclusion that setting the cache_key
in the first level dict of the cache is more efficient.
```Python
import os
import psutil
import time

from collections import defaultdict

cr = object()
uid = 1
fields = [object() for i in range(20)]
number_items = 500000

p = psutil.Process(os.getpid())
m = p.memory_info().rss
s = time.time()

cache_key = (cr, uid)
cache = defaultdict(lambda: defaultdict(dict))
for field in fields:
    for i in range(number_items):
        cache[field][i][cache_key] = 5.0
        # cache[cache_key][field][i] = 5.0

print('Memory: %s' % (p.memory_info().rss - m,))
print('Time: %s' % (time.time() - s,))

```
- Using `cache[field][i][cache_key]`:
   - Time: 3.17s
   - Memory: 3138MB
- Using `cache[cache_key][field][i]`:
   - Time: 1.43s
   - Memory: 756MB

Even worse, when the cache key tuple is instantiated inside the loop,
for the former cache structure (e.g. `cache[field][i][(cr, uid)]`),
the time goes from 3.17s to 25.63s and the memory from 3138MB to 3773MB

Here is the same proof of concept, but using the Odoo API and Cache:
```Python
import os
import psutil
import time

from odoo.api import Cache

model = env['res.users']
records = [model.new() for i in range(100000)]

p = psutil.Process(os.getpid())
m = p.memory_info().rss
s = time.time()

cache = Cache()
char_fields = [field for field in model._fields.values() if field.type == 'char']
for field in char_fields:
    for record in records:
        cache.set(record, field, 'test')

print('Memory: %s' % (p.memory_info().rss - m,))
print('Time: %s' % (time.time() - s,))
```
- Before (`cache[field][record_id][cache_key]` and cache_key tuple instantiated in the loop):
   - Time: 4.12s
   - Memory: 810MB
- After (`cache[cache_key][field][record_id]` and cache_key tuple stored in the env and re-used):
   - Time: 1.63s
   - Memory: 125MB

This can be played in an Odoo shell, for instance
by storing it in `/tmp/test.py`, and then
piping it to the Odoo shell:
`cat /tmp/test.py | ./odoo-bin shell -d 12.0`

closes odoo/odoo#29676</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This revision moves the `cache_key` to the first level
dict of the cache, instead of the last one.

Doing so, we reduce the number of times the reference
to the cache key is stored in the dict.

For instance,
for 100.000 records, 20 fields and 2 env (e.g. with and without sudo)
formerly, there were 100.000 * 20 * 2 occurences of cache key references
now, there is only 2 references.

Storing references to an object consumes memory.
Therefore, by reducing the number of object references
in the cache, we reduce the memory consumed by the cache.
Also, we reduce the time to access a value in the cache
as the cache size is smaller.

The time and memory consumption are therefore improved,
while keeping the advantages of revision
d7190a3fd043639ec21e91ddb30481b2989842c5
which was about sharing the cache of fields
which do not depends on the context, but
only on the cursor and user id.

This revision relies on the fact there are less different references
to the cache key then references to fields/records.
Indeed, this is more likely to have 100.000 different records stored
in the cache rather than 100.000 different environments.

Here is the Python proof of concept that was used
to make the conclusion that setting the cache_key
in the first level dict of the cache is more efficient.
```Python
import os
import psutil
import time

from collections import defaultdict

cr = object()
uid = 1
fields = [object() for i in range(20)]
number_items = 500000

p = psutil.Process(os.getpid())
m = p.memory_info().rss
s = time.time()

cache_key = (cr, uid)
cache = defaultdict(lambda: defaultdict(dict))
for field in fields:
    for i in range(number_items):
        cache[field][i][cache_key] = 5.0
        # cache[cache_key][field][i] = 5.0

print('Memory: %s' % (p.memory_info().rss - m,))
print('Time: %s' % (time.time() - s,))

```
- Using `cache[field][i][cache_key]`:
   - Time: 3.17s
   - Memory: 3138MB
- Using `cache[cache_key][field][i]`:
   - Time: 1.43s
   - Memory: 756MB

Even worse, when the cache key tuple is instantiated inside the loop,
for the former cache structure (e.g. `cache[field][i][(cr, uid)]`),
the time goes from 3.17s to 25.63s and the memory from 3138MB to 3773MB

Here is the same proof of concept, but using the Odoo API and Cache:
```Python
import os
import psutil
import time

from odoo.api import Cache

model = env['res.users']
records = [model.new() for i in range(100000)]

p = psutil.Process(os.getpid())
m = p.memory_info().rss
s = time.time()

cache = Cache()
char_fields = [field for field in model._fields.values() if field.type == 'char']
for field in char_fields:
    for record in records:
        cache.set(record, field, 'test')

print('Memory: %s' % (p.memory_info().rss - m,))
print('Time: %s' % (time.time() - s,))
```
- Before (`cache[field][record_id][cache_key]` and cache_key tuple instantiated in the loop):
   - Time: 4.12s
   - Memory: 810MB
- After (`cache[cache_key][field][record_id]` and cache_key tuple stored in the env and re-used):
   - Time: 1.63s
   - Memory: 125MB

This can be played in an Odoo shell, for instance
by storing it in `/tmp/test.py`, and then
piping it to the Odoo shell:
`cat /tmp/test.py | ./odoo-bin shell -d 12.0`

closes odoo/odoo#29676</pre>
</div>
</content>
</entry>
<entry>
<title>[FIX] base: missing tests on record cache</title>
<updated>2019-01-03T12:39:04+00:00</updated>
<author>
<name>Raphael Collet</name>
<email>rco@odoo.com</email>
</author>
<published>2019-01-03T09:02:40+00:00</published>
<link rel='alternate' type='text/html' href='http://erp.ceuconsult.com:9000/Odoo/commit/?id=cdc01c50b3691ac9e5d550afeb9085ff7476daed'/>
<id>cdc01c50b3691ac9e5d550afeb9085ff7476daed</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
</feed>
