summaryrefslogtreecommitdiff
path: root/odoo/api.py
AgeCommit message (Collapse)Author
2020-07-24[IMP] core: smaller and faster field cacheRaphael Collet
Consider a context-dependent field and two possible cache structures for storing the data of that field for N records and K context keys. The common use-case is to assume that N is much larger than K. We look at how many dicts are used in each cache structure, where we label dicts with N entries as "large", and dicts with K entries as "small". Structure | Number of dicts ---------------------------+------------------- cache[field][id][key] | 1 large + N small cache[field][key][id] | 1 small + K large It is known that "small" dicts have a large relative overhead when compared to "large" dicts. Given this fact, the second cache structure will generate less memory overhead. Here are numbers for N=16, showing the size in bytes taken by the cache structure. Larger values of N lead to larger size differences, always in favor of the second structure. Structure | K=1 | K=2 | K=3 | K=4 | K=5 | K=6 -----------------------+------+------+------+------+------+------ cache[field][id][key] | 4488 | 4488 | 4488 | 4488 | 4488 | 6536 cache[field][key][id] | 888 | 1536 | 2184 | 2832 | 3480 | 4256 Moreover, some memory operations are performed on many records and a single context key. Those operations are clearly simpler and faster with the second cache structure.
2020-06-03[FIX] core: prefetching of context-dependent fieldsRaphael Collet
Consider a context-dependent field, and successively access it on a recordset with different contexts. On the first context, the field is correctly computed in batch. After that, the field is always computed one by one. The bug is in the method that determines which records in a given set have no value in cache. On the first context, the cache is empty for the field, so all records are returned. After that, the method considers that all records have a value in cache: they do, but for another context key! Simply using the context key when looking up the cache fixes the issue. closes odoo/odoo#52360 X-original-commit: 35d69589d9b43afe6c6fc9779458323f7180153e Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com> Co-authored-by: Xavier-Do <xdo@odoo.com>
2020-05-19[REF] ir.autovacuum: declarative garbage collector registrationJulien Castiaux
The ir.autovacuum model purpose is to run several garbage collecting operations like removing files from the filestore when no attachment references them anymore. The precedent strategy to register new garbage collection tasks was to override the `power_on` method and to imperatively execute a vacuum cleaning method on a given model. All calls were executed in a single SQL transaction without any error handling, meaning a single fail during any call resulted in a complete failure of the entire vacuum cleaning chain. We introduce a new `@autovacuum` api decorator, its purpose it to register garbage collecting methods that will be safely executed in their own transaction by the vacuum cleaner. In order to ensure this new strategy is used, we deprecate `power_on` extensions. By the way, garbage-collecting methods can be quite heavy and we don't want users to directly call them. We now ensure they are private. closes odoo/odoo#47842 Task: 2154079 Signed-off-by: Raphael Collet (rco) <rco@openerp.com> Co-authored-by: Raphael Collet <rco@odoo.com> Co-authored-by: Olivier Dony <odo@odoo.com>
2020-03-19[FIX] core, base: leftover deprecation warningsXavier Morel
* from collections import <ABC> is deprecated, unclear why the deprecation warning didn't appear before (possibly only appears in 3.7/3.8?) either way `collections.abc` should be 3.3+ so switch everything to it. * add some more ignores on third-party packages deprecation warnings (meh) * while at it, mitigate generation of non-breaking space on some versions of Babel (in the french locale used by our tests anyway) closes odoo/odoo#47581 Related: odoo/enterprise#9214 Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
2020-02-25[FIX] doc: api.model no longer accept traditional style callDenis Ledoux
Since we dropped the old-api style compatibility layer, some years ago. This docstring was added to the 13.0 documentation thanks to odoo/odoo#44838 So it's best not to mention this old-api style in our latest documentation closes odoo/odoo#46223 X-original-commit: 63691c78006fbf7077eb13f3045be21a18bbe753 Signed-off-by: Denis Ledoux (dle) <dle@odoo.com>
2020-02-11[FIX] api, product: search by pricelistNicolas Martinelli
- Activate variants and pricelists - Go to Product > Products Variants - Search for anything on the 'Pricelist' filter A traceback is raised: 'Can only create cache keys from hashable values...'. This comes from the following change: https://github.com/odoo/odoo/blob/4b06fe19fa68255b7982d15e5847da2f6d6209fd/addons/web/static/src/js/views/control_panel/control_panel_model.js#L962 It returns a list instead of a string. Since a list is not hashable, it causes the issue. There are not much solutions since the context key `pricelist` can be a `list` or an `int` (an ID). We force the cache key conversion to a tuple to avoid the `TypeError` and handle the `list` use case (which was broken on top of crashing). opw-2187757 closes odoo/odoo#45064 X-original-commit: ec0a8767143b4c4818418281803c4f52fefa8d2a Signed-off-by: Nicolas Martinelli (nim) <nim@odoo.com>
2020-01-21[IMP] core: improve mapped and filtered performanceAdrian Torres
Previously, mapped was following a very naive approach, which was simply calling the field name passed as input for every record in a recordset, sequentially. The problem with this approach is that we will potentially recompute the same fields multiple times for differents records, when this could be done once per field for ALL records, and store this value in cache for further access. Another potential problem is that we don't take advantage of the ORM's prefetching to fetch all the records that are not in cache at once, instead of doing the same query for every record in the recordset. Yet another problem is the conversion of each cache value to a record format and then combining all of the individual records into a single recordset, which, depending on the size of the recordset, can take an unbelievable amount of CPU time. With this new implementation of `mapped()` we take care of all of these problems: This is done by first delegating `mapped()` from the model to the field, this mapped takes a recordset as input and it will try to batch compute and prefetch as much as possible for the entire recordset, but it will not keep these values for the actual output, it just stores everything in cache and then at the end, retrieves everything from the cache to guarantee the same order. After the mapped, the conversion from cache format to record format is delegated to the new `convert_to_record_multi` which will fetch all the ids and then perform a single browse to encapsulate all of the records into a single recordset with the least amount of overhead possible. Part of Task 2170344 closes odoo/odoo#42611 Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2020-01-21[IMP] api: move cache_key to the Environment and cache itAdrian Torres
Computing the `cache_key` turned out to be a big factor during the lifespan of a `BaseModel.mapped` call and a lot of this time is spent computing the same `cache_key` over and over. These unnecessary computations can be easily reduced to a couple by moving the `cache_key` method on the environment (instead of the field) and by implementing a memo for that method. The rationale is that the `cache_key` of a field does not change for a given environment. The result of this patch is up to 50% faster `Field.__get__` which in turn means a GLOBAL gain in performance, especially for methods / functions that rely heavily on `__get__` such as `BaseModel.mapped`. closes odoo/odoo#42674 Signed-off-by: Adrian Torres (adt) <adt@odoo.com>
2019-11-27[FIX] base: returning a domain from an onchangeXavier Morel
Turns out we've got an operator just for the pattern of "match value if there's one, otherwise match everything". Also remove an example of domain in onchange doc. closes odoo/odoo#40938 Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
2019-11-18[IMP] base: remove force_companyVictor Feyens
From now on, if one wants to force following operations to happen in a given company, use with_company(company) or with_company(cid) to update the environment.
2019-11-04[IMP] base, doc: orm page refactoring.Victor Feyens
closes odoo/odoo#39725 X-original-commit: 7ce7c58592c7c7202a37c73767c477be5a835752 Signed-off-by: Victor Feyens (vfe) <vfe@odoo.com>
2019-10-24[IMP] ORM: do not allow invalid allowed_company_ids context key.Victor Feyens
Since https://github.com/odoo/odoo/commit/a5b6f31cf28e5381e1c85f66730bcdb55998e643, the current companies of the user are saved in the context as "allowed_company_ids" context key. In case of invalid context content, the api was computing the intersection between the context content and the user company(ies) and falling back on user company_id(s) when catching an error. A sanity check was done, but no feedback was given to the user, saying that the context change was falsy. This commits changes this behavior to : * raise an AccessError when trying to access self.env.company(ies) when invalid or unauthorized companies are defined in the context. * take sudo mode into consideration, allowing inter-company impacts, even when current user doesn't have access to a given company, if the code is done in a sudoed environment. Co-Authored-By: Raphael Collet <rco@odoo.com>
2019-10-09[MERGE] forward port branch 13.0 up to f4105eb9c70Christophe Simonis
2019-10-02[DOC] api: warn about onchange pitfallsAdrian Torres
closes odoo/odoo#37836 X-original-commit: a8454381ad2151136ee2a49665f69a12b8dc7daf Signed-off-by: Adrian Torres (adt) <adt@odoo.com>
2019-09-12[REF] fields: setup and use of `depends_context`Raphael Collet
closes odoo/odoo#36795 Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2019-09-11[FIX] fields.py: improve computation time of records having a different ↵Denis Ledoux
value in cache Instead of using a filtered with `cache.get`, use a dedicated method in the Cache class to get the records having a different value in cache then asked. This is mainly to avoid the creation of intermediate `browse` of 1 record, when doing `for rec in self` in `filtered`. Creating browses is costly, and avoiding it leads to performance gains. The dedicated method `get_records_different_from` loops on the record ids, instead of on browse records
2019-09-11[IMP] api: lazy_property for user, company, companiesDenis Ledoux
These variable are not supposed to change within a same environment. The lazy property will compute these variables only once, then store the result, while the property were computing these variables each time they were called. e.g. for a 1000 iteration loop, with `env.company`, the company was computed 1000 times. With a lazy property, the company will be computed one time only.
2019-09-09[FIX] models: use transitive triggers instead of recursionRaphael Collet
On average, this reduces the total time spent in method `modified` by half (times measured on invoice creation and post).
2019-09-17[FIX] *: remove old deprecated modules/functionsJulien Castiaux
PEP-594 is deprecating a bunch of modules. As part of the cleanup, we are also dealing with long deprecated modules, functions and aliases. * `assert_` -> `assertTrue` * `assertEquals` -> `assertEqual` * `assertNotEquals` -> `assertNotEqual` * `assertAlmostEquals` -> `assertAlmostEqual` * `assertRaisesRegexp` -> `assertRaisesRegex` * `assertRegexpMatches` -> `assertRegex` * `base64.encodestring` -> `base64.encodebytes` * `base64.decodestring` -> `base64.decodebytes` * `inspect.getargspec` -> `inspect.signature` * `inspect.formatargspec` -> `inspect.signature` * `logging.warn` -> `logging.warning` closes odoo/odoo#36863 Task: 2003936 Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2019-08-26[FIX] base: multi-company codeFabien Pinckaers
2019-08-23[FIX] api: share `protected` among environmentsRaphael Collet
If a field is protected against recomputation, it must be protected in *all* environments.
2019-08-20[IMP] models: ORM speedupRaphael Collet
This branch is the combination of several optimizations in the ORM: * store field values once in the cache: the cache reflects more faithfully the database, only fields that explicitly depend on the context have an extra indirection in the cache; * delay recomputations by default: use method `recompute` to explicitly flush out pending recomputations; * delay updates in method `write`: updates are stored in a data structure that can be flushed efficiently to the database with method `flush` (which also flush out recomputations); * make method `modified` take advantage of inverse fields to inverse dependencies; * filter records by evaluating a domain on records in Python; * a computed field with `readonly=False` behaves like a normal field with an onchange method; * computed fields are computed in superuser mode by default. Work done by Toufik Ben Jaa, Raphael Collet, Denis Ledoux and Fabien Pinckaers. closes odoo/odoo#35659 Signed-off-by: Denis Ledoux <beledouxdenis@users.noreply.github.com>
2019-08-09[IMP] *: display some onchange warning with notiffja-odoo
* = base, web, sale, stock, sale_stock, stock_account Warnings for the onchange now takes a type to know if it needs to be displayed with a notification. Else it is displayed with a dialog notification (like before). Related to https://github.com/odoo/odoo/pull/32132 and https://github.com/odoo/odoo/pull/35209 Part of https://github.com/odoo/odoo/pull/35342 task-2047628 closes odoo/odoo#35342 Signed-off-by: Quentin Smetz (qsm) <qsm@odoo.com>
2019-07-17[REM] core: remove @api.multiAdrian Torres
kw_multi is the default API, so the decorator is no longer necessary and hasn't been for a long time
2019-07-17[REM] *: calls to @api.multiAdrian Torres
Multi is the default api for methods, it is not necessary to explicitly decorate methods with it, adds clutter and most people use it because they see that the rest of the code uses it. Done with `find . -type f -name '*.py' | xargs sed -i '/@api.multi/d'`
2019-07-09[REM] api: remove references to env.dirtyAdrian Torres
closes odoo/odoo#34556 Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2019-07-08[IMP] api: remove deprecated decoratorsRaphael Collet
2019-07-05[REM] api: remove helpers one and aggregateAdrian Torres
`api.one` has been deprecated since v9 because it often makes the code less clear and behaves in ways developers and readers may not expect since functions decorated with it usually expect a `list` of record(s) instead of the recordset, whereas most modern Odoo code expects `self` to be a recordset. The function `aggregate` was solely being used by `api.one` thus it has also been removed since there doesn't seem to be any other use for it thus far. closes odoo/odoo#34555 Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2019-07-05[REM] *: remove calls to `api.one` and adapt codeAdrian Torres
Adapt all code that was using `api.one` to recordset-style method and remove any and all calls to `api.one` in preparation for its removal.
2019-07-04[REF] *: `record.env.user._is_XXX()` -> `record.env.is_XXX()`Raphael Collet
Superuser mode implies `record.env.is_XXX()`.
2019-07-04[ADD] api: flag `su` on environmentsRaphael Collet
The flag defines a "superuser mode" on environments, which allows to bypass access rights without changing the current user id.
2019-05-29[IMP] api.py: Rename company_id/company_ids into company/companiesYannick Tivisse
The goal is to be coherent with the user property. Actually, company_id and company_ids on the environment are no fields. Calling env.company_id returns a browse record, not an id.
2019-05-23[REF] api: remove deprecated env.in_onchangeRaphael Collet
2019-05-13[IMP] base: Contextualize the multi companyYannick Tivisse
Purpose ======= Allow the user to select the allowed companies for which he wants to see records on top of selecting his current company. It is confusing for users to see the records from the company he is connected to and the records of the children companies. Instead of using the hierarchy of companies to access records across companies, the user can now select (from his set of allowed companies) the companies for which he wants to access records. /!\ This means that the user will interact with records from company A when in company B. Example: a SO has been created and confirmed in A. When in B, I create the invoice from it. Specifications ============== 1/ Deprecate the parent/children hierarchy on the res.company model. The fields are kept on the res.company model to ensure the retro-compatibility, but won't be used accross the standard code anymore. The only functional usage for this mechanism was to allow to see records from several companies by creating a virtual parent company, which will be possible with the new mechanism. 2/ By default, a user will only see the records of the company he is connected to (or records without a company). (It is still editable by the user if needed). For that, put this information in the user context, to allow having different configurations on different browser tabs. Instead of having domains like ['|', ('company_id', '=', False), ('company_id', 'child_of', user.company_id.id)] you'll have something like ['|', ('company_id', '=', False), ('company_id', 'in', company_ids)] Note that the 'company_ids' is a value that is passed in the evaluation context on the record rule, as we already have user, or time. company_ids is a list of the ids of all the enabled companies in the user's context. 3/ Out of the generic improvements brought by this task, this will illustrate issues that could exist since several versions. For example, it should not be possible to create a scrap order for the company A with a package of the company B, or it should not be possible to create an invoice on the company A with payment terms from the company B. Before the version 12.0, it was easy to encounter this kind of issues as the admin was the SUPERUSER_ID. A positive side effect of the fact that the SUPERUSER_ID has become an inactive user was to make it more difficult to introduce mismatch on the records, but haven't solved the issue, as it was still possible to do it with parent companies configuration. Some of these issues have been fixed in this commit, but all the business flows should be re-tested to check if an ir.rule should be introduced (eg: a multi company rule for stock.quand.package), if the company of a record is correctly transfered to another record created from the first record (eg: From a SO, create an invoice and a payment, the company of the sales order should be transfered on the invoice and the payment, even if the company of the sales order is A and I'm logged into the company B with the company A enabled. 4/ Currently, if I click on a button on a notification email (example 'View Task'), I face a traceback if I'm not logged into the company of the record. Now, if you click on a button and if you have access to the record, the correct company will be automatically set. 5/ If I display a kanban view with several records from several companies (and an image), all the images should be displayed. 6/ Currently if you copy paste an url, this will crash if you're not in the correct company. This won't be fixed because it's quite impossible to do it in a clean way. This task brings a workaround. Copy/Paste -> Traceback -> Log into the correct company, re-copy/paste -> Ok. 7/ 2 property methods have been added on the environment to retrieve the company on which the user is logged in and the companies the user enabled, on a specific tab. That way, when creating a record, instead of doing default=lambda self: self.env.user.company_id do default=lambda self: self.env.company_id On the other hand, to retrieve the enabled companies, do companies = self.env.company_ids 8/ Modify the Company Switcher widget to allow to log into another company WITHOUT writing on the res.users (and thus bringing cache invalidation issues and so on). Also allow to enable several companies and see records from several companies, and independantly of the other browser's tabs. 9/ When focusing on a tab, save the current company configuration on the local storage. That way, when doing 'CTRL+T' or a middle click, the context is propagated to the new tab. 10/ Improve the error message in case of multi company access errors. Now, when the user is in debug mode, display the related names of the records and the name of the user who brings the issue. 11/ Remove the context erasing when writing on a res.users This is probably coming from the migration to new API of the base module. The context was not propagated at this moment, which was a common mistake at that time. When migrating the module, probably by using the 'black box' method, as the context was not propagated, it was erased on the new version. This is now an issue because the context (i.e. the enabled companies) was erased when writing on a res.users, leading to tracebacks. See: https://github.com/odoo/odoo/commit/7eab8e26d3d46c53f4be924d6a34e80a66e74960#diff-4c2e738ee8f64f11806c889ea097b5e7R624 12/ Fix the crash manager on redirect warnings. The issue is the following - Create an invoice on a company without a configured CoA. - Set a partner - On the onchange_partner_id, a redirect warning is raised to propose you to configure a CoA - Click on 'Go to the configuration panel' - A generic warning says something like 'Do you want to discard your changes?' - Click on yes, the page refreshes, but not on the redirect action. Now, set correctly the action on the hash, and reload instead. The breadcrumb is lost for example, but you reach the correct action at least. 13/ Introduce a res.group to enable/disable the multi company per tab feature. 14/ To help the users to know which tab is in which company, add the possibility to have a favicon per company. When creating a company, the classical 'O' icon is colored by default in a random color. 15/ Remove the company switcher on the frontend. This was mainly there to allow a user to swicth to the company linked to the website. This behavior is now transparent to the user. If the website A is activated, then the company set on the context is the company of the website. 16/ Deprecated the _company_default_get method on the res.company model. Remove the method _get_company on the res.users model. 17/ Add 'allowed_company_ids' and 'current_company_id' on the pyeval context. You can now use those variables on domains in the views to access directly to the activated company.ies on the current tab. TaskID: 1960971 closes odoo/odoo#32341 Signed-off-by: Yannick Tivisse (yti) <yti@odoo.com>
2019-05-06[IMP] models: new implementation of prefetchingRaphael Collet
2019-03-22[IMP] api: RPC calls must always pass `context` as a keyword argumentRaphael Collet
2019-02-21[MERGE] forward port branch 12.0 up to 84143a34b31Christophe Simonis
2019-02-20[MERGE] forward port branch saas-11.3 up to 2c495502aa3Christophe Simonis
2019-02-20[MERGE] forward port branch 11.0 up to de8cefcef4dChristophe Simonis
2019-01-29[MERGE] forward port branch 11.0 up to 36d96e01503Christophe Simonis
2019-01-03[IMP] api: improve storing performance of the api cacheDenis Ledoux
This revision moves the `cache_key` to the first level dict of the cache, instead of the last one. Doing so, we reduce the number of times the reference to the cache key is stored in the dict. For instance, for 100.000 records, 20 fields and 2 env (e.g. with and without sudo) formerly, there were 100.000 * 20 * 2 occurences of cache key references now, there is only 2 references. Storing references to an object consumes memory. Therefore, by reducing the number of object references in the cache, we reduce the memory consumed by the cache. Also, we reduce the time to access a value in the cache as the cache size is smaller. The time and memory consumption are therefore improved, while keeping the advantages of revision d7190a3fd043639ec21e91ddb30481b2989842c5 which was about sharing the cache of fields which do not depends on the context, but only on the cursor and user id. This revision relies on the fact there are less different references to the cache key then references to fields/records. Indeed, this is more likely to have 100.000 different records stored in the cache rather than 100.000 different environments. Here is the Python proof of concept that was used to make the conclusion that setting the cache_key in the first level dict of the cache is more efficient. ```Python import os import psutil import time from collections import defaultdict cr = object() uid = 1 fields = [object() for i in range(20)] number_items = 500000 p = psutil.Process(os.getpid()) m = p.memory_info().rss s = time.time() cache_key = (cr, uid) cache = defaultdict(lambda: defaultdict(dict)) for field in fields: for i in range(number_items): cache[field][i][cache_key] = 5.0 # cache[cache_key][field][i] = 5.0 print('Memory: %s' % (p.memory_info().rss - m,)) print('Time: %s' % (time.time() - s,)) ``` - Using `cache[field][i][cache_key]`: - Time: 3.17s - Memory: 3138MB - Using `cache[cache_key][field][i]`: - Time: 1.43s - Memory: 756MB Even worse, when the cache key tuple is instantiated inside the loop, for the former cache structure (e.g. `cache[field][i][(cr, uid)]`), the time goes from 3.17s to 25.63s and the memory from 3138MB to 3773MB Here is the same proof of concept, but using the Odoo API and Cache: ```Python import os import psutil import time from odoo.api import Cache model = env['res.users'] records = [model.new() for i in range(100000)] p = psutil.Process(os.getpid()) m = p.memory_info().rss s = time.time() cache = Cache() char_fields = [field for field in model._fields.values() if field.type == 'char'] for field in char_fields: for record in records: cache.set(record, field, 'test') print('Memory: %s' % (p.memory_info().rss - m,)) print('Time: %s' % (time.time() - s,)) ``` - Before (`cache[field][record_id][cache_key]` and cache_key tuple instantiated in the loop): - Time: 4.12s - Memory: 810MB - After (`cache[cache_key][field][record_id]` and cache_key tuple stored in the env and re-used): - Time: 1.63s - Memory: 125MB This can be played in an Odoo shell, for instance by storing it in `/tmp/test.py`, and then piping it to the Odoo shell: `cat /tmp/test.py | ./odoo-bin shell -d 12.0` closes odoo/odoo#29676
2019-01-03[IMP] api: gain some memory by not creating the cache_key tuple multiple timesDenis Ledoux
2019-02-11[IMP] read, cache: faster read by updating the cache by fieldsDenis Ledoux
instead of updating the cache by records. We use `cr.fetchall()` and `zip` to get the values by field, which is more convenient to be stored in cache. Result of `cr.fetchall()`: ``` [ (3, 'Marc Demo', 'demo@example.com', '032'), (1, 'Mitchell Admin', 'admin@example.com', '001'), ] ``` After being grouped by field: ``` [ [3, 1], ['Marc Demo', 'Mitchell Admin], ['demo@example.com', 'admin@example.com'], ['032', '001'], ] ``` That way we can update the cache of a field for all records at once, by calling the builtin `update` method of `dict` with the `zip` of record ids and corresponding values. This significantly speeds up the storage of field values in the cache. closes odoo/odoo#30817
2019-02-11[IMP] api: avoid `record.id` indirectionsDenis Ledoux
The cache is quite time critical, as it is accessed millions times when reading multiple fields on hundred of thousands of records. Avoiding indirections and `if` statement when possible speeds up the cache access time.
2018-11-28[MERGE] forward port branch 11.0 up to bb6f6c57f97Christophe Simonis
2018-11-29[MERGE] forward port branch saas-11.3 up to c0eef427111Christophe Simonis
2019-01-25[IMP] api: improve storing performance of the api cacheDenis Ledoux
This revision moves the `cache_key` to the first level dict of the cache, instead of the last one. Doing so, we reduce the number of times the reference to the cache key is stored in the dict. For instance, for 100.000 records, 20 fields and 2 env (e.g. with and without sudo) formerly, there were 100.000 * 20 * 2 occurences of cache key references now, there is only 2 references. Storing references to an object consumes memory. Therefore, by reducing the number of object references in the cache, we reduce the memory consumed by the cache. Also, we reduce the time to access a value in the cache as the cache size is smaller. The time and memory consumption are therefore improved, while keeping the advantages of revision d7190a3fd043639ec21e91ddb30481b2989842c5 which was about sharing the cache of fields which do not depends on the context, but only on the cursor and user id. This revision relies on the fact there are less different references to the cache key then references to fields/records. Indeed, this is more likely to have 100.000 different records stored in the cache rather than 100.000 different environments. Here is the Python proof of concept that was used to make the conclusion that setting the cache_key in the first level dict of the cache is more efficient. ```Python import os import psutil import time from collections import defaultdict cr = object() uid = 1 fields = [object() for i in range(20)] number_items = 500000 p = psutil.Process(os.getpid()) m = p.memory_info().rss s = time.time() cache_key = (cr, uid) cache = defaultdict(lambda: defaultdict(dict)) for field in fields: for i in range(number_items): cache[field][i][cache_key] = 5.0 # cache[cache_key][field][i] = 5.0 print('Memory: %s' % (p.memory_info().rss - m,)) print('Time: %s' % (time.time() - s,)) ``` - Using `cache[field][i][cache_key]`: - Time: 3.17s - Memory: 3138MB - Using `cache[cache_key][field][i]`: - Time: 1.43s - Memory: 756MB Even worse, when the cache key tuple is instantiated inside the loop, for the former cache structure (e.g. `cache[field][i][(cr, uid)]`), the time goes from 3.17s to 25.63s and the memory from 3138MB to 3773MB Here is the same proof of concept, but using the Odoo API and Cache: ```Python import os import psutil import time from odoo.api import Cache model = env['res.users'] records = [model.new() for i in range(100000)] p = psutil.Process(os.getpid()) m = p.memory_info().rss s = time.time() cache = Cache() char_fields = [field for field in model._fields.values() if field.type == 'char'] for field in char_fields: for record in records: cache.set(record, field, 'test') print('Memory: %s' % (p.memory_info().rss - m,)) print('Time: %s' % (time.time() - s,)) ``` - Before (`cache[field][record_id][cache_key]` and cache_key tuple instantiated in the loop): - Time: 4.12s - Memory: 810MB - After (`cache[cache_key][field][record_id]` and cache_key tuple stored in the env and re-used): - Time: 1.63s - Memory: 125MB This can be played in an Odoo shell, for instance by storing it in `/tmp/test.py`, and then piping it to the Odoo shell: `cat /tmp/test.py | ./odoo-bin shell -d 12.0` closes odoo/odoo#29676 closes odoo/odoo#30554
2019-01-25[IMP] api: gain some memory by not creating the cache_key tuple multiple timesDenis Ledoux
2018-11-23[FIX] fields: combine special values during onchangeRaphael Collet
Consider a many2one field `foo_id` on model `bar`, with an inverse one2many field `bar_ids` on model `foo`. During an onchange, the statement bar.foo_id = foo puts a special value in cache to add `bar` to the value of `foo.bar_ids` without explicitly reading `foo.bar_ids`. Executing the above statement a second time, the cache of `foo.bar_ids` is no longer empty. This causes the actual value of `foo.bar_ids` to be read and updated. The issue is that this can be slow for large values of `foo.bar_ids`. Avoid reading the value of the one2many field by handling the case where the cache contains the special value: simply update the special value to take into account the second assignment. closes odoo/odoo#28982
2018-09-11[IMP] api, models: _write performances on importXavier Morel
Followup from the previous commit: before this, _write takes 32% of total runtime, of which 13.7% is ultimately assignable to add_todo. Turns out for the test case we mostly keep adding records to recorsets where they're already present, so the 13.7% of runtime in add_todo are mostly spent creating orderedsets (11.17) then converting those back into recordsets (1.75%) with some time spent creating lists and appending records (already present) in them. Checking if the records are already present before merging them in decreases add_todo's runtime cost to 0.5%, and ultimately _write to 20%. As many of the ignorable calls were merging a singleton or an empty recorset into the parent recordset, optimising for these cases was added to BaseModel's <= and >=.