[FIX] *: Reset module states on registry init error

Commit 763d714 introduced cron job locking for databases which had
modules with states set to 'to x', however if an
installation/uninstallation/upgrade fails, the state will stay at 'to
x', and it may stay in that state for an indefinite amount of time,
meaning that cron jobs could stay locked forever.

This commit fixes this in part by adding a cleanup function to loading.py that
will be executed whenever load_modules fails, the function will change
every 'to x' module to their original state, effectively unlocking the
execution of cron jobs.

This however only works to prevent "zombie" transient states for
brand new databases, however for existing databases which already
contain some modules in a zombie state it won't do anything unless
a module is installed/uninstalled/upgraded, which may never happen.

This is where the second part comes in (ir_cron.py), when failing to
execute crons, we check if the failure was due to bad module state
and if an arbitrary amount of time (5 hours as of this commit) has passed
since the last time it was supposed to be executed, if it is the case, it means
that the cron execution failed around 5 * 60 times (1 failure per minute for 5h)
in which case we assume that the crons are stuck because the db
has zombie states and we force a call to reset_module_states.

3 files changed, 27 insertions, 2 deletions

diff --git a/odoo/modules/__init__.py b/odoo/modules/__init__.py
index 530f81ff5e3..b223df99280 100644
--- a/odoo/modules/__init__.py
+++ b/odoo/modules/__init__.py
@@ -7,7 +7,7 @@
 
 from . import db, graph, loading, migration, module, registry
 
-from odoo.modules.loading import load_modules
+from odoo.modules.loading import load_modules, reset_modules_state
 
 from odoo.modules.module import (
     adapt_version,
diff --git a/odoo/modules/loading.py b/odoo/modules/loading.py
index e9744844841..950704e45e9 100644
--- a/odoo/modules/loading.py
+++ b/odoo/modules/loading.py
@@ -431,3 +431,24 @@ def load_modules(db, force_demo=False, status=None, update_module=False):
             _logger.log(25, "All post-tested in %.2fs, %s queries", time.time() - t0, odoo.sql_db.sql_counter - t0_sql)
     finally:
         cr.close()
+
+
+def reset_modules_state(db_name):
+    """
+    Resets modules flagged as "to x" to their original state
+    """
+    # Warning, this function was introduced in response to commit 763d714
+    # which locks cron jobs for dbs which have modules marked as 'to %'.
+    # The goal of this function is to be called ONLY when module
+    # installation/upgrade/uninstallation fails, which is the only known case
+    # for which modules can stay marked as 'to %' for an indefinite amount
+    # of time
+    db = odoo.sql_db.db_connect(db_name)
+    with db.cursor() as cr:
+        cr.execute(
+            "UPDATE ir_module_module SET state='installed' WHERE state IN ('to remove', 'to upgrade')"
+        )
+        cr.execute(
+            "UPDATE ir_module_module SET state='uninstalled' WHERE state='to install'"
+        )
+        _logger.warning("Transient module states were reset")
diff --git a/odoo/modules/registry.py b/odoo/modules/registry.py
index c92da2df13e..50a0be95986 100644
--- a/odoo/modules/registry.py
+++ b/odoo/modules/registry.py
@@ -79,7 +79,11 @@ class Registry(Mapping):
                 try:
                     registry.setup_signaling()
                     # This should be a method on Registry
-                    odoo.modules.load_modules(registry._db, force_demo, status, update_module)
+                    try:
+                        odoo.modules.load_modules(registry._db, force_demo, status, update_module)
+                    except Exception:
+                        odoo.modules.reset_modules_state(db_name)
+                        raise
                 except Exception:
                     _logger.exception('Failed to load registry')
                     del cls.registries[db_name]