[IMP] tours: make chrome request a websocket port from the OS

This should be a smarter and properly reliable version of #42071: in
that, the runner requests a port, closes it, and gives the port to
Chrome. However this apparently turns out to be less reliable than
hoped for and the port we just released can immediately be picked up
by somebody else (the original PR assumed the allocation of ephemeral
ports would be random or FIFO but that may not be the case, especially
inside containers).

This uses the same technique of requesting port 0 so the OS allocates
one, but it's Chrome requesting & immediately connecting so there
should be no race condition possible, and we keep the property that as
long as ephemeral ports are available Chrome will be able to open one
without conflicts or overlaps.

This leaves the issue of *retrieving* the port chrome got. Thankfully
it turns out we use a custom user-data-dir in which case Chrome writes
the port it got to `$DATA_DIR/DevToolsActivePort`[0]. Despite the
file's name it *also* contains the path for the devtools endpoint so
we need to only read the first line (rather than be able to read and
intify the entire thing).

Wait up to 10s before giving up entirely, and wait 100ms between each
check for the file's existence: on my machine without significant load
the file appears after 80 to 150ms, waiting up to 90ms seems ok (it's
not like we're in a super hurry as tours tend to be pretty long).

Other paths explored before moc used his eyes and brain and found out
about DevToolsActivePort:

* Chrome prints ws URL on the stderr, however because we don't know
  how much garbage Chrome might send there we need to send it to a
  continuous sink otherwise Chrome *might* end up blocking on its
  stderr because we're not reading from it. This turns out to be a bit
  of a mess of processes or additional threads.
* xdo suggested we check what ports Chrome listens on using something
  like netstat/ss (turns out `psutil` has support for that OOTB),
  which worked great except on WSL (where it didn't work at all), and
  the future-proofness was a bit questionable as Chrome might add
  other servers in the future.
* fme suggested using socket activation support[1] and passing in the
  port we'd opened without closing it, which would really have been
  ideal, however it turns out it was removed a few months later when
  chrome added pipes support[2], which was a pain to realize as chrome
  doesn't exactly do any useful error reporting (so unknown options
  just disappear into a void to be never seen or heard of ever).
* And while the pipes system[3] has *serious* positive attributes
  (even lower initialization overhead, we could remove the websocket
  dependency, also avoids wasting sockets though that's not too much
  of an issue here) it would require rewriting a lot more than just
  the initialization as it uses its own logical protocol
  (NUL-terminated JSON). TBF most of the messaging stuff is properly
  contained into just a few `_websocket` methods but still...

[0] https://bugs.chromium.org/p/chromium/issues/detail?id=624837#c4
[1] https://bugs.chromium.org/p/chromium/issues/detail?id=624837
[2] https://chromium-review.googlesource.com/c/chromium/src/+/954405/3#message-ab7415a7db7b94787300d987216e9ce60db47bc2
[3] https://chromium-review.googlesource.com/c/chromium/src/+/954405/3

opw-2378464

Replaces #64758

closes odoo/odoo#64844

Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Co-authored-by: Xavier Dollé <xdo@odoo.com>
Co-authored-by: Christophe Monniez <moc@odoo.com>

1 files changed, 22 insertions, 21 deletions

diff --git a/odoo/tests/common.py b/odoo/tests/common.py
index af270c5b7f8..2abeb7bd3a8 100644
--- a/odoo/tests/common.py
+++ b/odoo/tests/common.py
@@ -6,6 +6,7 @@ helpers and classes to write tests.
 """
 import base64
 import collections
+import difflib
 import functools
 import importlib
 import inspect
@@ -14,32 +15,35 @@ import json
 import logging
 import operator
 import os
+import pathlib
 import platform
+import pprint
 import re
-import requests
 import shutil
 import signal
-import socket
 import subprocess
 import sys
 import tempfile
 import threading
 import time
 import unittest
-import difflib
-import werkzeug.urls
+from collections import defaultdict
 from contextlib import contextmanager
 from datetime import datetime, date
+from itertools import zip_longest as izip_longest
 from unittest.mock import patch
+from xmlrpc import client as xmlrpclib
 
-from collections import defaultdict
+import requests
+import werkzeug.urls
 from decorator import decorator
-from itertools import zip_longest as izip_longest
 from lxml import etree, html
-from xmlrpc import client as xmlrpclib
 
+import odoo
+from odoo import api
 from odoo.models import BaseModel
 from odoo.osv.expression import normalize_domain, TRUE_LEAF, FALSE_LEAF
+from odoo.service import security
 from odoo.sql_db import Cursor
 from odoo.tools import float_compare, single_email_re
 from odoo.tools.misc import find_in_path
@@ -52,13 +56,6 @@ except ImportError:
     websocket = None
 
 
-import odoo
-import pprint
-from odoo import api
-from odoo.service import security
-
-
-
 _logger = logging.getLogger(__name__)
 
 # The odoo library is supposed already configured.
@@ -756,8 +753,18 @@ class ChromeBrowser():
     def _spawn_chrome(self, cmd):
         if os.name != 'posix':
             return
+
         pid = os.fork()
         if pid != 0:
+            port_file = pathlib.Path(self.user_data_dir, 'DevToolsActivePort')
+            for _ in range(100):
+                time.sleep(0.1)
+                if port_file.is_file():
+                    with port_file.open('r', encoding='utf-8') as f:
+                        self.devtools_port = int(f.readline())
+                    break
+            else:
+                raise unittest.SkipTest('Failed to detect chrome devtools port after 2.5s.')
             return pid
         else:
             if platform.system() != 'Darwin':
@@ -775,11 +782,6 @@ class ChromeBrowser():
     def _chrome_start(self):
         if self.chrome_pid is not None:
             return
-        with socket.socket() as s:
-            s.bind(('localhost', 0))
-            if hasattr(socket, 'SO_REUSEADDR'):
-                s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
-            _, self.devtools_port = s.getsockname()
 
         switches = {
             '--headless': '',
@@ -803,9 +805,8 @@ class ChromeBrowser():
             '--autoplay-policy': 'no-user-gesture-required',
             '--window-size': self.window_size,
             '--remote-debugging-address': HOST,
-            '--remote-debugging-port': str(self.devtools_port),
+            '--remote-debugging-port': '0',
             '--no-sandbox': '',
-            '--disable-crash-reporter': '',
             '--disable-gpu': '',
         }
         cmd = [self.executable]