[FIX] base_import: tests failing with older chardet

Some distributions still bundle chardet 2.3, which have some guessing
divergences / incompatibilities with python (resolved in 3.x):

1. UTF-{16,32} with BOM is guessed as LE/BE, which when used to decode
   the string doesn't strip out the BOM. Handle this by checking if
   the BOM is present and converting the encoding name to the
   non-marked version in that case.
2. The ISO-8859-1 test string is guessed as ISO-8859-2 (TBF the
   decoding does make some sense). Allow multiple targets/guesses to
   "fix" that.

closes odoo/odoo#33179

Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>

1 files changed, 12 insertions, 3 deletions

diff --git a/addons/base_import/tests/test_csv_magic.py b/addons/base_import/tests/test_csv_magic.py
index 0a35a4731c1..b7975337361 100644
--- a/addons/base_import/tests/test_csv_magic.py
+++ b/addons/base_import/tests/test_csv_magic.py
@@ -26,18 +26,27 @@ class TestEncoding(ImportCase):
         options.setdefault('separator', '\t')
         test_text = "text\tnumber\tdate\tdatetime\n%s\t1.23.45,67\t\t\n" % text
         for encoding in ['utf-8', 'utf-16', 'utf-32', *encodings]:
+            if isinstance(encoding, tuple):
+                encoding, es = encoding
+            else:
+                es = [encoding]
             preview = self._make_import(
                 test_text.encode(encoding)).parse_preview(dict(options))
 
             self.assertIsNone(preview.get('error'))
             guessed = preview['options']['encoding']
-            self.assertIsNotNone(guessed, encoding)
-            self.assertEqual(codecs.lookup(guessed).name, codecs.lookup(encoding).name)
+            self.assertIsNotNone(guessed)
+            self.assertIn(
+                codecs.lookup(guessed).name, [
+                    codecs.lookup(e).name
+                    for e in es
+                ]
+            )
 
     def test_autodetect_encoding(self):
         """ Check that import preview can detect & return encoding
         """
-        self._check_text("Iñtërnâtiônàlizætiøn", ['iso-8859-1'])
+        self._check_text("Iñtërnâtiônàlizætiøn", [('iso-8859-1', ['iso-8859-1', 'iso-8859-2'])])
 
         self._check_text("やぶら小路の藪柑子。海砂利水魚の、食う寝る処に住む処、パイポパイポ パイポのシューリンガン。", ['eucjp', 'shift_jis', 'iso2022_jp'])