[IMP] base_import: attempt to automatically guess the separator

It would probably be even better to iterate the file content and get
the non-quoted non-alphanumeric characters as separator
candidates (instead of a hard-coded list) however Python does not seem
to have a decoding iterator (taking bytes and yielding an iterator of
codepoints or even grapheme clusters) — incidentally uniseg seems to
require up-front decoding as well — so that's not really convenient as
we may be dealing with large-ish files and not want to load it
entirely in memory.

An alternative would be to use TextIOWrapper and iterate the file by
buffers of a few ks, and classify that based on either codepoints or
grapheme clusters.

1 files changed, 22 insertions, 2 deletions

diff --git a/addons/base_import/models/base_import.py b/addons/base_import/models/base_import.py
index 31de56d79e3..3055bcc9e3c 100644
--- a/addons/base_import/models/base_import.py
+++ b/addons/base_import/models/base_import.py
@@ -322,10 +322,30 @@ class Import(models.TransientModel):
         if encoding != 'utf-8':
             csv_data = csv_data.decode(encoding).encode('utf-8')
 
+        separator = options.get('separator')
+        if not separator:
+            # default for unspecified separator so user gets a message about
+            # having to specify it
+            separator = ','
+            for candidate in (',', ';', '\t', ' ', '|', unicodedata.lookup('unit separator')):
+                # pass through the CSV and check if all rows are the same
+                # length & at least 2-wide assume it's the correct one
+                it = pycompat.csv_reader(io.BytesIO(csv_data), quotechar=options['quoting'], delimiter=candidate)
+                w = None
+                for row in it:
+                    width = len(row)
+                    if w is None:
+                        w = width
+                    if width == 1 or width != w:
+                        break # next candidate
+                else: # nobreak
+                    separator = options['separator'] = candidate
+                    break
+
         csv_iterator = pycompat.csv_reader(
             io.BytesIO(csv_data),
-            quotechar=str(options['quoting']),
-            delimiter=str(options['separator']))
+            quotechar=options['quoting'],
+            delimiter=separator)
 
         return (
             row for row in csv_iterator