[ADD] base_import: support for XLS and ODS files

* added intermediate ``_read_file`` step dispatching between CSV, ODS
  and XLS(X) parsing
  - primary dispatch on mime type, secondary on file extension:
    + OS may not provide a relevant mime type if no software locally
      installed for the filetype (e.g. Windows sends excel files as
      application/octet-stream if excel is not installed, and ODS files as
      zip if OpenOffice/LibreOffice isn't installed)
    + applications may re-register extensions to non-standard mimetypes
      breaking the dispatcher

    So if the mimetype is found trust it, otherwise try with the
    filename's extension (if any)
  - all readers skip lines with only empty cells in their output
  - ODS and XLS content are "CSVified", rows are converted to arrays of
    unicode strings
* UI altered to not assume CSV files everywhere, and avoid returning
  garbage preview data for non-CSV imports

Various:
* added a ``can_import`` utility function to tests.common, can be used
  to skip tests if an optional Python dependency is not installed (but
  one would like tests to run if the dependency is available)
* fixed datetime issue in ir_fields
* added converter for monetary fields (iso float)
* spreadsheet don't have integers, twiddling required to ensure integral
  values won't be serialized as floats (breaking conversion back to
  Python)
* improved some tests by asserting no error is generated (bonus: logs
  the error message if there is one, rather than just saying the result
  is blown)
* because some systems only provide elderly versions of
  XLRD (e.g. current debian stable provides 0.9.2 from April 2013)
  - keep using xldate_as_tuple instead of 0.9.3's xldate_as_datetime,
    workaround is simple
  - check for xlrd.xlsx in case of pre-0.8 XLRD

Authorship:
Ronak Baxi <rba@odoo.com>
Mohammed Shekha <msh@openerp.com>

Task 10792
Closes #7285

1 files changed, 98 insertions, 0 deletions

diff --git a/addons/base_import/odf_ods_reader.py b/addons/base_import/odf_ods_reader.py
new file mode 100644
index 00000000000..0f0142f3f04
--- /dev/null
+++ b/addons/base_import/odf_ods_reader.py
@@ -0,0 +1,98 @@
+# Copyright 2011 Marco Conti
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# sourced from https://github.com/marcoconti83/read-ods-with-odfpy
+# further altered locally
+
+from odf import opendocument
+from odf.table import Table, TableRow, TableCell
+from odf.text import P
+
+
+class ODSReader(object):
+
+    # loads the file
+    def __init__(self, file=None, content=None, clonespannedcolumns=None):
+        if not content:
+            self.clonespannedcolumns = clonespannedcolumns
+            self.doc = opendocument.load(file)
+        else:
+            self.clonespannedcolumns = clonespannedcolumns
+            self.doc = content
+        self.SHEETS = {}
+        for sheet in self.doc.spreadsheet.getElementsByType(Table):
+            self.readSheet(sheet)
+
+    # reads a sheet in the sheet dictionary, storing each sheet as an
+    # array (rows) of arrays (columns)
+    def readSheet(self, sheet):
+        name = sheet.getAttribute("name")
+        rows = sheet.getElementsByType(TableRow)
+        arrRows = []
+
+        # for each row
+        for row in rows:
+            arrCells = []
+            cells = row.getElementsByType(TableCell)
+
+            # for each cell
+            for count, cell in enumerate(cells, start=1):
+                # repeated value?
+                repeat = 0
+                if count != len(cells):
+                    repeat = cell.getAttribute("numbercolumnsrepeated")
+                if not repeat:
+                    repeat = 1
+                    spanned = int(cell.getAttribute('numbercolumnsspanned') or 0)
+                    # clone spanned cells
+                    if self.clonespannedcolumns is not None and spanned > 1:
+                        repeat = spanned
+
+                ps = cell.getElementsByType(P)
+                textContent = u""
+
+                # for each text/text:span node
+                for p in ps:
+                    for n in p.childNodes:
+                        if n.nodeType == 1 and n.tagName == "text:span":
+                            for c in n.childNodes:
+                                if c.nodeType == 3:
+                                    textContent = u'{}{}'.format(textContent, n.data)
+
+                        if n.nodeType == 3:
+                            textContent = u'{}{}'.format(textContent, n.data)
+
+                if textContent:
+                    if not textContent.startswith("#"):  # ignore comments cells
+                        for rr in range(int(repeat)):  # repeated?
+                            arrCells.append(textContent)
+                else:
+                    for rr in range(int(repeat)):
+                        arrCells.append("")
+
+            # if row contained something
+            if arrCells:
+                arrRows.append(arrCells)
+
+            #else:
+            #    print ("Empty or commented row (", row_comment, ")")
+
+        self.SHEETS[name] = arrRows
+
+    # returns a sheet as an array (rows) of arrays (columns)
+    def getSheet(self, name):
+        return self.SHEETS[name]
+
+    def getFirstSheet(self):
+        return next(iter(self.SHEETS.itervalues()))