[IMP] refactored translation system to merge web translations with addons translations

- Moved the web *.po files to /i18n to be consistent
  with the addons convention. Using /po was considered
  for a while because it played better with LP's auto-
  detection of PO Templates, but that is not necessary
  anymore, we now have full control on LP templates.

- In order to support addons that contain translations
  for both the web addon and the regular addon part,
  both kinds of translations are now merged in a single
  addon/i18n/addon.pot file. Terms that are used by
  the web part are now marked with a PO annotation:
     #. openerp-web
  so the web client can recognize them and only load
  the relevant translations in the browser memory.
  This is important because a complete PO file can
  be rather large, e.g. account/i18n/de.po = 400KB.

- The web translation export scripts were updated to
  behave properly for addons that have a non-web
  part, and will merge the web translation in the
  original POT file, annotating the web translations
  as needed. These scripts are Unix-only and meant
  to be used by OpenERP packagers when needed.

- The GetText spec says that PO auto-comments indicating
  the source location have this form:
   #: /path/to/file:lineno
  However OpenERP's POT export system defaults to a modified
  version of this format with an extra 'type' field:
   #: type:/path/to/file:lineno
  The babel extractors we use have the GetText format 
  hardcoded so a small patch is needed on the server
  to make it more lenient and accept the standard
  source annotation, defaulting to 'code' type.
  This does not matter for openerp-web, but makes sure
  the server will not fail to load the new PO files 
  that contain openerp-web translations with standard
  annotations.
  The patch for making the server more lenient was
  checked in trunk at revision 4002 
  rev-id odo@openerp.com-20120202143210-05p1w24t6u77cyv8

- The existing translation sync and export wizards for
  regular addons have not been updated to consider
  web addons, so for the time being we will have
  to export regular addons terms first, and run the
  web export script (gen_translations.sh) on the
  addons directory afterwards. This could be improved
  later.

As soon as this change is merged we will have to
perform a full update of addons translation
templates in order to include the web terms as well.

bzr revid: odo@openerp.com-20120202145603-ffo0il0qnfp3r6gt

1 files changed, 54 insertions, 17 deletions

diff --git a/npybabel.py b/npybabel.py
index 81cd6abf60a..b2888cd4a2c 100755
--- a/npybabel.py
+++ b/npybabel.py
@@ -5,33 +5,50 @@ import sys
 from pkg_resources import load_entry_point
 import re
 import json
-import xml.etree.ElementTree as elt
+from lxml import etree as elt
+from babel.messages import extract
 
 if __name__ == '__main__':
     sys.exit(
         load_entry_point('Babel==0.9.6', 'console_scripts', 'pybabel')()
     )
-    
+
 XMLJS_EXPR = re.compile(r"""(?:\_t *\( *((?:"(?:[^"\\]|\\.)*")|(?:'(?:[^'\\]|\\.)*')) *\))""")
 
+TRANSLATION_FLAG_COMMENT = "openerp-web"
+
 def extract_xmljs(fileobj, keywords, comment_tags, options):
+    """Extract messages from Javascript code embedded into XML documents.
+    This complements the ``extract_javascript`` extractor which works
+    only on pure .js files, and the``extract_qweb`` extractor, which only
+    extracts XML text.
+
+    :param fileobj: the file-like object the messages should be extracted
+                    from
+    :param keywords: a list of keywords (i.e. function names) that should
+                     be recognized as translation functions
+    :param comment_tags: a list of translator tags to search for and
+                         include in the results
+    :param options: a dictionary of additional options (optional)
+    :return: an iterator over ``(lineno, funcname, message, comments)``
+             tuples
+    :rtype: ``iterator``
+    """
     content = fileobj.read()
     found = XMLJS_EXPR.finditer(content)
-    result = []
     index = 0
     line_nbr = 0
     for f in found:
-        mes = f.group(1)
-        mes = json.loads(mes)
+        msg = f.group(1)
+        msg = json.loads(msg)
         while index < f.start():
             if content[index] == "\n":
                 line_nbr += 1
             index += 1
-        result.append((line_nbr, None, mes, ""))
-    return result
+        yield (line_nbr, None, msg, [TRANSLATION_FLAG_COMMENT])
 
 def extract_qweb(fileobj, keywords, comment_tags, options):
-    """Extract messages from XXX files.
+    """Extract messages from qweb template files.
     :param fileobj: the file-like object the messages should be extracted
                     from
     :param keywords: a list of keywords (i.e. function names) that should
@@ -44,22 +61,42 @@ def extract_qweb(fileobj, keywords, comment_tags, options):
     :rtype: ``iterator``
     """
     result = []
-    def handle_text(str):
-        str = (str or "").strip()
-        if not str:
-            return
-        result.append((0, None, str, ""))
-    
+    def handle_text(text, lineno):
+        text = (text or "").strip()
+        if len(text) > 1: # Avoid mono-char tokens like ':' ',' etc.
+            result.append((lineno, None, text, [TRANSLATION_FLAG_COMMENT]))
+
     def iter_elements(current_element):
         for el in current_element:
             if "t-js" not in el.attrib and \
                     not ("t-jquery" in el.attrib and "t-operation" not in el.attrib) and \
                     not ("t-translation" in el.attrib and el.attrib["t-translation"].strip() == "off"):
-                handle_text(el.text)
+                handle_text(el.text, el.sourceline)
                 iter_elements(el)
-            handle_text(el.tail)
-    
+            handle_text(el.tail, el.sourceline)
+
     tree = elt.parse(fileobj)
     iter_elements(tree.getroot())
 
     return result
+
+def extract_javascript(fileobj, keywords, comment_tags, options):
+    """Extract messages from Javascript source files. This extractor delegates
+    to babel's buit-in javascript extractor, but adds a special comment
+    used as a flag to identify web translations. 
+
+    :param fileobj: the file-like object the messages should be extracted
+                    from
+    :param keywords: a list of keywords (i.e. function names) that should
+                     be recognized as translation functions
+    :param comment_tags: a list of translator tags to search for and
+                         include in the results
+    :param options: a dictionary of additional options (optional)
+    :return: an iterator over ``(lineno, funcname, message, comments)``
+             tuples
+    :rtype: ``iterator``
+    """
+    for (message_lineno, funcname, messages, comments) in \
+        extract.extract_javascript(fileobj, keywords, comment_tags, options):
+        comments.append(TRANSLATION_FLAG_COMMENT)
+        yield (message_lineno, funcname, messages, comments)