VOL-4840 - Makefile edits
VOL-4874, VOL-4877, VOL-4840
howto/edit_voltha_docs.rst
--------------------------
* Also need to run "make docs" to generate html content.
Makefile
--------
* (local use) fixed non-functional makefile targets.
* Begin splitting monolithic makefile into reusable libs.
* Extended help to document all targets (make help VERBOSE=1)
* Extended lint to support more targets.
* Replace wildcard target (%) with a list of sphinx-build named targets:
+ Amusing behavior but "make clean" should not initiate "gendocs".
+ Likewise "make lint" should not attempt to generate documentation.
* Introduced simple patching logic for virtualenv. Local users are
now able to use make clean, lint, etc with a newer interpreter.
Baby steps toward VOL-4874.
* Conditional make vars added:
+ NO_PATCH=1
+ NO_OTHER_REPO_DOCS=1 (venv patching not yet supported by foreign repos).
makefiles/consts.mk
-------------------
* basic values: space, dot, null, HIDE.
makefiles/help/include.mk
-------------------------
* makefile target help with context.
makefiles/lint/*.mk
-------------------
* lib makefiles capturing syntax checking logic.
makefiles/patches/include.mk
----------------------------
* Targets for creation and applying python virtualenv patches.
patches/*/patch
patches/python_310_migration.sh
-------------------------------
* Logic used to support dynamic python module pathching in a virtual env directory.
* Module edits are trivial -- attempt to import new modules and fallback to existing import syntax.
Change-Id: I437289a751423217b4193b75b62c4a295db18ffd
diff --git a/patches/README.md b/patches/README.md
new file mode 100644
index 0000000..6019c05
--- /dev/null
+++ b/patches/README.md
@@ -0,0 +1,64 @@
+# Migration support for python 3.10+
+
+## Intent
+
+Applications of patches will enable local use of newer python versions
+available with a recent OS install or package manager installation.
+Makefile targets will fail out of the box w/o modifying collection imports.
+
+Create new patches as needed to help testing along.
+Eventually this hierarchy can be dismantled once the latest interpreter
+is generally in use.
+
+## patches/
+
+This directory contains patch files that can be directly applied to sources
+in the python virtualenv directory. Patches are created to support use of
+three python version variants for the robot ramework:
+ python 2x (deprecated)
+ python >= 3.10 (collections.abc required)
+ python < 3.10 (collections.abc optional)
+
+
+# vst_venv/
+
+Makefile will first create a python virtualenv directory to selectively
+use packages. After setup patches are applied to venv (as a transition)
+to fully enable support for local interpreter use for newer OS installs.
+
+
+# staging/
+
+The staging directory is used for comparison with the vst_venv directory
+to generate patches. Populate the directory with a copy of a cleanly
+patched virtual interpreter then modify files benath/ staging to generate
+a patch from.
+
+% make sterile
+% make vst_venv
+% mkdir staging
+% rsync -rv --checksum vst_venv/. staging/.
+[NOTE] Make python 3.10+ migration edits beneath staging as needed
+% make patch-gather
+% make sterile
+% make lint
+
+
+# makefile targets
+
+make patch-gather:
+ * gather a list of potential sources to edit.
+
+make patch-apply:
+ * Gather a list of patch files.
+ * Apply each in turn to sources in the virtualenv directory.
+
+make patch-create:
+ * make patch-create VENV_NAME="venv_docs" PATCH_PATH="lib/python3.10/site-packages/sphinx/util/typing.py"
+ * make sterile
+ * make help # reinstall virtualenv and verify help target
+
+# Verify installation
+ * make lint -or- make test
+
+# [EOF]
\ No newline at end of file
diff --git a/patches/lib/python3.10/site-packages/robot/utils/normalizing.py/patch b/patches/lib/python3.10/site-packages/robot/utils/normalizing.py/patch
new file mode 100644
index 0000000..fc9ddd4
--- /dev/null
+++ b/patches/lib/python3.10/site-packages/robot/utils/normalizing.py/patch
@@ -0,0 +1,14 @@
+--- vault/lib/python3.10/site-packages/robot/utils/normalizing.py 2022-11-26 06:59:47.438751606 -0500
++++ vst_venv/lib/python3.10/site-packages/robot/utils/normalizing.py 2022-11-26 06:57:29.960476182 -0500
+@@ -13,10 +13,7 @@
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+
+-try: # python >= 3.10
+- from collections.abc import MutableMapping
+-except ImportError: # python 2x
+- from collections import MutableMapping
++from collections import MutableMapping
+
+ from .platform import IRONPYTHON, PY_VERSION, PY3
+ from .robottypes import is_dict_like, is_unicode
diff --git a/patches/lib/python3.10/site-packages/robot/utils/robottypes3.py/patch b/patches/lib/python3.10/site-packages/robot/utils/robottypes3.py/patch
new file mode 100644
index 0000000..399a769
--- /dev/null
+++ b/patches/lib/python3.10/site-packages/robot/utils/robottypes3.py/patch
@@ -0,0 +1,16 @@
+--- vault/lib/python3.10/site-packages/robot/utils/robottypes3.py 2022-11-26 07:00:17.126386733 -0500
++++ vst_venv/lib/python3.10/site-packages/robot/utils/robottypes3.py 2022-11-26 06:57:29.956476232 -0500
+@@ -13,12 +13,7 @@
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+
+-try: # python >= 3.10
+- from collections.abc import Mapping
+- from collections import UserString
+-except ImportError: # python 2x
+- from collections import Mapping, UserString
+-
++from collections import Mapping, UserString
+ from io import IOBase
+
+ from .platform import RERAISED_EXCEPTIONS
diff --git a/patches/lib/python3.10/site-packages/sphinx/util/typing.py/patch b/patches/lib/python3.10/site-packages/sphinx/util/typing.py/patch
new file mode 100644
index 0000000..d049e3b
--- /dev/null
+++ b/patches/lib/python3.10/site-packages/sphinx/util/typing.py/patch
@@ -0,0 +1,19 @@
+--- staging/lib/python3.10/site-packages/sphinx/util/typing.py 2022-11-28 06:22:14.637278053 -0500
++++ venv_docs/lib/python3.10/site-packages/sphinx/util/typing.py 2022-11-28 06:34:36.246702890 -0500
+@@ -34,15 +34,7 @@
+ return ref._eval_type(globalns, localns)
+
+ if sys.version_info > (3, 10):
+-
+- try: # ver >= 3.10.6
+- from types import UnionType as types_Union
+- except ImportError:
+- try: # ver < 3.10.6
+- from types import Union as types_Union
+- except ImportError:
+- types_Union = None
+-
++ from types import Union as types_Union
+ else:
+ types_Union = None
+
diff --git a/patches/python_310_migration.sh b/patches/python_310_migration.sh
new file mode 100755
index 0000000..e02a75c
--- /dev/null
+++ b/patches/python_310_migration.sh
@@ -0,0 +1,93 @@
+#!/bin/bash
+# -----------------------------------------------------------------------
+# Copyright 2022 Open Networking Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# -----------------------------------------------------------------------
+
+set -euo pipefail
+
+dst="vst_venv"
+src="staging"
+pat="patches"
+
+## Update logic to: find . -name 'patch')
+declare -a fyls=()
+fyls+=('lib/python3.10/site-packages/robot/utils/normalizing.py')
+fyls+=('lib/python3.10/site-packages/robot/utils/robottypes3.py')
+fyls+=('lib/python3.10/site-packages/sphinx/util/typing.py')
+
+echo
+echo "==========================================================================="
+echo "CMD: $0"
+echo "PWD: $(/bin/pwd)"
+echo "ARGV: $*"
+echo "==========================================================================="
+
+if [ $# -eq 0 ]; then set -- apply; fi
+
+while [ $# -gt 0 ]; do
+ opt="$1"; shift
+ case "$opt" in
+
+ -*venv) dst="$1"; shift ;;
+
+ help)
+ cat <<EOH
+apply - generate patches from vault source.
+backup - Archive patch directory
+gather - collect potential python files to edit.
+EOH
+ ;;
+
+ apply)
+ pushd "$dst" >/dev/null || { echo "pushd $dst failed"; exit 1; }
+ for fyl in "${fyls[@]}";
+ do
+ # Conditional install, jenkins may not support interpreter yet.
+ if [ ! -e "$fyl" ]; then
+ echo "[SKIP] $fyl"
+ continue
+ fi
+
+ echo "[APPLY] $fyl"
+ patch -R -p1 < "../$pat/$fyl/patch"
+ done
+ popd >/dev/null || { echo "popd $dst failed"; exit 1; }
+ ;;
+
+ backup)
+ mkdir ~/backups
+ pushd "$src" || { echo "pushd $dst failed"; exit 1; }
+ tar czvf ~/backups/vault."$(date '+%Y%m%d%H%M%S')" "${fyls[@]}"
+ popd || { echo "popd $dst failed"; exit 1; }
+ ;;
+
+ gather)
+ for fyl in "${fyls[@]}";
+ do
+ patchDir="$pat/$fyl"
+ mkdir -p "$patchDir"
+ diff -Naur "$src/$fyl" "$dst/$fyl" | tee "$pat/$fyl/patch"
+ done
+ find "$pat" -print
+ ;;
+
+ *)
+ echo "ERROR: Unknown action [$opt]"
+ exit 1
+ ;;
+ esac
+done
+
+# [EOF]