VOL-4840 - Makefile edits
VOL-4874, VOL-4877, VOL-4840
howto/edit_voltha_docs.rst
--------------------------
* Also need to run "make docs" to generate html content.
Makefile
--------
* (local use) fixed non-functional makefile targets.
* Begin splitting monolithic makefile into reusable libs.
* Extended help to document all targets (make help VERBOSE=1)
* Extended lint to support more targets.
* Replace wildcard target (%) with a list of sphinx-build named targets:
+ Amusing behavior but "make clean" should not initiate "gendocs".
+ Likewise "make lint" should not attempt to generate documentation.
* Introduced simple patching logic for virtualenv. Local users are
now able to use make clean, lint, etc with a newer interpreter.
Baby steps toward VOL-4874.
* Conditional make vars added:
+ NO_PATCH=1
+ NO_OTHER_REPO_DOCS=1 (venv patching not yet supported by foreign repos).
makefiles/consts.mk
-------------------
* basic values: space, dot, null, HIDE.
makefiles/help/include.mk
-------------------------
* makefile target help with context.
makefiles/lint/*.mk
-------------------
* lib makefiles capturing syntax checking logic.
makefiles/patches/include.mk
----------------------------
* Targets for creation and applying python virtualenv patches.
patches/*/patch
patches/python_310_migration.sh
-------------------------------
* Logic used to support dynamic python module pathching in a virtual env directory.
* Module edits are trivial -- attempt to import new modules and fallback to existing import syntax.
Change-Id: I437289a751423217b4193b75b62c4a295db18ffd
diff --git a/makefiles/consts.mk b/makefiles/consts.mk
new file mode 100644
index 0000000..c34d119
--- /dev/null
+++ b/makefiles/consts.mk
@@ -0,0 +1,28 @@
+# -*- makefile -*-
+# -----------------------------------------------------------------------
+# Copyright 2022 Open Networking Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# -----------------------------------------------------------------------
+
+null :=#
+space :=$(null) $(null)
+dot :=.
+HIDE ?=@
+
+# use bash for pusdh/popd and quick failures.
+# virtual env(s) activate has undefined vars so no -u
+# ^---+ verify this is still true
+export SHELL := bash -e -o pipefail
+
+# [EOF]
diff --git a/makefiles/help/include.mk b/makefiles/help/include.mk
new file mode 100644
index 0000000..ccba3f4
--- /dev/null
+++ b/makefiles/help/include.mk
@@ -0,0 +1,31 @@
+# -*- makefile -*-
+# -----------------------------------------------------------------------
+# Copyright 2017-2022 Open Networking Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# -----------------------------------------------------------------------
+
+# Parent makefile should include this early so help
+# message will be prefixed by a usage statement.
+help ::
+ @echo "Usage: $(MAKE) [options] [target] ..."
+ @echo
+ @echo '[CLEAN]'
+ @echo ' clean Remove generated targets'
+ @echo ' sterile clean + remove virtual env interpreter install'
+ @echo
+ @echo '[HELP]'
+ @echo ' help Display program help'
+ @echo ' help-verbose Display additional targets and help'
+
+# [EOF]
diff --git a/makefiles/help/trailer.mk b/makefiles/help/trailer.mk
new file mode 100644
index 0000000..bf548db
--- /dev/null
+++ b/makefiles/help/trailer.mk
@@ -0,0 +1,27 @@
+# -*- makefile -*-
+# -----------------------------------------------------------------------
+# Copyright 2017-2022 Open Networking Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# -----------------------------------------------------------------------
+
+# include this makefile late so text will be displayed at the end6
+
+help ::
+ @echo
+ @echo '[NOTE: python 3.10+]'
+ @echo ' The interpreter is not yet fully supported across foreign repositories.'
+ @echo ' While working locally, if make fails to build a target try:'
+ @echo ' $(MAKE) $${target} NO_OTHER_REPO_DOCS=1'
+
+# [EOF]
diff --git a/makefiles/help/variables.mk b/makefiles/help/variables.mk
new file mode 100644
index 0000000..a50b233
--- /dev/null
+++ b/makefiles/help/variables.mk
@@ -0,0 +1,36 @@
+# -*- makefile -*-
+# -----------------------------------------------------------------------
+# Copyright 2017-2022 Open Networking Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# -----------------------------------------------------------------------
+
+# Include variables.mk after library makefiles have been included
+
+ifdef VERBOSE
+ help :: help-variables
+else
+ help ::
+ @echo
+ @echo '[VARIABLES] - Conditional makefile behavior'
+ @echo ' see also: help-variables'
+endif
+
+help-variables:
+ @echo
+ @echo '[VARIABLES] - Conditional makefile behavior'
+ @echo ' NO_PATCHES= Do not apply patches to the python virtualenv'
+ @echo ' NO_OTHER_REPO_DOCS= No foreign repos, only apply target to local sources.'
+ @echo ' VERBOSE= Display extended help'
+
+# [EOF]
diff --git a/makefiles/lint.mk b/makefiles/lint.mk
new file mode 100644
index 0000000..cb4b0a7
--- /dev/null
+++ b/makefiles/lint.mk
@@ -0,0 +1,27 @@
+# -*- makefile -*-
+# -----------------------------------------------------------------------
+# Copyright 2017-2022 Open Networking Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# -----------------------------------------------------------------------
+
+help::
+ @echo
+ @echo "[LINT]"
+
+include $(MAKEDIR)/lint/json.mk
+include $(MAKEDIR)/lint/python.mk
+include $(MAKEDIR)/lint/robot.mk
+include $(MAKEDIR)/lint/yaml.mk
+
+# [EOF]
diff --git a/makefiles/lint/json.mk b/makefiles/lint/json.mk
new file mode 100644
index 0000000..0a11c71
--- /dev/null
+++ b/makefiles/lint/json.mk
@@ -0,0 +1,35 @@
+# -*- makefile -*-
+# -----------------------------------------------------------------------
+# Copyright 2017-2022 Open Networking Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# -----------------------------------------------------------------------
+
+JSON_FILES ?= $(error JSON_FILES= is rqeuired)
+
+.PHONY: lint-json
+
+lint : lint-json
+
+lint-json: vst_venv
+ source ./$</bin/activate \
+ ; set -u \
+ ; for jsonfile in $(JSON_FILES); do \
+ echo "Validating json file: $$jsonfile" ;\
+ python -m json.tool $$jsonfile > /dev/null ;\
+ done
+
+help::
+ @echo " lint-json Syntax check json sources"
+
+# [EOF]
diff --git a/makefiles/lint/python.mk b/makefiles/lint/python.mk
new file mode 100644
index 0000000..5aecd58
--- /dev/null
+++ b/makefiles/lint/python.mk
@@ -0,0 +1,34 @@
+# -*- makefile -*-
+# -----------------------------------------------------------------------
+# Copyright 2017-2022 Open Networking Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# -----------------------------------------------------------------------
+
+PYTHON_FILES ?= $(error PYTHON_FILES= is required)
+
+.PHONY: lint-python
+
+lint : lint-python
+
+# check deps for format and python3 cleanliness
+lint-python: vst_venv
+ source ./$</bin/activate \
+ ; set -u \
+ ; pylint --py3k $(PYTHON_FILES) \
+ ; flake8 --max-line-length=99 --count $(PYTHON_FILES)
+
+help::
+ @echo " lint-python Syntax check using pylint and flake8"
+
+# [EOF]
diff --git a/makefiles/lint/robot.mk b/makefiles/lint/robot.mk
new file mode 100644
index 0000000..ec5579e
--- /dev/null
+++ b/makefiles/lint/robot.mk
@@ -0,0 +1,41 @@
+# -*- makefile -*-
+# -----------------------------------------------------------------------
+# Copyright 2017-2022 Open Networking Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# -----------------------------------------------------------------------
+
+ROBOT_FILES ?= $(error ROBOT_FILES= is required)
+
+LINT_ARGS ?= --verbose --configure LineTooLong:130 -e LineTooLong \
+ --configure TooManyTestSteps:65 -e TooManyTestSteps \
+ --configure TooManyTestCases:50 -e TooManyTestCases \
+ --configure TooFewTestSteps:1 \
+ --configure TooFewKeywordSteps:1 \
+ --configure FileTooLong:2000 -e FileTooLong \
+ -e TrailingWhitespace
+
+
+.PHONY: lint-robot
+
+lint : lint-robot
+
+lint-robot: vst_venv
+ source ./$</bin/activate \
+ ; set -u \
+ ; rflint $(LINT_ARGS) $(ROBOT_FILES)
+
+help::
+ @echo " lint-robot Syntax check robot sources using rflint"
+
+# [EOF]
diff --git a/makefiles/lint/yaml.mk b/makefiles/lint/yaml.mk
new file mode 100644
index 0000000..c9cb392
--- /dev/null
+++ b/makefiles/lint/yaml.mk
@@ -0,0 +1,32 @@
+# -*- makefile -*-
+# -----------------------------------------------------------------------
+# Copyright 2017-2022 Open Networking Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# -----------------------------------------------------------------------
+
+YAML_FILES ?= $(error YAML_FILES= is required)
+
+.PHONY: lint-yaml
+
+lint : lint-yaml
+
+lint-yaml: vst_venv
+ source ./$</bin/activate \
+ ; set -u \
+ ; yamllint -s $(YAML_FILES)
+
+help::
+ @echo " lint-yaml Syntax check yaml source using yamllint"
+
+# [EOF]
diff --git a/makefiles/patches/help.mk b/makefiles/patches/help.mk
new file mode 100644
index 0000000..ce459fd
--- /dev/null
+++ b/makefiles/patches/help.mk
@@ -0,0 +1,44 @@
+# -*- makefile -*-
+# -----------------------------------------------------------------------
+# Copyright 2017-2022 Open Networking Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# -----------------------------------------------------------------------
+
+ifdef VERBOSE
+ help :: help-patches
+else
+ help ::
+ @echo
+ @echo "[PATCHES] - helper on the road to python 3.10+ based testing"
+ @echo ' see also: help-patches'
+endif
+
+help-patches:
+ @echo
+ @echo "[PATCHES] - helper on the road to python 3.10+ based testing"
+ @echo " patch-apply Apply patches to the virtualenv directory"
+ @echo " patch-create"
+ @echo " patch-gather Gather a list of potential patch sources"
+ @echo " patch-init Clone the virtualenv directory for patch creation."
+
+
+
+
+help-trailer ::
+ @echo "[SEE ALSO] patches-help"
+
+help-verbose:
+ $(HIDE)$(MAKE) --no-print-directory help VERBOSE=1
+
+# [EOF]
diff --git a/makefiles/patches/include.mk b/makefiles/patches/include.mk
new file mode 100644
index 0000000..1c0c869
--- /dev/null
+++ b/makefiles/patches/include.mk
@@ -0,0 +1,65 @@
+# -*- makefile -*-
+# -----------------------------------------------------------------------
+# Copyright 2022 Open Networking Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# -----------------------------------------------------------------------
+
+include $(MAKEDIR)/patches/help.mk
+
+patch-gather-args += --exclude=Makefile
+patch-gather-args += --exclude-dir=vault
+patch-gather-args += --exclude-dir=makefiles
+patch-gather-args += --exclude-dir=staging
+patch-gather-args += --exclude-dir=patches
+
+# patch-gather-args += -e 'from collections import'
+patch-gather-args += '-e' 'from collections import Mapping'
+patch-gather-args += '-e' 'from collections import MutableMapping'
+
+VENV_NAME ?= $(error $(MAKE) VENV_NAME= is required)
+PATCH_PATH ?= $(error $(MAKE) PATCH_PATH= is required)
+
+## -----------------------------------------------------------------------
+## -----------------------------------------------------------------------
+patch-gather:
+ grep -r $(patch-gather-args)
+
+## -----------------------------------------------------------------------
+## -----------------------------------------------------------------------
+patch-diff:
+ $(HIDE)diff -qr staging $(VENV_NAME) \
+ | awk '{print "# diff -Naur "$$2" "$$4}' \
+ | tee $@.log
+
+## -----------------------------------------------------------------------
+## -----------------------------------------------------------------------
+patch-create:
+ mkdir -p patches/$(PATCH_PATH)
+ diff -Naur staging/$(PATCH_PATH) $(VENV_NAME)/$(PATCH_PATH) | tee patches/$(PATCH_PATH)/patch
+ exit 1
+
+## -----------------------------------------------------------------------
+## -----------------------------------------------------------------------
+patch-init:
+ find "$(VENV_NAME)" -name '__pycache__' -type d -print0 \
+ | xargs -I'{}' --null --no-run-if-empty $(RM) -r {}
+ mkdir -p staging
+ rsync -rv --checksum "$(VENV_NAME)/." "staging/."
+ @echo "Modify files beneath staging/ to create a patch source"
+
+# [SEE ALSO]
+# ---------------------------------------------------------------------------
+# https://bobbyhadz.com/blog/python-importerror-cannot-import-name-mapping-from-collections
+# ---------------------------------------------------------------------------
+# [EOF]
diff --git a/makefiles/virtualenv.mk b/makefiles/virtualenv.mk
new file mode 100644
index 0000000..faa7c60
--- /dev/null
+++ b/makefiles/virtualenv.mk
@@ -0,0 +1,32 @@
+# -*- makefile -*-
+## -----------------------------------------------------------------------
+# Copyright 2017-2022 Open Networking Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+## -----------------------------------------------------------------------
+
+# virtualenv for the robot tools
+# VOL-2724 Invoke pip via python3 to avoid pathname too long on QA jobs
+
+vst_venv-2 : vst_venv/bin/activate requirements.txt
+
+vst_venv/bin/activate:
+ virtualenv -p python3 $@ ;\
+ source ./$@/bin/activate ;\
+ python -m pip install -r requirements.txt
+
+zzz: xyz
+xyz: vst_venv/bin/activate
+ source vst_venv/bin/activate
+
+# [EOF]
diff --git a/makefiles/voltha/docs-catchall-targets.mk b/makefiles/voltha/docs-catchall-targets.mk
new file mode 100644
index 0000000..72ed700
--- /dev/null
+++ b/makefiles/voltha/docs-catchall-targets.mk
@@ -0,0 +1,44 @@
+# -*- makefile -*-
+# -----------------------------------------------------------------------
+# Copyright 2022 Open Networking Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# -----------------------------------------------------------------------
+
+voltha-docs-catchall += docs
+voltha-docs-catchall += html
+voltha-docs-catchall += dirhtml
+voltha-docs-catchall += singlehtml
+voltha-docs-catchall += pickle
+voltha-docs-catchall += json
+voltha-docs-catchall += htmlhelp
+voltha-docs-catchall += qthelp
+voltha-docs-catchall += devhelp
+voltha-docs-catchall += epub
+voltha-docs-catchall += latex
+voltha-docs-catchall += latexpdf
+voltha-docs-catchall += latexpdfja
+voltha-docs-catchall += text
+voltha-docs-catchall += man
+voltha-docs-catchall += texinfo
+voltha-docs-catchall += info
+voltha-docs-catchall += gettext
+voltha-docs-catchall += changes
+voltha-docs-catchall += xml
+voltha-docs-catchall += pseudoxml
+voltha-docs-catchall += linkcheck
+voltha-docs-catchall += doctest
+voltha-docs-catchall += coverage
+# voltha-docs-catchall += clean
+
+# [EOF]