@mattyvau, @Bazza, and @Hooverdan.
I’ve finally managed to put some time into this one. As it happens, due to more recent preparation on the rpm build systems side of things, our openSUSE rpms are not affected by this issue. Mainly as they resource and have as a dependency the site-package (system installed) python2-setuptools. Where as our CentOS rpm builds do not have this dependency and thus end up pulling in, via buildout and it’s use of initially an old setuptools to get ez-setup so the setuptools can ‘self-update’. And the core of the issue is that it’s tricky to pin within this convoluted, self-updating process. But if the latter stages (used in building the rpm) of this same process finds a site-package setuptools update for python2 that is newer than the initially downloaded 33.1.1 version required to run ez_setup.py then it doesn’t grab that days version from PyPi; such as is happening in our CentOS build.
Proper fix is to move to Python 3 of course and that process is underway, but slowly. However once we have our ‘Built on openSUSE’ Stable channel rpm out we can focus on addressing this large technical debt in our testing channel. The Buildout project themselves, and specifically their bootstrap part that we use, have been moving to and testing against Python3 versions for some time and I believe are under way with incorporating the depricated elements they need from older setuptools so hopefully we can leverage their work, and contribute back in necessary, as we move our entire source and rpm build system and all our own source and dependencies over to Python 3.
Incidentally the easiest re-producer I’ve found is our emergency rock-on db/image cleaner script:
Leap15.2 beta
leap15-2:~ # /opt/rockstor/bin/delete-rockon
Delete metadata, containers and images of a Rock-on
Usage: /opt/rockstor/bin/delete-rockon <rockon name>
Current CentOS rpm:
[root@rockprod ~]# /opt/rockstor/bin/delete-rockon
/opt/rockstor/eggs/setuptools-46.1.3-py2.7.egg/pkg_resources/py2_warn.py:21: UserWarning: Setuptools will stop working on Python 2
************************************************************
You are running Setuptools on Python 2, which is no longer
supported and
>>> SETUPTOOLS WILL STOP WORKING <<<
in a subsequent release (no sooner than 2020-04-20).
Please ensure you are installing
Setuptools using pip 9.x or later or pin to `setuptools<45`
in your environment.
If you have done those things and are still encountering
this message, please follow up at
https://bit.ly/setuptools-py2-warning.
************************************************************
sys.version_info < (3,) and warnings.warn(pre + "*" * 60 + msg + "*" * 60)
Delete metadata, containers and images of a Rock-on
Usage: /opt/rockstor/bin/delete-rockon <rockon name>
But here’s another wrinkle. The python2-setuptools available in both Leap15.1 and Leap15.2beta do present as an update to the initially downloaded by buildout bootstrap.py 33.1.1 update:
Leap15.1
leap15-1:~ # rpm -qa | grep setuptools
python3-setuptools-40.5.0-lp151.1.1.noarch
python2-setuptools-40.5.0-lp151.1.1.noarch
leap15-1:~ # ls -la /opt/rockstor/develop-eggs/
total 8
drwxr-xr-x 1 root root 72 Apr 20 00:13 .
drwxr-xr-x 1 root root 362 Apr 20 00:13 ..
-rw-r--r-- 1 root root 33 Apr 19 23:49 rockstor.egg-link
-rw-r--r-- 1 root root 32 Apr 19 23:49 setuptools.egg-link
leap15-1:~ # cat /opt/rockstor/develop-eggs/setuptools.egg-link
/usr/lib/python2.7/site-packages
Leap15.2beta
leap15-2:~ # rpm -qa | grep setuptools
python3-setuptools-40.5.0-lp152.2.3.noarch
python2-setuptools-40.5.0-lp152.2.3.noarch
leap15-2:~ # ls -la /opt/rockstor/develop-eggs/
total 8
drwxr-xr-x 1 root root 72 Apr 19 23:50 .
drwxr-xr-x 1 root root 362 Apr 19 23:50 ..
-rw-r--r-- 1 root root 33 Apr 19 23:49 rockstor.egg-link
-rw-r--r-- 1 root root 32 Apr 19 23:49 setuptools.egg-link
leap15-2:~ # cat /opt/rockstor/develop-eggs/setuptools.egg-link
/usr/lib/python2.7/site-packages
Where as our CentOS rpm build without a site install of python-setuptools (it’s name in CentOS) ends up pulling in the very latest egg as per the
which is now, in 3.9.2-57 rpm even newer.
However the site package (distro provided) for CentOS is:
[root@rockprod rockstor-dev]# rpm -qa | grep setuptools
python-setuptools-0.9.8-7.el7.noarch
python3-setuptools-39.2.0-10.el7.noarch
And it looks like in this case our tower of tools/dependencies build system chooses upgrade to the latest from Pypi anyway. So what works for us in our ‘Built on openSUSE’ variants does not serve similarly in our CentOS build. This looks to be due to the extremely old version of python-setuptools in CentOS.
During the build of all the openSUSE variants the plain source and the production build (used in creating the rpm) both stick with this 3.1.1 egg up until late in the build process, but some build element ends up substituting the site-package in our openSUSE builds, which works for us, and the overly new and shiny in the CentOS bilds, which does not work for us; or soon won’t anyway.
I am currently investigating our options on this one.
I would obviously prefer to use the distro variant as it’s build in concert with the exact python version we run under (the distro python2 variant for now).
So this has turned out to be quite a deep problem and is, as indicated all around, just another call to our technical debt. We are already using the latest version of six across all distro rpms since 3.9.2-53 so that should assist with our in-progress move to Python 3. And again this is pulled down during our build except for in the cases of Leap15.2beta & Tumbleweed rpm builds as they have sufficiently new enough packages to not have to include them in the rpm itself.
So just thought I’d use this thread to document my findings / progress thus far on this issue and the direction I’m trying to take is to depend upon as many distro python packages as possible as they are build together by the upstream distro maintainer and so are more likely to play nicely together. But as detailed above this is getting a little tricky. So this technical debt must be at least part our next focus, once we have feature parity, or near enough, in our ‘Built on openSUSE’ offerings. And I’d say we are actually pretty close. Plus we have to pick our battles here also.
Hope that helps.