[Catalog-sig] setuptools/distribute/easy_install/pkg

Post by M.-A. Lemburg
If I place two files named
egenix_mxodbc_connect_client-2.0.2-py2.6.egg
egenix-mxodbc-connect-client-2.0.2.win32-py2.6.prebuilt.zip
into the same directory and let easy_install running on Linux
scan this, it considers the second file for Windows as best
match.
Is the algorithm used for determining the best match documented
somewhere ?
I've had a look at the implementation, but this left me rather
clueless.
I thought that setuptools would prefer the .egg file over
the prebuilt .zip file - binary files being easier to install
than "source" files.

After some experiments, I found that the follow change
in filename (swapping platform and python version, in addition
to use '-' instead of '.) works:

egenix-mxodbc-connect-client-2.0.2-py2.6-win32.prebuilt.zip

OTOH, this one doesn't (notice the difference ?):

egenix-mxodbc-connect-client-2.0.2.py2.6-win32.prebuilt.zip

The logic behind all this looks rather fragile to me.

Thanks,
--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Mar 12 2013)

Python Projects, Consulting and Support ... http://www.egenix.com/
mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

PJ Eby

2013-03-12 21:26:22 UTC

After some experiments, I found that the follow change
in filename (swapping platform and python version, in addition
egenix-mxodbc-connect-client-2.0.2-py2.6-win32.prebuilt.zip
egenix-mxodbc-connect-client-2.0.2.py2.6-win32.prebuilt.zip
The logic behind all this looks rather fragile to me.

easy_install only guarantees sane version parsing for distribution
files built using setuptools' naming algorithms. If you use
distutils, it can only make guesses, because the distutils does not
have a completely unambiguous file naming scheme. And if you are
naming the files by hand, God help you. ;-)

M.-A. Lemburg

2013-03-14 10:07:07 UTC

The problem appears to be a bug in setuptools' package_index.py.

The function interpret_distro_name() creates a set of possible
separations of the found name into project name and version.

It does find the right separation, but for some reason, the
code using that function does not check the found project
names against the project name the user is trying to install,
but simply takes the last entry of the list returned by the
above function.

As a result, easy_install downloads and tries to install
project files that don't match the project name in some
cases.

Here's another example where it fails (say you're on a x64 Linux box):

# easy_install egenix-pyopenssl

As example, say it finds these distribution files:

'egenix-pyopenssl-0.13.1.1.0.1.5-py2.7_ucs2-linux-x86_64-prebuilt.zip',
'egenix_pyopenssl-0.13.1.1.0.1.5-py2.7-linux-x86_64.egg',
'egenix-pyopenssl-0.13.1.1.0.1.5-py2.7_ucs2-macosx-10.5-x86_64-prebuilt.zip',
'egenix-pyopenssl-0.13.1.1.0.1.5-py2.7_ucs4-macosx-10.5-x86_64-prebuilt.zip',

It then creates different interpretations of those names, puts
them in a list and sorts them. Here's the end of that list:

egenix-pyopenssl; 0.13.1.1.0.1.5 <<-- this would be the correct .egg file
egenix-pyopenssl; 0.13.1.1.0.1.5-py2.7-ucs2-linux-x86-64-prebuilt
egenix-pyopenssl; 0.13.1.1.0.1.5-py2.7-ucs2-macosx-10.5-x86-64-prebuilt
egenix-pyopenssl; 0.13.1.1.0.1.5-py2.7-ucs4-macosx-10.5-x86-64-prebuilt
egenix-pyopenssl-0.13.1.1.0.1.5-py2.7-ucs2-macosx; 10.5-x86-64-prebuilt
egenix-pyopenssl-0.13.1.1.0.1.5-py2.7-ucs4-macosx; 10.5-x86-64-prebuilt

It picks the last entry, which would be for a project called
"egenix-pyopenssl-0.13.1.1.0.1.5-py2.7-ucs4-macosx" - not the one
the user searched.

I'm trying to find a way to get it to use the correct .egg file
The .egg files does have precedence over the other files, since
easy_install regards them as source files with lower precedence.

This is important, because the /simple/ index page will have links
not only to .egg files, but also to our prebuilt .zip files,
which use a source file compatible setup.py interface.
--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Mar 14 2013)

Post by M.-A. Lemburg
Python Projects, Consulting and Support ... http://www.egenix.com/
mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

PJ Eby

2013-03-14 16:39:44 UTC

Actually, that's not quite true. It's picking:

egenix-pyopenssl; 0.13.1.1.0.1.5-py2.7-ucs4-macosx-10.5-x86-64-prebuilt

Because it thinks that
'0.13.1.1.0.1.5-py2.7-ucs4-macosx-10.5-x86-64-prebuilt' is a higher
version than 0.13.1.1.0.1.5.

It does also record the possibility you mentioned, but it doesn't pick
that one. The project names actually *do* have to match.

If you open a ticket on the setuptools tracker, 'll try to see if I
can get it to recognize that strings like py2.7, macosx, ucs, and the
like are terminators for a version number. I don't know how
successful I'll be, though. Basically, those zip files are (I assume)
bdist_dumb distributions being taken for source distributions, and
easy_install doesn't actually support bdist_dumb files at the moment.

M.-A. Lemburg

2013-03-14 18:11:59 UTC

egenix-pyopenssl; 0.13.1.1.0.1.5-py2.7-ucs4-macosx-10.5-x86-64-prebuilt
Because it thinks that
'0.13.1.1.0.1.5-py2.7-ucs4-macosx-10.5-x86-64-prebuilt' is a higher
version than 0.13.1.1.0.1.5.
It does also record the possibility you mentioned, but it doesn't pick
that one. The project names actually *do* have to match.

Ah, ok, that makes sense then.

Is there any way to have "0.13.1.1.0.1.5-<something>" sort before
"0.13.1.1.0.1.5" ? (e.g. like is done for release candidates)

Ideally, I'd like to get this to work without any changes
to setuptools, even though it would of course be better
not to take stuff after a Python version marker into account
when looking for a package version (since the Python marker
is actually a new component in the file name).

Post by M.-A. Lemburg
If you open a ticket on the setuptools tracker, 'll try to see if I
can get it to recognize that strings like py2.7, macosx, ucs, and the
like are terminators for a version number. I don't know how
successful I'll be, though. Basically, those zip files are (I assume)
bdist_dumb distributions being taken for source distributions, and
easy_install doesn't actually support bdist_dumb files at the moment.

If you could point me to that tracker, I'll open a ticket :-)

Thanks,
--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Mar 14 2013)

Post by PJ Eby
Python Projects, Consulting and Support ... http://www.egenix.com/
mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

PJ Eby

2013-03-14 21:03:14 UTC