You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
125 lines
5.3 KiB
125 lines
5.3 KiB
|
|
===========================
|
|
Grant Proposal for CFFI 1.0
|
|
===========================
|
|
|
|
*Accepted by the PSF board on April 4, 2015*
|
|
|
|
This Grant Proposal is to give a boost towards "CFFI 1.0". Two main
|
|
issues with the current CFFI need to be solved: the difficulties of
|
|
installation, and the potentially large time taken at import.
|
|
|
|
1. The difficulties of installation can be seen from outside by looking
|
|
at various workarounds and 3rd-party documentation that have grown into
|
|
existence. For example, the `setup.py` of projects like cryptography,
|
|
PyNaCl and bcrypt deploys workarounds that are explicitly documented in
|
|
https://caremad.io/2014/11/distributing-a-cffi-project/.
|
|
|
|
2. The time taken at import is excessive in some cases. For example,
|
|
importing `pygame-cffi` on a Raspberry Pi ARM board takes on the order
|
|
of 10 to 20 seconds (and this is the "fast" case where the compiler
|
|
doesn't need to be invoked any more).
|
|
|
|
|
|
Technical Overview
|
|
------------------
|
|
|
|
"CFFI" is an existing Python project which complements the ctypes,
|
|
SWIG and Cython approaches to ease writing C Extension Modules for
|
|
Python. It has several advantages over the previous approaches, which
|
|
are presented at the start of the documentation at
|
|
http://cffi.readthedocs.org/en/latest/ . It has been very successful
|
|
so far: http://pypi-ranking.info/alltime records almost 7 million
|
|
downloads (for comparison, the #1 of all packages has almost 36
|
|
million downloads). CFFI works on any Python >= 2.6, including 3.x,
|
|
as well as on PyPy.
|
|
|
|
One problem is that while getting started with CFFI is very easy, the
|
|
installation process of a package that uses CFFI has got its rough
|
|
edges. CFFI (at least in its "verify()" mode) is based on calling the
|
|
C compiler to get information about the exact C types, structures,
|
|
argument types to functions, and so on. The C compiler is invoked
|
|
transparently at run-time, and the results cached. A
|
|
correctly-installed package using CFFI should cache the results at
|
|
installation time, but it can be difficult to ensure that no more
|
|
run-time compiler invocation is needed; doing so requires following
|
|
some extra guidelines or understanding some internal details. (The
|
|
problem is particularly acute on Windows where a typical user might
|
|
not have a proper C compiler installed.)
|
|
|
|
To fix this, we have in mind adding a different CFFI mode (replacing
|
|
"verify()"), while keeping the access to the underlying C library
|
|
unmodified. In this mode, the code containing the cdef() and verify()
|
|
invocations would be moved to a separate Python source file. Running
|
|
that Python file would produce a dynamically-linked library. There
|
|
would be no caching logic involved; you would need to run it
|
|
explicitly during development whenever you made changes to it, to
|
|
re-generate and re-compile the dynamically-linked library.
|
|
|
|
When distributed, the same file would be run (once) during
|
|
installation. This can be fully automated in setuptools-based
|
|
setup.py files; alternatively, it can be done in distutils-based
|
|
setup.py files by requiring prior manual installation of CFFI itself.
|
|
|
|
A major difference with the existing verify() approach would be that
|
|
the ``.so/.dll/.dylib`` file would not be immediately loaded into the
|
|
process; you would load it only from the installed program at
|
|
run-time, and get the ``ffi`` and ``lib`` objects in this way (these
|
|
are the two objects that you use so far to access a C library with
|
|
verify()).
|
|
|
|
Additionally, this would solve another issue: every import of a large
|
|
CFFI-using package takes a while so far. This is caused by CFFI
|
|
needing to parse again the C source code given in the cdef() (adding a
|
|
run-time dependency to the ``pycparser`` and ``ply`` packages). CFFI
|
|
also computes a CRC to know if it can reuse its cache. In the
|
|
proposed change, all the cdef() code would be pre-parsed and stored in
|
|
the dynamically-linked library, and no CRC would be needed. This
|
|
would massively reduce the import times.
|
|
|
|
|
|
Grant objective
|
|
---------------
|
|
|
|
The objective is to give a boost towards "CFFI 1.0", which needs to have
|
|
the functionalities described above in order to solve the two main
|
|
issues with the current CFFI: the difficulties of installation, and the
|
|
time taken at import.
|
|
|
|
Included in the objective: the internal refactorings of CFFI that are
|
|
needed to get it done cleanly. The goal is to avoid simply adding
|
|
another layer on top of the old unchanged CFFI.
|
|
|
|
This work may happen eventually in any case, but support from the PSF
|
|
would help make it happen sooner rather than later.
|
|
|
|
|
|
Grant size
|
|
----------
|
|
|
|
2'500 US$ for supporting the development time. This would cover 2.5
|
|
weeks of full-time work at the part-time cost of 25 US$ per hour.
|
|
|
|
The estimated work time until the CFFI 1.0 release is a bit larger
|
|
than that (I estimate it at roughly 4 weeks), but 2.5 weeks should
|
|
cover all the basics. An extended grant size of 4'000 US$ would be
|
|
appreciated but not required ``:-)``
|
|
|
|
|
|
Grant beneficiaries
|
|
-------------------
|
|
|
|
Armin Rigo, main author of CFFI, committing 2.5 weeks of full-time
|
|
work.
|
|
|
|
|
|
Grant follow-up
|
|
---------------
|
|
|
|
I will report on the success of the grant on the CFFI mailing list and
|
|
on the blog I usually post to (the PyPy blog) and mention the PSF as
|
|
providing the grant. The PSF will receive an email pointing to these
|
|
postings once they are out. Moreover a full CFFI 1.0 release should
|
|
follow (likely starting with beta versions); the PSF will receive
|
|
another email pointing to it.
|