Merge pull request #27 from frenzymadness/backports_from_CPython

Backports from CPython + project refresh
fedora-python · Mar 21, 2024 · 371007a · 371007a
2 parents a2bd55e + d958a74
commit 371007a
Show file tree

Hide file tree

Showing 9 changed files with 181 additions and 168 deletions.
diff --git a/.appveyor.yml b/.appveyor.yml
@@ -1,10 +1,13 @@
 environment:
   matrix:
-  - TOXENV: "py34"
-  - TOXENV: "py35"
   - TOXENV: "py36"
   - TOXENV: "py37"
   - TOXENV: "py38"
+  - TOXENV: "py39"
+  - TOXENV: "py310"
+  - TOXENV: "py311"
+  - TOXENV: "py312"
+  - TOXENV: "py313"
 
 install:
   - C:\Python38\python -m pip install tox

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -0,0 +1,27 @@
+on:
+  push:
+    branches:
+    - "*"
+  pull_request:
+    branches:
+    - "*"
+
+name: Run Tox tests
+
+jobs:
+  tox_test:
+    name: Tox test
+    steps:
+    - uses: actions/checkout@v4
+    - name: Run Tox tests
+      id: test
+      uses: fedora-python/tox-github-action@main
+      with:
+        tox_env: ${{ matrix.tox_env }}
+        dnf_install: "python3-test python3.*-test pypy3-test"
+    strategy:
+      matrix:
+        tox_env: [py36, py37, py38, py39, py310, py311, py312, py313, pypy3]
+
+    # Use GitHub's Linux Docker host
+    runs-on: ubuntu-latest
diff --git a/.travis.yml b/.travis.yml
diff --git a/Dockerfile b/Dockerfile
diff --git a/README.md b/README.md
@@ -2,7 +2,10 @@
 
 Copy of `compileall` module from CPython source code with some new features, namely:
 
-* compatibility with Python >= 3.4 & PyPy 3
+* compatibility with Python >= 3.6 & PyPy 3
+
+The following features were first implemented in this project and then included
+into the standard libraty of CPython.
 
 * default recursion limit is now "unlimited" (actually limited by `sys.getrecursionlimit()`)
 
@@ -21,7 +24,7 @@ Copy of `compileall` module from CPython source code with some new features, nam
 
 * From [PyPI](https://pypi.org/project/compileall2/) via `pip install compileall2`
 
-* RPMs will be available in [Fedora COPR](https://copr.fedorainfracloud.org/coprs/lbalhar/compileall2/)
+* In Fedora Linux, compileall2.py is a part of python-srpm-macros RPM package.
 
 ## Usage
 
@@ -61,46 +64,6 @@ Traceback (most recent call last):
 ZeroDivisionError: division by zero
 ```
 
-## Done
-
-* ✓ Start project :)
-
-* ✓ Make upstream tests running
-
-* Make `compileall2` compatible with CPythons:
-
-  * 3.8 ✓
-  * 3.7 ✓
-  * 3.6 ✓
-  * 3.5 ✓
-  * 3.4 ✓
-
-* ✓ Make `compileall2` compatible with PyPy 3
-
-* ✓ Remove maximum depth limit as described above
-
-* ✓ Add possibility to strip some part of a path to an original file from compiled one
-
-* ✓ Publish it to PyPI
-
-* ✓ Make it available in Fedora COPR
-
-* ✓ Test it with Python packages in COPR
-
-* Push it to Fedora rawhide
-
-  * ✓ %py_byte_compile RPM macro uses `compileall2` (done in [python-rpm-macros](https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/25))
-  * ✓ switch brp-python-bytecompile RPM script to `compileall2` (done in [redhat-rpm-config](https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/64#))
-
-* ✓ Test it in Fedora infrastructure with all Python packages (done in the mass rebuild for Fedora 31)
-
-* ✓ Prepare patches for upstream CPython
-    * [Pull request](https://github.com/python/cpython/pull/16012) merged and will be part of Python 3.9
-
-* ✓ Changes from upstream CPython backported back
-
-* ✓ Implemented important features for Fedora RPM build system
-
 ## Testing
 
 You can test it locally with tox or unittest directly:
@@ -113,45 +76,7 @@ Ran 107 tests in 3.714s
 OK (skipped=12)
 ```
 
-but running in a Docker container might be better because the superuser has privileges to write to `sys.path` which lowers the number of skipped tests.
-
-You can just build the prepared one:
-
-```shell
-$ docker build -t compileall2 .
-Sending build context to Docker daemon 177.2 kB
-Step 1/3 : FROM frenzymadness/fedora-python-tox:latest
- ---> 00f92ad0e1d3
-... etc ...
-```
-
-and run tests in it:
-
-```shell
-$ docker run --rm -it -e TOXENV=py37 -v $PWD:/src:Z -w /src  compileall2
-py37 installed: atomicwrites==1.3.0,attrs==19.3.0,compileall2==0.5.0,coverage==4.5.4,importlib-metadata==0.23,more-itertools==7.2.0,packaging==19.2,pluggy==0.13.0,py==1.8.0,pyparsing==2.4.5,pytest==5.2.3,six==1.13.0,wcwidth==0.1.7,zipp==0.6.0
-py37 run-test-pre: PYTHONHASHSEED='1615314833'
-py37 runtests: commands[0] | coverage run --append -m py.test
-==================================== test session starts =====================================
-platform linux -- Python 3.7.5, pytest-5.2.3, py-1.8.0, pluggy-0.13.0
-cachedir: .tox/py37/.pytest_cache
-rootdir: /src
-collected 107 items
-test_compileall2.py ............ss..................................................ss [ 61%]
-..............................ss.........                                              [100%]
-
-=============================== 101 passed, 6 skipped in 7.40s ===============================
-py37 runtests: commands[1] | coverage report -i '--omit=.tox/*'
-Name                  Stmts   Miss  Cover
------------------------------------------
-compileall2.py          232     48    79%
-test_compileall2.py     621      8    99%
------------------------------------------
-TOTAL                   853     56    93%
-__________________________________________ summary ___________________________________________
-  py37: commands succeeded
-  congratulations :)
-```
+but running in a container might be better because the superuser has privileges to write to `sys.path` which lowers the number of skipped tests.
 
 ## License
 

diff --git a/compileall2.py b/compileall2.py
@@ -4,7 +4,7 @@
 given as arguments recursively; the -l option prevents it from
 recursing into directories.
 
-Without arguments, if compiles all modules on sys.path, without
+Without arguments, it compiles all modules on sys.path, without
 recursing into subdirectories.  (Even though it should do so for
 packages -- for now, you'll have to deal with packages separately.)
 
@@ -36,11 +36,11 @@
 # introduced in Python 3.7. These cases are covered by variables here or by PY37
 # variable itself.
 if PY37:
-    pyc_struct_format = '<4sll'
+    pyc_struct_format = '<4sLL'
     pyc_header_lenght = 12
     pyc_header_format = (pyc_struct_format, importlib.util.MAGIC_NUMBER, 0)
 else:
-    pyc_struct_format = '<4sl'
+    pyc_struct_format = '<4sL'
     pyc_header_lenght = 8
     pyc_header_format = (pyc_struct_format, importlib.util.MAGIC_NUMBER)
 
@@ -106,7 +106,7 @@ def compile_dir(dir, maxlevels=None, ddir=None, force=False,
     workers:   maximum number of parallel workers
     invalidation_mode: how the up-to-dateness of the pyc will be checked
     stripdir:  part of path to left-strip from source file path
-    prependdir: path to prepend to beggining of original file path, applied
+    prependdir: path to prepend to beginning of original file path, applied
                after stripdir
     limit_sl_dest: ignore symlinks if they are pointing outside of
                    the defined path
@@ -120,23 +120,34 @@ def compile_dir(dir, maxlevels=None, ddir=None, force=False,
         stripdir = dir
         prependdir = ddir
         ddir = None
-    if workers is not None:
-        if workers < 0:
-            raise ValueError('workers must be greater or equal to 0')
-        elif workers != 1:
-            try:
-                # Only import when needed, as low resource platforms may
-                # fail to import it
-                from concurrent.futures import ProcessPoolExecutor
-            except ImportError:
-                workers = 1
+    if workers < 0:
+        raise ValueError('workers must be greater or equal to 0')
+    if workers != 1:
+        # Check if this is a system where ProcessPoolExecutor can function.
+        from concurrent.futures.process import _check_system_limits
+        try:
+            _check_system_limits()
+        except NotImplementedError:
+            workers = 1
+        else:
+            from concurrent.futures import ProcessPoolExecutor
     if maxlevels is None:
         maxlevels = sys.getrecursionlimit()
     files = _walk_dir(dir, quiet=quiet, maxlevels=maxlevels)
     success = True
-    if workers is not None and workers != 1 and ProcessPoolExecutor is not None:
+    if workers != 1 and ProcessPoolExecutor is not None:
+        mp_context_arg = {}
+        if PY37:
+            import multiprocessing
+            if multiprocessing.get_start_method() == 'fork':
+                mp_context = multiprocessing.get_context('forkserver')
+            else:
+                mp_context = None
+            mp_context_arg = {"mp_context": mp_context}
+        # If workers == 0, let ProcessPoolExecutor choose
         workers = workers or None
-        with ProcessPoolExecutor(max_workers=workers) as executor:
+        with ProcessPoolExecutor(max_workers=workers,
+                                 **mp_context_arg) as executor:
             results = executor.map(partial(compile_file,
                                            ddir=ddir, force=force,
                                            rx=rx, quiet=quiet,
@@ -178,7 +189,7 @@ def compile_file(fullname, ddir=None, force=False, rx=None, quiet=0,
                files each with one optimization level.
     invalidation_mode: how the up-to-dateness of the pyc will be checked
     stripdir:  part of path to left-strip from source file path
-    prependdir: path to prepend to beggining of original file path, applied
+    prependdir: path to prepend to beginning of original file path, applied
                after stripdir
     limit_sl_dest: ignore symlinks if they are pointing outside of
                    the defined path.
@@ -190,10 +201,8 @@ def compile_file(fullname, ddir=None, force=False, rx=None, quiet=0,
                           "in combination with stripdir or prependdir"))
 
     success = True
-    if PY36 and quiet < 2 and isinstance(fullname, os.PathLike):
-        fullname = os.fspath(fullname)
-    else:
-        fullname = str(fullname)
+    fullname = os.fspath(fullname)
+    stripdir = os.fspath(stripdir) if stripdir is not None else None
     name = os.path.basename(fullname)
 
     dfile = None
@@ -206,13 +215,13 @@ def compile_file(fullname, ddir=None, force=False, rx=None, quiet=0,
     if stripdir is not None:
         fullname_parts = fullname.split(os.path.sep)
         stripdir_parts = stripdir.split(os.path.sep)
-        ddir_parts = list(fullname_parts)
 
-        for spart, opart in zip(stripdir_parts, fullname_parts):
-            if spart == opart:
-                ddir_parts.remove(spart)
-
-        dfile = os.path.join(*ddir_parts)
+        if stripdir_parts != fullname_parts[:len(stripdir_parts)]:
+            if quiet < 2:
+                print("The stripdir path {!r} is not a valid prefix for "
+                      "source path {!r}; ignoring".format(stripdir, fullname))
+        else:
+            dfile = os.path.join(*fullname_parts[len(stripdir_parts):])
 
     if prependdir is not None:
         if dfile is None:
@@ -258,7 +267,7 @@ def compile_file(fullname, ddir=None, force=False, rx=None, quiet=0,
             if not force:
                 try:
                     mtime = int(os.stat(fullname).st_mtime)
-                    expect = struct.pack(*(pyc_header_format + (mtime,)))
+                    expect = struct.pack(*(pyc_header_format + (mtime & 0xFFFF_FFFF,)))
                     for cfile in opt_cfiles.values():
                         with open(cfile, 'rb') as chandle:
                             actual = chandle.read(pyc_header_lenght)
@@ -301,9 +310,8 @@ def compile_file(fullname, ddir=None, force=False, rx=None, quiet=0,
                 else:
                     print('*** ', end='')
                 # escape non-printable characters in msg
-                msg = err.msg.encode(sys.stdout.encoding,
-                                     errors='backslashreplace')
-                msg = msg.decode(sys.stdout.encoding)
+                encoding = sys.stdout.encoding or sys.getdefaultencoding()
+                msg = err.msg.encode(encoding, errors='backslashreplace').decode(encoding)
                 print(msg)
             except (SyntaxError, UnicodeError, OSError) as e:
                 success = False
@@ -408,8 +416,8 @@ def main():
                         type=int, help='Run compileall concurrently')
     parser.add_argument('-o', action='append', type=int, dest='opt_levels',
                         help=('Optimization levels to run compilation with. '
-                              'Default is -1 which uses optimization level of '
-                              'Python interpreter itself (specified by -O).'))
+                              'Default is -1 which uses the optimization level '
+                              'of the Python interpreter itself (see -O).'))
     parser.add_argument('-e', metavar='DIR', dest='limit_sl_dest',
                         help='Ignore symlinks pointing outsite of the DIR')
     parser.add_argument('--hardlink-dupes', action='store_true',
@@ -456,17 +464,15 @@ def main():
     # if flist is provided then load it
     if args.flist:
         try:
-            with (sys.stdin if args.flist=='-' else open(args.flist)) as f:
+            with (sys.stdin if args.flist=='-' else
+                    open(args.flist, encoding="utf-8")) as f:
                 for line in f:
                     compile_dests.append(line.strip())
         except OSError:
             if args.quiet < 2:
                 print("Error reading file list {}".format(args.flist))
             return False
 
-    if args.workers is not None:
-        args.workers = args.workers or None
-
     if PY37 and args.invalidation_mode:
         ivl_mode = args.invalidation_mode.replace('-', '_').upper()
         invalidation_mode = py_compile.PycInvalidationMode[ivl_mode]

diff --git a/setup.cfg b/setup.cfg
@@ -13,12 +13,14 @@ classifiers =
     Development Status :: 4 - Beta
     Intended Audience :: Developers
     Programming Language :: Python :: 3
-    Programming Language :: Python :: 3.5
     Programming Language :: Python :: 3.6
     Programming Language :: Python :: 3.7
     Programming Language :: Python :: 3.8
     Programming Language :: Python :: 3.9
     Programming Language :: Python :: 3.10
+    Programming Language :: Python :: 3.11
+    Programming Language :: Python :: 3.12
+    Programming Language :: Python :: 3.13
     Programming Language :: Python :: Implementation :: CPython
     Programming Language :: Python :: Implementation :: PyPy
     License :: OSI Approved :: Python Software Foundation License