bz2 module fails when building Python 3.7

Issue

I'm trying to cross compile Python 3.7 for Android. I see in my output that bz2 if failing with the following error

building '_bz2' extension
/home/dematic/SPE/python3-android/sdk/android-ndk-r19c/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -isystem /home/dematic/SPE/python3-android/build/19c-22-aarch64-linux-androideabi-4.9/include -isystem /home/dematic/SPE/python3-android/build/19c-22-aarch64-linux-androideabi-4.9/include/openssl -no-integrated-as -I. -I./Include -target aarch64-none-linux-androideabi22 -target aarch64-none-linux-androideabi22 -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -fPIC -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I./Include -I. -I/home/dematic/SPE/python3-android/src/Python-3.7.3/Include -I/home/dematic/SPE/python3-android/src/Python-3.7.3 -c /home/dematic/SPE/python3-android/src/Python-3.7.3/Modules/_bz2module.c -o build/temp.linux-aarch64-3.7/home/dematic/SPE/python3-android/src/Python-3.7.3/Modules/_bz2module.o
/home/dematic/SPE/python3-android/sdk/android-ndk-r19c/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -isystem /home/dematic/SPE/python3-android/build/19c-22-aarch64-linux-androideabi-4.9/include -isystem /home/dematic/SPE/python3-android/build/19c-22-aarch64-linux-androideabi-4.9/include/openssl -no-integrated-as -shared -target aarch64-none-linux-androideabi22 -fuse-ld=lld -L /home/dematic/SPE/python3-android/build/19c-22-aarch64-linux-androideabi-4.9/lib -target aarch64-none-linux-androideabi22 -fuse-ld=lld -L /home/dematic/SPE/python3-android/build/19c-22-aarch64-linux-androideabi-4.9/lib -target aarch64-none-linux-androideabi22 -fuse-ld=lld -L /home/dematic/SPE/python3-android/build/19c-22-aarch64-linux-androideabi-4.9/lib -fPIC -target aarch64-none-linux-androideabi22 build/temp.linux-aarch64-3.7/home/dematic/SPE/python3-android/src/Python-3.7.3/Modules/_bz2module.o -L. -L/home/dematic/SPE/python3-android/build/19c-22-aarch64-linux-androideabi-4.9/lib -lbz2 -lpython3.7m -o build/lib.linux-aarch64-3.7/_bz2.cpython-37m.so
ld.lld: error: /home/dematic/SPE/python3-android/build/19c-22-aarch64-linux-androideabi-4.9/lib/libbz2.a(bzlib.o) is incompatible with aarch64linux
clang: error: linker command failed with exit code 1 (use -v to see invocation)

I am building bzip2 1.0.6 without any issues, but I assume I'm not linking to it correctly or some other issue. Is there some sort of other architecture I'm supposed to be building?

This is the project I'm trying to build with https://github.com/GRRedWings/python3-android

Solution

I'm trying to cross compile Python 3.7 for Android. I see in my output that bz2 if failing with the following error

The Bzip2 makefiles are not written for cross-compiles. They effectively ignore a user's flags like CFLAGS and LDFLAGS. The makefiles actually blows away a user's CFLAGS and sets it to CFLAGS=-Wall -Winline -O2 -g $(BIGFILES). Your flags like -target aarch64-none-linux-androideabi22 are not used.

There are two Makefiles in play. One is called Makefile and it builds the static library, if I recall correctly. The second is Makefile-libbz2_so, and it build the shared object. You need to fix the omissions and apply the fixes to both makefiles.

You should probably use a patched Bzip like bzip2-noloader. It honors a user's CFLAGS, CXXFLAGS, LDFLAGS, etc. The check-in of interest is Commit 34d170f31106.

The makefile recipes in bzip2-noloader look similar to the following. They preserve Seward's original settings in BZIP_CFLAGS. But they also utilize CPPFLAGS and allow a user override in CFLAGS. The override will pickup your flags like -target aarch64-none-linux-androideabi22.

blocksort.o: blocksort.c
    $(CC) $(CPPFLAGS) $(BZIP_CFLAGS) $(CFLAGS) -c blocksort.c

Programs use LDFLAGS as expected:

bzip2: libbz2.a bzip2.o
    $(CC) $(CPPFLAGS) $(BZIP_CFLAGS) $(CFLAGS) $(LDFLAGS) -o bzip2 bzip2.o -L. -lbz2

Finally, the bzip2-noloader fork also honor's PREFIX, DESTDIR, etc. So you can perform staged installs, too.

---

I am building bzip2 1.0.6 without any issues ...

You are probably building for i686 or x86_64, and not Aarch64. The problem does not surface until link time. You can use objdump to inspect the object files, if interested.

---

Also note the makefile does this:

CC=gcc
AR=ar
RANLIB=ranlib
LDFLAGS=

You may need to tweak those variable assignments, too. Sometimes ar and ranlib use unusual names, like ranlib-5.0. And also be sure the tools are on-path.

---

The way to write makefiles to avoid these sorts of problems is detailed at 7.2.3 Variables for Specifying Commands in the GNU Coding Standards. The short of it is, (1) leave CFLAGS (and friends) for the user; and (2) if a flag is needed, then always supply it.

The GNU Coding Standards uses this as an example:

CFLAGS = -g
ALL_CFLAGS = -I. $(CFLAGS)
.c.o:
    $(CC) -c $(CPPFLAGS) $(ALL_CFLAGS) $<

Users can override the default CFLAGS of -g, and -I is always added because it is needed for the compile.

Answered By - jww