GNU Classpath

So you just installed Fedora, RHEL or CentOS and now you want to build GDB from source. How do you make sure everything you need to build it is installed? # dnf builddep gdb Did it say, No such command: builddep? Do this, then try again: # dnf ... [More] install dnf-plugins-core Did it say, dnf: command not found…? You’re using yum, try this: # yum-builddep gdb Did it say, yum-builddep: command not found…? Do this, then try again: # yum install yum-utils Thank you, you’re welcome. [Less]

Python‘s had this handy logging module since July 2003. A lot of things use it, so if you’re trying to understand or debug some Python code then a handy snippet to insert somewhere is: import logging logging.basicConfig(level=1) Those two lines ... [More] cause all loggers to log everything to the console. Check out the logging.basicConfig docs to see what else you could do. [Less]

Andy Wingo wrote some thoughts on rms and gnu. Although I don’t agree with the description of RMS as doing nothing for GNU, the part describing GNU itself is spot on: Software does not, by itself, change the world; it lacks agency. It is the ... [More] people that maintain, grow, adapt, and build the software that are the heart of the GNU project — the maintainers of and contributors to the GNU packages. They are the GNU of whom I speak and of whom I form a part. [Less]

Apparently it’s been a while since I ran containers on my office computer—and by a while, I mean, since November 2016—because if your initial install was RHEL or CentOS 7.2 or older then neither Docker nor Podman will work: # yum -q -y install podman ... [More] skopeo buildah # podman pull registry.access.redhat.com/ubi7/ubi Error: could not get runtime: kernel does not support overlay fs: overlay: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior. Reformat the filesystem with ftype=1 to enable d_type support. Running without d_type is not supported.: driver not supported So… ugh. I didn’t have any disks it’d work on either: # for i in $(awk '{ if ($3 == "xfs") print $2 }' /etc/mtab); do xfs_info $i; done | grep ftype naming =version 2 bsize=4096 ascii-ci=0 ftype=0 naming =version 2 bsize=4096 ascii-ci=0 ftype=0 naming =version 2 bsize=4096 ascii-ci=0 ftype=0 naming =version 2 bsize=4096 ascii-ci=0 ftype=0 I didn’t reformat anything though. podman pull wants overlayFS on /var/run/containers/storage, and buildah bud wants it on /var/lib/containers/storage. I made loopback disks for them both: Find/make space somewhere, then create a directory to put the images in: # mkdir -p /store/containers Create a big file, whatever size you want, for the disk image. I made mine 20GiB. It took a couple minutes, my disks are slow: # dd if=/dev/zero of=/store/containers/var_lib_containers.img bs=1M count=20K Find a free loop device and associate the file to it: # losetup -f /dev/loop1 # losetup /dev/loop1 /store/containers/var_lib_containers.img Format the “device”, then detach it from the file: # mkfs -t xfs -n ftype=1 /dev/loop1 # losetup -d /dev/loop1 Mount the “disk”, and see if it worked: # mount -oloop /store/containers/var_lib_containers.img /var/lib/containers # df -h /var/lib/containers Filesystem Size Used Avail Use% Mounted on /dev/loop1 20G 33M 20G 1% /var/lib/containers It worked? Make it permanent: # echo "/store/containers/var_lib_containers.img /var/lib/containers xfs defaults,loop 1 2" >> /etc/fstab Rinse and repeat for the other drive it needed. Then try again: # podman pull registry.access.redhat.com/ubi7/ubi Trying to pull registry.access.redhat.com/ubi7/ubi...Getting image source signatures Copying blob bff3b73cbcc4 done Copying blob 7b1c937e0f67 done Copying config 6fecccc91c done Writing manifest to image destination Storing signatures 6fecccc91c83e11ae4fede6793e9410841221d4779520c2b9e9fb7f7b3830264 #victorydance [Less]

FSF and GNU the FSF is now working with GNU leadership on a shared understanding of the relationship for the future. Joint statement on the GNU Project The GNU Project we want to build is one that everyone can trust to defend their freedom.

Yesterday I made a Fedora 30 VM on my RHEL 7 box, and for some reason I couldn’t log in as root after the installation finished. Well, it’s been a while, so I had to look it up, and following the instructions didn’t work either—I finally managed to ... [More] get a shell, but the terminal was corrupted. Because it was a VM? Because the instructions were out of date? I’ve no idea. Anyway, here’s what I did, with the stuff that wasn’t in the instructions underlined: Reboot and wait for the GRUB menu to appear. In the menu, highlight any entry and press e to edit it. Find the line beginning with linux. Remove the rhgb and quiet options, then add init=/bin/sh at the end of the line. Press Ctrl-X to boot with those options. After a while you should get a root shell. The prompt was sh-5.0# on my system, not sh-4.2# like the instructions say, but it doesn’t matter. Run the commands in the instructions: /usr/sbin/load_policy -i mount -o remount,rw / passwd root mount -o remount,ro / The instructions say to reboot now, but none of the commands to reboot the system worked at this point. Probably they expected systemd. No problem, I hit “Force Reset” in Virtual Machine Manager. I probably should have run a sync or two beforehand, but I didn’t think to. Ta-da, working system [Less]

I’ve been following the status of Lisp on ppc64le lately. I’m running ppc64le Debian sid. Just after I had set up my system, I did some experimentation with what Debian packages had to offer. ECL was the only Lisp that worked, so I started using it ... [More] for various projects. (I’ve since learned on #sbcl that CLISP built from source is also a good option.) Ideally I wanted to be able to use SBCL, so I wondered how far into an SBCL bootstrap I could get with ECL as the host compiler. A few months ago, I found the answer was not very far. Since then though, the SBCL maintainers have been hard at work on two fronts: making SBCL bootstrap against ECL again (on any architecture) and porting SBCL to PPC64. Recently with a few minor local changes, I was able to bootstrap SBCL natively on my Talos II starting with ECL, then build SBCL again with the bootstrapped SBCL. I had to build ECL from the tip of its development branch, but it has a nice build system and, being based on C, doesn’t require any bootstrapping steps. So the SBCL ppc64 port is really shaping up; hopefully the next release will advertise ppc64le support. For now, plenty of packages work for me already, like SILME and Quicklisp. I’d also like to try the same bootstrap procedure on ppc64be ABI version 1 (Debian sid) and ppc64be ABI version 2 (another distro, probably Adélie). I’m working on setting up an environment with qemu-system-ppc64. Anyway, I’m happy that I can now use SBCL on my Talos II. [Less]

elfutils 0.177 was released with various bug fixes (if you ever had issues updating > 2GB ELF files using libelf, this release is for you!) and some new features. One of the features is eu-elfclassify, a utility by Florian Weimer to analyze ELF ... [More] objects. People use various tricks to construct ELF files that might make it non-trivial to determine what kind of ELF file you might be dealing with. Even a simple question like “is this a program executable or shared library?” might be tricky given the fact that (static) PIE executables look a lot like shared libraries. And some “shared libraries” are also “program executables”. e.g. Qt likes to provide some information about how the files have been build. So you can link against it as a shared library, but you can also execute it as if it was a program: $ /usr/lib/x86_64-linux-gnu/libQt5Core.so.5 This is the QtCore library version Qt 5.11.3 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 8.3.0)Installation prefix: /usr Library path: lib/x86_64-linux-gnu Include path: include/x86_64-linux-gnu/qt5 Processor features: sse3 sse2[required] ssse3 fma cmpxchg16b sse4.1 sse4.2 movbe popcnt aes avx f16c rdrand bmi avx2 bmi2 rdseed glibc does the same thing for its shared libraries. Which is nice if you just quickly need to know what libc version is installed on a system, but might make it tricky to determine what kind of ELF file something really is. eu-classify has a mode that will tell you whether such a file is primarily a shared library or primarily a program executable. And of course is able to classify it as both a library and a program. Hopefully eu-classify can replace the usage of the file (1) utility in various tools, with a more precise way to classify ELF files. Usage: elfclassify [OPTION...] FILE... Determine the type of an ELF file. All of the classification options must apply at the same time to a particular file. Classification options can be negated using a "--not-" prefix. Since modern ELF does not clearly distinguish between programs and dynamic shared objects, you should normally use either --executable or --shared to identify the primary purpose of a file. Only one of the --shared and --executable checks can pass for a file. If you want to know whether an ELF object might a program or a shared library (but could be both), then use --program or --library. Some ELF files will classify as both a program and a library. If you just want to know whether an ELF file is loadable (as program or library) use --loadable. Note that files that only contain (separate) debug information (--debug-only) are never --loadable (even though they might contain program headers). Linux kernel modules are also not --loadable (in the normal sense). Without any of the --print options, the program exits with status 0 if the requested checks pass for all input files, with 1 if a check fails for any file, and 2 if there is an environmental issue (such as a file read error or a memory allocation error). When printing file names, the program exits with status 0 even if no file names are printed, and exits with status 2 if there is an environmental issue. On usage error (e.g. a bad option was given), the program exits with a status code larger than 2. The --quiet or -q option suppresses some error warning output, but doesn't change the exit status. Classification options --core File is an ELF core dump file --debug-only File is a debug only ELF file (separate .debug, .dwo or dwz multi-file) --elf File looks like an ELF object or archive/static library (default) --elf-archive File is an ELF archive or static library --elf-file File is an regular ELF object (not an archive/static library) --executable File is (primarily) an ELF program executable (not primarily a DSO) --library File is an ELF shared object (DSO) (might also be an executable) --linux-kernel-module File is a linux kernel module --loadable File is a loadable ELF object (program or shared object) --program File is an ELF program executable (might also be a DSO) --shared File is (primarily) an ELF shared object (DSO) (not primarily an executable) --unstripped File is an ELF file with symbol table or .debug_* sections and can be stripped further Input flags -f, --file Only classify regular (not symlink nor special device) files --no-stdin Do not read files from standard input (default) --stdin Also read file names to process from standard input, separated by newlines --stdin0 Also read file names to process from standard input, separated by ASCII NUL bytes -z, --compressed Try to open compressed files or embedded (kernel) ELF images Output flags --matching If printing file names, print matching files (default) --no-print Do not output file names --not-matching If printing file names, print files that do not match --print Output names of files, separated by newline --print0 Output names of files, separated by ASCII NUL Additional flags -q, --quiet Suppress some error output (counterpart to --verbose) -v, --verbose Output additional information (can be specified multiple times) -?, --help Give this help list --usage Give a short usage message -V, --version Print program version Report bugs to https://sourceware.org/bugzilla. [Less]

Compiling with the GCC sanitizers and then fuzzing the resulting binaries might find real bugs. But not all such bugs are security issues. When a CVE is filed there is some pressure to treat such an issue with urgency and push out a fix as soon as ... [More] possible. But taking your time and making sure an issue can be replicated/exploited without the binary being instrumented by the sanitizer is often better. This was the case for CVE-2019-12900 “BZ2_decompress in decompress.c in bzip2 through 1.0.6 has an out-of-bounds write when there are many selectors“. The bzip2 project had lost the domain which it had used for the last 15 years. And it hadn’t seen an official release since 2010. The bzip2 project homepage, documentation and downloads had already been moved back to sourceware.org. And a new bug tracker, development mailinglist and git repository had been setup. But we were still in the middle of a code cleanup (removing references to the old homepage, updating the manual and adding various cleanups that distros had made to the code) when the CVE was filed. The issue reported was discovered by a fuzzer ran against a bzip2 binary compiled with gcc -fsanitizer=undefined. Which produced the following error: decompress.c:299:10: runtime error: index 18002 out of bounds for type 'UChar [18002]' The DState struct given to the BZ2_decompress function has a field defined as UChar selectorMtf[BZ_MAX_SELECTORS]; where BZ_MAX_SELECTORS is 18002. So the patch that came with the security report looked totally reasonable. --- a/decompress.c +++ b/decompress.c @@ -284,15 +284,15 @@ Int32 BZ2_decompress ( DState* s ) 284 /*--- Now the selectors ---*/ 285 GET_BITS(BZ_X_SELECTOR_1, nGroups, 3); 286 if (nGroups < 2 || nGroups > 6) RETURN(BZ_DATA_ERROR); 287 GET_BITS(BZ_X_SELECTOR_2, nSelectors, 15); 288 - if (nSelectors < 1) RETURN(BZ_DATA_ERROR); + if (nSelectors < 1 || nSelectors > BZ_MAX_SELECTORS) RETURN(BZ_DATA_ERROR); 289 for (i = 0; i < nSelectors; i++) { 290 j = 0; 293 while (True) { 294 GET_BIT(BZ_X_SELECTOR_3, uc); 295 if (uc == 0) break; 296 j++; 297 if (j >= nGroups) RETURN(BZ_DATA_ERROR); 298 } 299 s->selectorMtf[i] = j; /* array overrun! */ 300 } Without the new nSelectors > BZ_MAX_SELECTORS guard the code could write beyond the selectorMtf array, which is undefined behavior. The undefined behavior in this case would be writing to memory addresses after the array. Given that an attacker could define nSelectors as big as they want, they would be able to override any memory after the array. This seemed urgent enough to do a new release quickly with this fix. bzip2 1.0.7 was released. But the next day we already got bug reports that the fix broke decompression of some existing .bz2 files. This didn’t really make sense at first. BZ_MAX_SELECTORS was the theoretical maximum number of selectors that could validly be used in a .bz2 file. But some testing did confirm that these files did define a handful more selectors than were actually used. It turned out that some alternative bzip2 implementations used a slightly bigger maximum for the number of selectors (rounded up to a factor 8) which they might define, but didn’t expect to be used. Julian Seward came up with a fix that split the max number of selectors in two. The original theoretical max that bzip2 would encode, and a bigger (rounded up to a factor 8) max that would be accepted when decompressing. This seemed to fix the issue for real, while still accepting some slightly “wrong” .bz2 files. The original code had worked for these because the array overwrite was only a few bytes, and the DState struct has extra state right after the selectorMtf array. The UChar len[BZ_N_GROUPS][BZ_MAX_ALPHA_SIZE] array (6 * 258 = 6192 bytes), which was only written to after the selectors were read. So the memory overwrite was almost immediately corrected and didn’t do any harm because it was just such a small amount. The new code would still protect against real “too big” nSelector values. But we still didn’t feel completely confident we had fixed things correctly. One issue was that bzip2 never had a really good testsuite. Testing was mostly done ad-hoc by developers on a random collection of .bz2 files that they happened to have around. Luckily some alternative bzip2 implementations had created more formal testsuites. The .bz2 testfiles of those projects were collected and a testframe was created that ran bzip2 on both correct and known bad .bz2 files (optionally using valgrind to catch bad memory usage). This was a really good thing. The testsuite was added to the bzip2 buildbot. Which immediately flagged one testcase (32767.bz2) as BAD! The 32767.bz2 testcase has the max number of selectors that the file format allows (2^15 - 1 = 32676). The .bz2 file format reserves 15 bits for the number of selectors used in a block. This is because to express the max of 18002 selectors can only be expressed when using 15 bits. That testcase could be decompressed correctly by bzip2 1.0.6 (or earlier), but not by the new bzip2 version that checked the number of selectors was “sane“. When the original bzip2 1.0.6 code was compiled with gcc -fsanitize=undefined the selectorMtf array overwrite was (correctly) reported. But surprisingly when ran under valgrind memcheck no bad memory usage was reported. Some more investigation revealed that although this was an example of the most extreme possible selectorMtf array overwrite, it still only wrote over already allocated memory and that memory was not used before being assigned correct values. The selectorMtf array could hold 18002 bytes. 32767 – 18002 = 14765 bytes that could be overwritten after the array. But the DState struct had 3 more arrays after the selectorMtf and len arrays. Each defined as UInt32 [BZ_N_GROUPS][BZ_MAX_ALPHA_SIZE], which is 3 * 4 * 6 * 258 = 18576 bytes. And all state after the selectorMtf array in the DState struct would be assigned values right after reading the selectors. And none of the excess selector values would ever be used. So even though there really was an array overwrite, it was completely harmless! That knowledge allowed us to write a much simpler patch that just skipped over the extra selectors without storing them. And release bzip2 1.0.8 that decompressed all the same files that 1.0.6 and earlier could. In the end it was good for the bzip2 project to have a bit of an emergency. It brought people together who cared deeply about making sure bzip2 survives as a project, it got us automated release scripts, a new testsuite, buildbots, various other fixes upstreamed from distros and bzip2 is now part of oss fuzz (so we might get earlier warnings about similar issues in the future) and there is now a kind of roadmap for how to move forward But part of the panic was also completely unnecessary. Yes, there was a way to trigger undefined behavior, but with any current compiler that behavior was actually defined, it would write over known (bounded) memory, memory that otherwise was correctly used and defined. We should have insisted on having a real reproducer, that could be triggered under valgrind memcheck. The instrumentation of the undefined sanitizer was not enough to show a real issue. We were lucky, it could certainly have been, or become, a real issue if the DState structure layout would have been different, if some constants were larger or smaller or if the compiler was smarter (it could have decided that writing after the array could never happen and so “optimize” the program assuming some loops were bounded). So fixing the bug was certainly the right thing to do. But in practice it never was a real security issue and we placed too much value in the fact that a CVE was assigned to it. [Less]

Compiling with the GCC sanitizers and then fuzzing the resulting binaries might find real bugs. But not all such bugs are security issues. When a CVE is filed there is some pressure to treat such an issue with urgency and push out a fix as soon as ... [More] possible. But taking your time and making sure an issue can be replicated/exploited without the binary being instrumented by the sanitizer is often better. This was the case for CVE-2019-12900 “BZ2_decompress in decompress.c in bzip2 through 1.0.6 has an out-of-bounds write when there are many selectors“. The bzip2 project had lost the domain which it had used for the last 15 years. And it hadn’t seen an official release since 2010. The bzip2 project homepage, documentation and downloads had already been moved back to sourceware.org. And a new bug tracker, development mailinglist and git repository had been setup. But we were still in the middle of a code cleanup (removing references to the old homepage, updating the manual and adding various cleanups that distros had made to the code) when the CVE was filed. The issue reported was discovered by a fuzzer ran against a bzip2 binary compiled with gcc -fsanitizer=undefined. Which produced the following error: decompress.c:299:10: runtime error: index 18002 out of bounds for type 'UChar [18002]' The DState struct given to the BZ2_decompress function has a field defined as UChar selectorMtf[BZ_MAX_SELECTORS]; where BZ_MAX_SELECTORS is 18002. So the patch that came with the security report looked totally reasonable. --- a/decompress.c +++ b/decompress.c @@ -284,15 +284,15 @@ Int32 BZ2_decompress ( DState* s ) 284 /*--- Now the selectors ---*/ 285 GET_BITS(BZ_X_SELECTOR_1, nGroups, 3); 286 if (nGroups < 2 || nGroups > 6) RETURN(BZ_DATA_ERROR); 287 GET_BITS(BZ_X_SELECTOR_2, nSelectors, 15); 288 - if (nSelectors < 1) RETURN(BZ_DATA_ERROR); + if (nSelectors < 1 || nSelectors > BZ_MAX_SELECTORS) RETURN(BZ_DATA_ERROR); 289 for (i = 0; i < nSelectors; i++) { 290 j = 0; 293 while (True) { 294 GET_BIT(BZ_X_SELECTOR_3, uc); 295 if (uc == 0) break; 296 j++; 297 if (j >= nGroups) RETURN(BZ_DATA_ERROR); 298 } 299 s->selectorMtf[i] = j; /* array overrun! */ 300 } Without the new nSelectors > BZ_MAX_SELECTORS guard the code could write beyond the selectorMtf array, which is undefined behavior. The undefined behavior in this case would be writing to memory addresses after the array. Given that an attacker could define nSelectors as big as they want, they would be able to override any memory after the array. This seemed urgent enough to do a new release quickly with this fix. bzip2 1.0.7 was released. But the next day we already got bug reports that the fix broke decompression of some existing .bz2 files. This didn’t really make sense at first. BZ_MAX_SELECTORS was the theoretical maximum number of selectors that could validly be used in a .bz2 file. But some testing did confirm that these files did define a handful more selectors than were actually used. It turned out that some alternative bzip2 implementations used a slightly bigger maximum for the number of selectors (rounded up to a factor 8) which they might define, but didn’t expect to be used. Julian Seward came up with a fix that split the max number of selectors in two. The original theoretical max that bzip2 would encode, and a bigger (rounded up to a factor 8) max that would be accepted when decompressing. This seemed to fix the issue for real, while still accepting some slightly “wrong” .bz2 files. The original code had worked for these because the array overwrite was only a few bytes, and the DState struct has extra state right after the selectorMtf array. The UChar len[BZ_N_GROUPS][BZ_MAX_ALPHA_SIZE] array (6 * 258 = 6192 bytes), which was only written to after the selectors were read. So the memory overwrite was almost immediately corrected and didn’t do any harm because it was just such a small amount. The new code would still protect against real “too big” nSelector values. But we still didn’t feel completely confident we had fixed things correctly. One issue was that bzip2 never had a really good testsuite. Testing was mostly done ad-hoc by developers on a random collection of .bz2 files that they happened to have around. Luckily some alternative bzip2 implementations had created more formal testsuites. The .bz2 testfiles of those projects were collected and a testframe was created that ran bzip2 on both correct and known bad .bz2 files (optionally using valgrind to catch bad memory usage). This was a really good thing. The testsuite was added to the bzip2 buildbot. Which immediately flagged one testcase (32767.bz2) as BAD! The 32767.bz2 testcase has the max number of selectors that the file format allows (2^15 - 1 = 32767). The .bz2 file format reserves 15 bits for the number of selectors used in a block. This is because to express the max of 18002 selectors can only be expressed when using 15 bits. That testcase could be decompressed correctly by bzip2 1.0.6 (or earlier), but not by the new bzip2 version that checked the number of selectors was “sane“. When the original bzip2 1.0.6 code was compiled with gcc -fsanitize=undefined the selectorMtf array overwrite was (correctly) reported. But surprisingly when ran under valgrind memcheck no bad memory usage was reported. Some more investigation revealed that although this was an example of the most extreme possible selectorMtf array overwrite, it still only wrote over already allocated memory and that memory was not used before being assigned correct values. The selectorMtf array could hold 18002 bytes. 32767 – 18002 = 14765 bytes that could be overwritten after the array. But the DState struct had 3 more arrays after the selectorMtf and len arrays. Each defined as UInt32 [BZ_N_GROUPS][BZ_MAX_ALPHA_SIZE], which is 3 * 4 * 6 * 258 = 18576 bytes. And all state after the selectorMtf array in the DState struct would be assigned values right after reading the selectors. And none of the excess selector values would ever be used. So even though there really was an array overwrite, it was completely harmless! That knowledge allowed us to write a much simpler patch that just skipped over the extra selectors without storing them. And release bzip2 1.0.8 that decompressed all the same files that 1.0.6 and earlier could. In the end it was good for the bzip2 project to have a bit of an emergency. It brought people together who cared deeply about making sure bzip2 survives as a project, it got us automated release scripts, a new testsuite, buildbots, various other fixes upstreamed from distros and bzip2 is now part of oss fuzz (so we might get earlier warnings about similar issues in the future) and there is now a kind of roadmap for how to move forward But part of the panic was also completely unnecessary. Yes, there was a way to trigger undefined behavior, but with any current compiler that behavior was actually defined, it would write over known (bounded) memory, memory that otherwise was correctly used and defined. We should have insisted on having a real reproducer, that could be triggered under valgrind memcheck. The instrumentation of the undefined sanitizer was not enough to show a real issue. We were lucky, it could certainly have been, or become, a real issue if the DState structure layout would have been different, if some constants were larger or smaller or if the compiler was smarter (it could have decided that writing after the array could never happen and so “optimize” the program assuming some loops were bounded). So fixing the bug was certainly the right thing to do. But in practice it never was a real security issue and we placed too much value in the fact that a CVE was assigned to it. [Less]