Link-Time-Optimization

[maxgerhardt]

Playing around with a software PWM implementation a bit, I noticed that this piece of code

#include "ez80f92.h"
#define PWM_PORT_DR      IO(PC_DR)
#define MAX_PWM_CHANNELS   8
#define PWM_TIMER_FREQ_HZ  50
#define PWM_RESOLUTION     256

static uint8_t pwm_flat[PWM_RESOLUTION * MAX_PWM_CHANNELS];
static volatile uint8_t inv_pwm_mask = 0xff;
static uint8_t pwm_ctr;

__attribute__((interrupt))
void PRT1_Handler(void)
{
    IO(TMR1_CTL); // clear interrupt flag

    // OR 8 channels
    uint8_t bits =
        pwm_flat[pwm_ctr * MAX_PWM_CHANNELS + 0] |
        pwm_flat[pwm_ctr * MAX_PWM_CHANNELS + 1] |
        pwm_flat[pwm_ctr * MAX_PWM_CHANNELS + 2] |
        pwm_flat[pwm_ctr * MAX_PWM_CHANNELS + 3] |
        pwm_flat[pwm_ctr * MAX_PWM_CHANNELS + 4] |
        pwm_flat[pwm_ctr * MAX_PWM_CHANNELS + 5] |
        pwm_flat[pwm_ctr * MAX_PWM_CHANNELS + 6] |
        pwm_flat[pwm_ctr * MAX_PWM_CHANNELS + 7];

    // write port
    PWM_PORT_DR = (PWM_PORT_DR & inv_pwm_mask) | (bits);

    // advance pointer and counter
    pwm_ctr++;
}

Was translated to a piece of code where the multiplication by 8 (to get the base pointer essentially) was done by a call to the __ishl (integer shift left) function, even though the multiplicant "8" (or shift amount: 3) is statically known

000004cc <__Z12PRT1_Handlerv>:
 4cc:	d9                	exx
 4cd:	08                	ex af,af'
 4ce:	fd e5             	push iy
 4d0:	fd 21 c4 00 04    	ld iy,0x400c4
 4d5:	ed 38 83          	in0 a,(0x83)
 4d8:	3a c4 08 04       	ld a,(0x408c4)
 4dc:	b7                	or a,a
 4dd:	ed 62             	sbc hl,hl
 4df:	6f                	ld l,a
 4e0:	0e 03             	ld c,0x03
 4e2:	cd 19 08 00       	call 0x0819
 4e6:	e5                	push hl
 4e7:	d1                	pop de
 4e8:	fd 19             	add iy,de
 4ea:	fd 7e 01          	ld a,(iy+1)
 4ed:	fd b6 00          	or a,(iy+0)
 4f0:	fd b6 02          	or a,(iy+2)
 4f3:	fd b6 03          	or a,(iy+3)
..

00000819 <__ishl>:
 819:	cb 41             	bit 0,c
 81b:	28 01             	jr z,0x081e
 81d:	29                	add hl,hl

0000081e <.bit0>:
 81e:	cb 49             	bit 1,c
 820:	28 02             	jr z,0x0824
 822:	29                	add hl,hl
 823:	29                	add hl,hl

00000824 <.bit1>:
 824:	cb 51             	bit 2,c
 826:	28 04             	jr z,0x082c
 828:	29                	add hl,hl
 829:	29                	add hl,hl
 82a:	29                	add hl,hl
 82b:	29                	add hl,hl

0000082c <.bit2>:
 82c:	cb 59             	bit 3,c
 82e:	20 06             	jr nz,0x0836
 830:	cb 61             	bit 4,c
 832:	c8                	ret z

00000833 <__ishl_16>:
 833:	65                	ld h,l
 834:	2e 00             	ld l,0x00

00000836 <__ishl_8>:
 836:	29                	add hl,hl
 837:	29                	add hl,hl
 838:	29                	add hl,hl
 839:	29                	add hl,hl
 83a:	29                	add hl,hl
 83b:	29                	add hl,hl
 83c:	29                	add hl,hl
 83d:	29                	add hl,hl
 83e:	c9                	ret

This was a bit sad to see as I would have loved it to either use the builtin SLA r or an inline of the 8 ADD instructions or just one MULT instruction.

I think the cause of this is that it cannot perform link time optimization of the generated code because LTO is disabled during compilation and linking staging (maybe also "whole program optimization"?).

While it is easy to generate LTO enabled objects with clang through -flto, the GNU linker (ez80-none-elf-ld) then of course doesn't know what to do with these objects and fails.

ez80-none-elf-ld -o .pio\build\agonlight2\firmware.elf -T C:\Users\Max\.platformio\packages\framework-arduinoez80\variants\agonlight2\ldscript.ld --no-warn-rwx-segments -nostdlib --gc-sections -defsym=EXTRAM_START=0x40000 -defsym=EXTRAM_SIZE=0x80000 -defsym=INTRAM_START=0xb7e000 -defsym=INTRAM_SIZE=0x2000 -defsym=FLASH_SIZE=0x20000 .pio\build\agonlight2\src\main.cpp.o -L.pio\build\agonlight2 .pio\build\agonlight2\libFrameworkArduinoVariant.a .pio\build\agonlight2\libFrameworkArduino.a
.pio\build\agonlight2\src\main.cpp.o: file not recognized: file format not recognized

I don't understand the linker well enough but there are plugin capabilities so maybe I'm just not building or passing the right plugin for it to do LTO. Or, I need to use a different linker all together to produce the ELF file, like the llvm linker tools (?).

[maxgerhardt]

Huh, on Linux there are a few interesting files generated in

llvm-build/ez80-none-elf/lib/libLTO.so
llvm-build/ez80-none-elf/bfd-plugins/libdep.so

(which is not copied to the release distribution folder), maybe that's the right plug for it. But trying to load libLTO.so as a linker plugin into ez80-none-elf-ld just fails, so it doesn't seem to be compatible..

$ ~/agon/agondev-src/llvm-build/ez80-none-elf/bin/ez80-none-elf-ld -o firmware.elf -plugin ~/agon/agondev-src/llvm-build/ez80-none-elf/lib/bfd-plugins/libdep.so -plugin ~/agon/agondev-src/llvm-build/ez80-none-elf/lib/libLTO.so  -T ldscript.ld -v --no-warn-rwx-segments -nostdlib --gc-sect
ions -defsym=EXTRAM_START=0x40000 -defsym=EXTRAM_SIZE=0x80000 -defsym=INTRAM_START=0xb7e000 -defsym=INTRAM_S
IZE=0x2000 -defsym=FLASH_SIZE=0x20000 .pio/build/agonlight2/src/main.cpp.o -L.pio/build/agonlight2 .pio/buil
d/agonlight2/libFrameworkArduinoVariant.a .pio/build/agonlight2/libFrameworkArduino.a
GNU ld (GNU Binutils) 2.45
/home/max/agon/agondev-src/llvm-build/ez80-none-elf/bin/ez80-none-elf-ld: /home/max/agon/agondev-src/llvm-build/ez80-none-elf/lib/libLTO.so: error loading plugin: /home/max/agon/agondev-src/llvm-build/ez80-none-elf/lib/libLTO.so: undefined symbol: _onload

On the other hand, I read in llvm/llvm-project#64216 that if Clang is provided with the path of the GNU binutils beforehand, then it will produce llvmgold.so which might work as a linker plugin?

[maxgerhardt]

I managed to build the LLVMgold.so plugin on Linux with a simple modification in make_llvmtools.sh (adding -DLLVM_BINUTILS_INCDIR=$BINUTILSDIR/include in the LLVM build config), but even with that plugin the linker doesn't understand the -flto objects generated by Clang

max@DESKTOP-65I2GH4:~/agon/test-link$ ~/agon/agondev-src/llvm-build/ez80-none-elf/bin/ez80-none-elf-ld -o firmware.elf  -plugin ~/agon/agondev-src/llvm-build/ez80-none-elf/lib/LLVMgold.so
 -T ldscript.ld -v --no-warn-rwx-segments -nostdlib --gc-sections -defsym=EXTRAM_START=0x40000 -defsym=EXTRAM_SIZE=0x80000 -defsym=INTRAM_START=0xb7e000 -defsym=INTRAM_SIZE=0x2000 -defsym=
FLASH_SIZE=0x20000 .pio/build/agonlight2/src/main.cpp.o -L.pio/build/agonlight2 .pio/build/agonlight2/libFrameworkArduinoVariant.a .pio/build/agonlight2/libFrameworkArduino.a
GNU ld (GNU Binutils) 2.45
/home/max/agon/agondev-src/llvm-build/ez80-none-elf/bin/ez80-none-elf-ld: .pio/build/agonlight2/src/main.cpp.o (symbol from plugin): unsupported bfd mach 0
LLVM ERROR: Failed to setup codegen
Aborted (core dumped)

Maybe this is the completely wrong path and I need to build the native lld linker or compile to LLVM bitcode and do a manual opt, llvm-lto, llvm-link, llvm-llc or whatever. to optimize it on the bitcode level and then convert it to object files / ELF.

Edit: Mhm, it seems that as soon as -flto is turned on in clang, it stops producing object files and does

$ file main.cpp.o
main.cpp.o: LLVM IR bitcode

this instead, which LD cannot read even with the LLVMgold plugin (?), was previously

main.cpp.o: ELF 32-bit LSB relocatable, *unknown arch 0xdc* version 1 (SYSV), not stripped

Edit 2: I managed to build lld in the llvm-project, but it just doesn't understand the IR code that -flto generates.

$ ~/agon/agondev-src/llvm-build/ez80-none-elf/bin/lld-link  -flavor gnu -o firmware.elf -T ldscript.ld .pio/build/agonlight2/src/main.cpp.o -L.pio/build
/agonlight2 .pio/build/agonlight2/libFrameworkArduinoVariant.a .pio/build/agonlight2/libFrameworkArduino.a
lld-link: error: .pio/build/agonlight2/src/main.cpp.o: could not infer e_machine from bitcode target triple ez80-none-unknown-elf
lld-link: error: variant.c.o: could not infer e_machine from bitcode target triple ez80-none-unknown-elf
..

[envenomator]

I have never played with the IR code that LLVM produces, nor with the IR linker that is built (llvm-link). The AgDev toolchain uses the LLVM to produce assembly that is assembled and 'linked' using the FASMG. I replaced the latter steps with the GNU assembler/linker, but they obviously don't understand IR, only ELF32Z80 formatted objects.
So you might try to experiment and see if any additional IR optimizations can be done, optimized/linked as IR objects using the llvm-linker (built as part of the toolchain), and only then convert the resulting IR-linked object to assembly sources, assemble them using GNU as and link them further to additional existing assembly sources/libs.

[envenomator]

Your disassembly shows the function '__Z12PRT1_Handlerv', while the source says 'void PRT1_Handler(void)'

[maxgerhardt]

The function name is only mangled because I wrote the code in a .cpp file, so that's normal.

As for he IR, I'm not yet sure whether the IR bitcode outputted by clang (when -flto is used) is valid and can be reconverted into the assembly / object files.. Need to experiment.