QEMU: From Soup to Nuts
- 6 days ago
- 9 min read
This blog covers learning QEMU Internals from basic principles by writing the CH32V003 board for QEMU (RISC-V microcontroller). This board is affordable at 10 cents and is similar to lower-end Atmel boards. cnlohr’s ch32fun is a great repository of example firmware and a Hardware Abstraction Library (HAL). This post uses the uartdemo example, so we need to implement all the “bring up” components at minimum:
Reset Configuration Controller (RCC)
Universal Synchronous/Asynchronous Receiver/Transmitter (USART) for debugging
Core Private Peripherals (used for printf)
Follow the repository’s instructions on how to compile the examples. The Executable and Linkable Format (ELF) files output by the build are the emulation target.
General Development Methodology
To add a new board to an emulator, you need a few elements:
A new set of machine files that register in QEMU as another selectable machine.
A method to load the Read-Only Memory (ROM) into the proper section of memory (provided by QEMU).
Instrumentation to see current Program Counter (PC), memory reads, and writes (implemented in this blog post).
Data sheet and reference manual for the board with any additional peripherals you want to test.
A binary analysis program (Ghidra, Binary Ninja, IDA) that can read the Instruction Set Architecture (ISA) of the firmware.
Functionality to get to a bare “bring up” state: implementing a timer and USART (or Universal Asynchronous Receiver-Transmitter (UART)) to help debug any encountered issues.
Implementation of necessary peripherals to test the firmware and features of interest. In this case, the USART to start for Part 1 and the Inter-Integrated Circuit (I2C) bus for Part 2. (Coming soon!)
Here are some tips:
Find and get to know the QEMU monitor. When debugging a stuck instruction, seeing the system state is vital. Press CTRL-A to bring up the monitor.
The -S command arg (argument) pauses the machine at start up, which is good for setting breakpoints.
info registers dumps the register state. Lots of functions will loop because two registers are not equal, so static analysis only provides half the story.
Use the -s arg for QEMU to stand up a GNU Debugger (GDB) server on localhost:1234; note, the longhand -gdb arg is useful for a different binding address.
With the machine frozen from the above argument, set breakpoints with b 0xADDR
You can skip around in the binary with set $pc=0xADDR
Use a text editor that has a full text search function, like VSCode. This allows tracing where functions are called, a technique that made this blog post possible.
Just for fun! If you have worked with Qiling, there are similar options provided there as well. Use the debug verbosity with the verbose=QL_VERBOSE.DEBUG flag provided to the class. For debugging, set the .debugger attribute to one of the documented options.
Adding a New Board
Find the proper folder for your board. For a RISC-V board, it would go under hw/riscv/ and include/hw/riscv. Add file(s) to the folder, and then do the following in meson.build. Copy the .add line, and add a new name for the board:
riscv_ss.add(when: 'CONFIG_CH32', if_true: files('ch32v003.c'))Then in Kconfig, add something similar but without the CONFIG_ prefix:
config CH32
bool
default y
depends on RISCV32Note: This is the bare minimum, and you may have more dependencies or helper files.
QEMU is massive, so only compile the target you are working on for development. To start building, make a build/ directory with mkdir build and go into it (cd build). Then run ../configure --target-list=riscv32-softmmu.
Make a small helper file that will recompile and re-execute QEMU on every run. This will be slow on the first cycle, but the build system is smart enough to detect the changes and only rebuild those files. Name it “qemu.sh” to make it easy to type. Replace uartdemo.elf with the ELF you want to test. Make sure to copy the ELF files into the build/ directory as well. We’ll explain the arguments further in.
make -j24
./qemu-system-riscv32 -d in_asm,int,unimp -D qemu-debug.log -M ch32v003 -nographic -bios none -kernel uartdemo.elfTo actually start the machine, hit c (continue) from the monitor.
Loading a Firmware Image
RISC-V for regular machines uses the Open Supervisor Binary Interface (OpenSBI) for the bootloader. This file is referenced in hw/riscv/boot.c and its respective header. However, a bare-metal system typically boots directly to address 0x0. As such, we specify -bios none which makes QEMU not use the opensbi-riscv32-generic-fw_dynamic.bin (this is present in the pc-bios/ directory).
The -kernel option will load an ELF file to the appropriate parts in the emulated flash. Note that this shortcut does not work in x86, as the directives to pass in the image are a bit messy. For a more general (but verbose) way to load in a bare metal image, use the Generic Loader with -device loader,file=.
Writing Your New Board
It’s always best to have a starting template when making a new board. Check if there is a similar one in the hw/ directory under the ISA of interest. If there isn’t, use the virt board, which should be the most generic. For the CH32V003, as it’s a microcontroller, we’ll implement USART to get text output from the firmware.
Copy over the virt.c and virt.h (from hw/riscv/ and include/hw/riscv, respectively) to the ch32v003 equivalent. There are three functions that will conflict with the compilation process, so remove those from both files:
virt_is_iommu_sys_enabled
imsic_num_bits
virt_is_acpi_enabled
Appearing in the Machine List
There are two items to modify to appear in the machine list ( -M help). First, the name on the line mc->desc = "RISC-V board compatible with CH32V003"; that shows up under the BOARD_machine_class_init function. This changes the definition that appears in the listing.
The second, changing the TypeInfo .name field, which is the actual name that appears in the listing and in the command line argument passed to -M.
With this, we'll be able to appear in the machine listing!
~/qemu/build$ ./qemu-system-riscv32 -M help
Supported machines are:
amd-microblaze-v-generic AMD Microblaze-V generic platform
ch32v003 RISC-V board compatible with CH32V003
none empty machine
opentitan RISC-V Board compatible with OpenTitan
sifive_e RISC-V Board compatible with SiFive E SDK
sifive_u RISC-V Board compatible with SiFive U SDK
spike RISC-V Spike board (default)
virt RISC-V VirtIO boardStripping out Unnecessary Parts
This part takes time, but we’ll provide a patch file that should work with the commit baa79455fa92984ff0f4b9ae94bed66823177a27 on the QEMU repository. A good rule of thumb is try to run the image in the copied machine and start to look at what doesn't work or doesn't fit with your image. The virt machine has full machine peripherals, which aren't needed:
Advanced Configuration and Power Interface (ACPI)
Peripheral Component Interconnect Express (PCIe)
Input-Output Memory Management Unit (IOMMU)
Flattened Device Tree (FDT)
Add a lot of log statements to figure out what is hit and what you should expect. With the little helper script to constantly recompile, you’ll quickly figure out the “load bearing” code for your board.
Also, you want to start renaming things (which is probably best done with find and replace) from virt to ch32v003. Below is a patch file of the basic changes that will get you to the point where the firmware files will load to the proper addresses in QEMU.
When we get to this spot, while we don't see anything happen, we can do a sanity check. Run the file, and see what loads to 0x0. The uartdemo.elf will show:
~/qemu/build$ ./qemu.sh
[1/18] Generating qemu-version.h with a custom command (wrapped by meson to capture output)
SUCCESS! We loaded the firmware!
QEMU 10.1.50 monitor - type 'help' for more information
(qemu) x 0x0
00000000: 0x2a20006f…and, if we open the uartdemo.elf in Ghidra and jump to 0x0, we can see the same bytes (in big endian order).
Instrumentation
QEMU provides various ways to instrument the binary. The simplest being the qemu_printf() function, which is equivalent to the POSIX printf() you know and love. Another one that is generally overlooked but is much more useful for emulator development is the qemu_log function, which has the same signature but logs to a file (so you don't lose the important info you're looking for).
To see the debugging, you must supply the -d flag along with the types of logs you are looking for, my list is in_asm,int,unimp. Then, with the -D flag, you specify the file you want to output to. The file will show the opcodes and addresses it's executing, which shows where you’re getting “snagged.”
Unimplemented Device
One helper function you should definitely start with is the unimp device, which is defined in hw/misc/unimp.h (made with a call to create_unimplemented_device("name", base_addr, size)). This makes every call to this memory region log to the QEMU debug log, which quickly shows you what needs implementing.
Booting
When executing the ELF loading patch, there will appear to be nothing at first. Debugging is enabled, so let's check out our qemu-debug.log file and see the last instructions before it repeats with a bunch of errors.
IN: handle_reset
0x000002fa: e000f7b7 lui a5,-131057
0x000002fe: 4705 addi a4,zero,1
0x00000300: c398 sw a4,0(a5)
0x00000302: 61a00793 addi a5,zero,1562
0x00000306: 34179073 csrrw zero,mepc,a5
----------------
IN: handle_reset
0x00000300: c398 sw a4,0(a5)
riscv_cpu_do_interrupt: hart:0, async:0, cause:00000007, epc:0x00000300, tval:0xe000f000, desc=fault_store
CSR_MTVEC: reserved mode not supportedAnd when we look at it in Ghidra, we get a lot more context:
It’s attempting to write to 0xe000f000, which, when looking at the CH32V003 Reference Manual main memory map, is a part of the Core Private Peripherals:
Doing a text search for that address, you can find it's under the Programmable Fast Interrupt Controller (PFIC), specifically regarding the System Counter:
All that to say the memory region needs to be mapped (probably as Memory Mapped Input/Output (MMIO)). Taking a look at the QEMU Memory Application-Programming Interface (API), we can see there is a call for memory_region_init_io(). One of the parameters is a MemoryRegionOps, which is a struct that holds callback functions for when that section of memory is hit. Let's also map the peripherals for the next steps as well.
static void corepriv_write (void *opaque, hwaddr addr,
uint64_t val, unsigned size)
{
qemu_log("corepriv w: %lx, val %lx\n", addr, val);
}
static uint64_t corepriv_read (void *opaque, hwaddr addr,
unsigned size)
{
switch(addr){
default:
qemu_log("corepriv r: %lx, size %d\n", addr, size);
}
return 0;
}
static const MemoryRegionOps corepriv_ops = {
.read = corepriv_read,
.write = corepriv_write,
.endianness = DEVICE_NATIVE_ENDIAN,
};
static void peripherals_write (void *opaque, hwaddr addr,
uint64_t val, unsigned size)
{
switch(addr){
default:
qemu_log("peripherals w: %lx, val %lx\n", addr, val);
}
}
static uint64_t peripherals_read (void *opaque, hwaddr addr,
unsigned size)
{
switch(addr){
default:
qemu_log("peripherals r: %lx, size %d\n", addr, size);
}
return 0;
}
static const MemoryRegionOps peripherals_ops = {
.read = peripherals_read,
.write = peripherals_write,
.endianness = DEVICE_NATIVE_ENDIAN,
};
static void ch32v003_machine_init(MachineState *machine)
{
MemoryRegion *corepriv = g_new(MemoryRegion, 2);
MemoryRegion *peripherals = g_new(MemoryRegion, 3);
...
virt_flash_map(s, system_memory);
memory_region_init_io(corepriv, NULL, &corepriv_ops, NULL, "corepriv", s->memmap[CH32V003_COREPRIV].size);
memory_region_add_subregion(system_memory, s->memmap[CH32V003_COREPRIV].base, corepriv);
memory_region_init_io(peripherals, NULL, &peripherals_ops, NULL, "peripherals", s->memmap[CH32V003_PERIPHERALS].size);
memory_region_add_subregion(system_memory, s->memmap[CH32V003_PERIPHERALS].base, peripherals);
...
}When we look at our debug log, we can see we get a lot further, PROGRESS! It now gets stuck at 0x5b2, where it’s trying to read a peripheral, repeatedly. This is a sign that it’s a status bit that needs setting before it can proceed.
IN: SystemInit
0x000005ac: 431c lw a5,0(a4)
0x000005ae: 00679693 slli a3,a5,6
0x000005b2: fe06dde3 bgez a3,-6 # 0x5ac
peripherals r: 21000, size 4
[repeated to infinity]Looking at the reference manual for the offset from the peripheral base (0x40000000 + 0x21000 = 0x4002100), it points to the R32_RCC_CTLR, aka the clock control register. It’s branching only if the value is greater than or equal to zero (bgez), comparing it to the a3 register and then 6 bytes backward relative in the program, if it is. The slli instruction takes the value read from that peripheral and shifts it to the left by 6 bits. Let’s see what is 6 bits from index 31 (which would be bit 25).
This checks out, and it’s waiting for the reset configuration controller phased-locked loop to be ready for system timer and watchdog operations. It’s an essential part of the chip to be able to reset itself. Next, set bit 25 to 1, which is 0x2000000. Here’s how the read callback should look:
static uint64_t peripherals_read (void *opaque, hwaddr addr,
unsigned size)
{
switch(addr){
case 0x21000:
return 0x2000000; // RCC_CTLR PLLRDY
break;
default:
qemu_log("peripherals r: %lx, size %d\n", addr, size);
}
return 0;
}Armed with this technique, you’re now off to the races against the other status bits that need to be set for bootup. To make things easier, here is a table of expected values:
There is another core peripheral, the system clock, which must count up. So define a global variable that increments and returns it on every read.
UART
Before we run, we need to walk. Let's implement the UART peripheral. It’s a couple more lines of code and gives us much more insight into the black box of the emulator.
Another way to output text is via the Single Wire Debugging Interface, which maps to ch32fun's printf() calls. It’s a little different to implement, but put it’s the upper byte written to the 0xf4 offset of the core private peripherals address (0xE000000). This is useful for the Part 2 firmware, which exclusively outputs over this method.
Now, the debug log is hanging on the R32_USART_STATR, which shows we are exactly where we want to be. Let’s look at what the firmware wants for the status:
----------------
IN: _write.constprop.0
0x0000027a: 04067613 andi a2,a2,64
0x0000027e: de6d beqz a2,-6 # 0x278
peripherals r: 13800, size 4
[repeated to infinity]This one is simple, as it expects 64 (0x40) from the andi instruction. Add that to the callback switch statement to enable the Transmission Complete (TC) bit, which is the send completion status.
Now, we want to see the actual output (handled under offset 0x13804, which is the data register). The lowest byte of this register is both read and write, so we must print the characters we read from it in the write callback:
case 0x13804:
qemu_printf("%c", (char)val);
break;Now, when executing the uartdemo.elf:
~/qemu/build$ ./qemu.sh
[1/19] Generating qemu-version.h with a custom command (wrapped by meson to capture output)
[2/3] Compiling C object libqemu-riscv32-softmmu.a.p/hw_riscv_ch32v003.c.o
[3/3] Linking target qemu-system-riscv32
SUCCESS! We loaded the firmware!
QEMU 10.1.50 monitor - type 'help' for more information
(qemu) Count: 0
Count: 1
Count: 2
Count: 3
Count: 4
Count: 5…and that’s a wrap! We can see the program is fully looping with no pausing or getting stuck, and there is output. Stay tuned for Part 2 where we implement the I2C controller to display to a screen.
Below is the final patch file which shows all the changes needed to get to the UART working.
