Friday, January 4, 2019

RISC-V on FPGA (the tinyFPGA) via WSL

In my previous blog, I wrote about the problems I encountered with programming the tinyFPGA BX in the Windows Subsystem for Linux (WSL). Now that I have that resolved, onto something much more interesting: installing the RISC-V (specifically the PicoRV32) soft CPU onto the tinyFPGA board!

TL;DR - install libgmp3-dev, libmpfr-dev, libmpc-dev for WSL. Edit Makefile to call explicit RISC-V compiler (only riscv32i worked for me) and name the serial port tinyFPGA is plugged into. Be sure to have plenty of disk space (and patience, or a really fast computer)

Now, this is something well outside of my skill set and comfort zone: A soft CPU on an FPGA! Fortunately, Luke posted this awesome RISC-V example project on TinyFPGA BX on the tinyFPGA discourse forum that seemed to be fairly straightforward. The target environment was a "real" Linux machine, I chose WSL, instead.

Note that as of the date of this blog, there are several useful, but not-yet-merged PR's for the tinyFPGA BX repo that are probably useful to review.

I keep all my projects in a /workspace directory, so my tinyFPGA BX clone looks like this:
gojimmypi@MYHOST : ~
0 $ cd ~/workspace
gojimmypi@MYHOST : ~/workspace
0 $ git clone --recursive
Cloning into 'TinyFPGA-BX'...
remote: Enumerating objects: 108, done.
remote: Total 108 (delta 0), reused 0 (delta 0), pack-reused 108
Receiving objects: 100% (108/108), 648.10 KiB | 250.00 KiB/s, done.
Resolving deltas: 100% (19/19), done.
Checking connectivity... done.
gojimmypi@MYHOST : ~/workspace
See the icestorm install from TinyFPGA-BX icestorm template; I clone all of my GitHub stuff into my ~/workspace directory:
cd ~/workspace/
# tinyFPGA BX
git clone --recursive
cd ~/workspace/

# icestorm
git clone icestorm
cd icestorm
make -j$(nproc)
sudo make install
cd ~/workspace/

# arachne-pnr
git clone arachne-pnr
cd arachne-pnr
make -j$(nproc)
sudo make install
cd ~/workspace/

# yosys
git clone yosys
cd yosys
make -j$(nproc)
sudo make install
cd ~/workspace/

# picorv32
git clone

# see online instructions
Note that I encountered a permission error for the make download-tools :
Checking connectivity... done.
+ mv /var/cache/distfiles/riscv-gcc.git.part /var/cache/distfiles/riscv-gcc.git
mv: cannot move ‘/var/cache/distfiles/riscv-gcc.git.part’ to ‘/var/cache/distfiles/riscv-gcc.git’: Permission denied
make: *** [download-tools] Error 1
So consider using: sudo make download-tools Note that after I encountered the permission error, even the sudo version returned an error (I think the data was not re-created, rather used existing downloads)... so in order to fix it, I needed to manually adjust permissions:
sudo chmod 666 /var/cache/distfiles/riscv-gcc.git.part
sudo mv /var/cache/distfiles/riscv-gcc.git.part /var/cache/distfiles/riscv-gcc.git

However, as I was trying to to this in WSL instead of a "real" Linux environment, I stumbled a bit. First - the RISC-V toolchain did not compile - instead ending in a simple "recipe for 'build-tools' failed". Early on in the process, this error was encountered:
checking for the correct version of gmp.h... no
configure: error: Building GCC requires GMP 4.2+, MPFR 2.4.0+ and MPC 0.8.0+.
Try the --with-gmp, --with-mpfr and/or --with-mpc options to specify
their locations.  Source code for these libraries can be found at
their respective hosting sites as well as at  See also for additional info.  If
you obtained GMP, MPFR and/or MPC from a vendor distribution package,
make sure that you have installed both the libraries and the header
files.  They may be located in separate packages.
checking whether stdint.h conforms to C99... Makefile:415: recipe for target 'stamps/build-gcc-newlib-stage1' failed
make[2]: *** [stamps/build-gcc-newlib-stage1] Error 1
make[2]: *** Waiting for unfinished jobs....

It is unfortunate that the build attempt continued, as it takes an incredibly long time on my machine. Eventually it returned with:
make[2]: Leaving directory '/home/gojimmypi/workspace/picorv32/riscv-gnu-toolchain-riscv32i/build'
Makefile:154: recipe for target 'build-riscv32i-tools-bh' failed
make[1]: *** [build-riscv32i-tools-bh] Error 2
make[1]: Leaving directory '/home/gojimmypi/workspace/picorv32'
Makefile:160: recipe for target 'build-tools' failed
make: *** [build-tools] Error 2
A quick google search and I found this helpful how-to on stackoverflow. Although a bit detailed and verbose, the only information I needed was in the sentence:
For Debian based system including Ubuntu, Install libgmp-dev, libmpfr-dev and libmpc-dev packages.
For my Ubuntu-like WSL this translated to:
sudo apt-get install libgmp3-dev
sudo apt-get install libmpfr-dev 
sudo apt-get install libmpc-dev
(TODO: do I need libgmp3-dev or simply libgmp-dev ?)

After that, the build was successful! However, I was still not able to actually call the compiler:
/usr/bin/install -c -m 644 b-header-vars /opt/riscv32imc/lib/gcc/riscv32-unknown-elf/8.2.0/plugin/include/b-header-vars
make[5]: Leaving directory '/home/gojimmypi/workspace/picorv32/riscv-gnu-toolchain-riscv32imc/build/build-gcc-newlib-stage2/gcc'
make[4]: Leaving directory '/home/gojimmypi/workspace/picorv32/riscv-gnu-toolchain-riscv32imc/build/build-gcc-newlib-stage2'
make[3]: Leaving directory '/home/gojimmypi/workspace/picorv32/riscv-gnu-toolchain-riscv32imc/build/build-gcc-newlib-stage2'
mkdir -p stamps/ && touch stamps/build-gcc-newlib-stage2
make[2]: Leaving directory '/home/gojimmypi/workspace/picorv32/riscv-gnu-toolchain-riscv32imc/build'
make[1]: Leaving directory '/home/gojimmypi/workspace/picorv32'
gojimmypi@MYHOST : ~/workspace/picorv32
0 $ riscv32-unknown-elf-gcc
riscv32-unknown-elf-gcc: command not found
Fortunately this was a simple matter of the command not being in the path. I started a fresh WSL instance to be sure, and indeed it was not added to my path. So I manually edited Luke's Makefile to explicitly name the path. I searched from it in DOS, and found several of them in each of these directories (just as described in the instructions):
 Directory of C:\Users\gojimmypi\AppData\Local\Packages\CanonicalGroupLimited.UbuntuonWindows_79rhkp1fndgsc\LocalState\rootfs\opt

01/03/2019  07:47 PM    <DIR>          riscv32i
01/03/2019  09:08 PM    <DIR>          riscv32ic
01/03/2019  10:26 PM    <DIR>          riscv32im
01/03/2019  11:47 PM    <DIR>          riscv32imc
               0 File(s)              0 bytes
In particular, look at those creation dates! The full monty compile took quite a long time on my system. Together, the 4 of them take up over 3GB of disk space in the /opt directory! I'm not sure how accurate that is - as when going to the parent directory in Windows:
The .\LocalState\rootfs directory is less than 2 GB, despite .\LocalState\rootfs\opt containing over 3GB. The reality is that somehow after this exercise, overall my disk has nearly 20GB less free space. :/

Here's what the rootfs directory looks like in Windows File Explorer (rick-click, properties)

However when running this command (note that I piped it to a file, which is much faster than sending 65MB to the screen):
dir C:\Users\gojimmypi\AppData\Local\Packages\CanonicalGroupLimited.UbuntuonWindows_79rhkp1fndgsc\LocalState\rootfs\ /s > myWSL_rootfs.txt
The resultant file size is considerably larger than reported in Windows Explorer:
     Total Files Listed:
           933087 File(s) 31,192,151,953 bytes
           125544 Dir(s)   5,549,752,320 bytes free
Note my local picorv32 from github ended up here on my Windows file system:
It appears as /home/gojimmypi/workspace/picorv32 in WSL:
gojimmypi@MYHOST : ~/workspace/picorv32
0 $ pwd
gojimmypi@MYHOST : ~/workspace/picorv32
0 $ ls
dhrystone  picorv32.core                      riscv-gnu-toolchain-riscv32im    testbench.v
firmware   picorv32.v     riscv-gnu-toolchain-riscv32i   riscv-gnu-toolchain-riscv32imc    testbench_wb.v
Makefile   picosoc        riscv-gnu-toolchain-riscv32ic  scripts                         testbench_ez.v  tests
gojimmypi@MYHOST : ~/workspace/picorv32
0 $
And sure enough: a whopping 20GB!
gojimmypi@MYHOST : ~/workspace/picorv32
0 $ du -h
4.9G    ./riscv-gnu-toolchain-riscv32imc
20K     ./scripts/csmith
24K     ./scripts/cxxdemo
16K     ./scripts/icestorm
16K     ./scripts/presyn
32K     ./scripts/quartus
28K     ./scripts/romload
56K     ./scripts/smtbmc
12K     ./scripts/tomthumbtg
40K     ./scripts/torture
36K     ./scripts/vivado
8.0K    ./scripts/yosys
12K     ./scripts/yosys-cmp
300K    ./scripts
205K    ./tests
20G     .
One more thing for future reference: I was not starting with a clean WSL. I don't know if anything I had previously installed was also a missing requirement. Using history | grep "sudo apt-get install", I found that I had previously manually installed these items:
sudo apt-get install gdb
sudo apt-get install libdevice-serialport-perl
sudo apt-get install libyaml-perl
sudo apt-get install binutils
sudo apt-get install build-essential clang bison flex libreadline-dev      
sudo apt-get install gawk tcl-dev libffi-dev git mercurial graphviz
sudo apt-get install xdot pkg-config python python3 libftdi-dev
sudo apt-get install libtool
sudo apt-get install automake
sudo apt-get install make unrar-free autoconf automake libtool gcc g++ gperf
sudo apt-get install flex bison texinfo gawk ncurses-dev libexpat-dev python-dev python python-serial
sudo apt-get install sed git unzip bash help2man wget bzip2
sudo apt-get install libtool-bin
sudo apt-get install lzip
sudo apt-get install extract
sudo apt-get install cu
sudo apt-get install setserial
sudo apt-get install gmp
sudo apt-get install libgmp3-dev
sudo apt-get install libmpfr-dev
sudo apt-get install libmpc-dev
Back to calling the compiler: here are the lines I edited in the Makefile to explicitly call the RISC-V compiler I wanted:
firmware.elf: start.S firmware.c 
 /opt/riscv32i/bin/riscv32-unknown-elf-gcc -march=rv32imc -nostartfiles -Wl,-Bstatic,-T,,--strip-debug,,--cref  -ffreestanding -nostdlib -o firmware.elf start.S firmware.c

firmware.bin: firmware.elf
 /opt/riscv32i/bin/riscv32-unknown-elf-objcopy -O binary firmware.elf /dev/stdout > firmware.bin
One of the things I wondered: once the RISC-V processor is created on the FPGA chip... how to I get it to actually run code that I create? Is there a special "soft-JTAG" created? A "soft-UART"? If so, which pins would be used? Perhaps some sort of bootloader? As it turns out... none of those; the solution is quite simple, thanks to some clever work by Luke when implementing his tinyFPGA board. When using the tinyprog command-line tool, in addition to the -p to program the FPGA, there's a -u option to also upload firmware! And yes - this is the firmware code that the soft RISC-V CPU will execute. Brilliant! Ok, perhaps not the most efficient to recreate the entire CPU when the app changes, but it is certainly easy.

So the "app" that gets compiled by the new RISC-V toolchain is the firmware found here. So ok, clearly there are several un-intuitive things going on there. But it is an excellent template to get started and to learn a tremendous amount this RISC-V implementation in Verilog on an FPGA.

As noted in my previous blog, there were some problems in WSL talking to my tinyFPGA board. Specifically, unlike in DOS, the tinyprog tool does not automatically find my tingFPGA board. Thus another minor change was needed to the Makefile. My tinyFPGA board appears as COM15: in Windows... thus /dev/ttyS15 in WSL. So for the upload part, I changed my Makefile to explicitly name that port:
upload: hardware.bin firmware.bin
        tinyprog --com /dev/ttyS15 -p hardware.bin -u firmware.bin
Also: note that once uploaded, the tinyFPGA powers on into "bootloader mode". Pressing reset apparently returns it to boot loader mode as well. That would seem rather unfortunate for any actual in-the-field projects. I wonder if there's some way to not power on like that?

In order to exit bootloader mode, I use this command:
tinyprog -b -c /dev/ttyS15
Note that in order to program the tinyFPGA, we do need to be in bootloader mode, otherwise we'll see an error like this:
0 $ make upload
tinyprog --com /dev/ttyS15 -p hardware.bin -u firmware.bin

    TinyProg CLI
    Using device id 1d50:6130
Traceback (most recent call last):
  File "/home/gojimmypi/.local/bin/tinyprog", line 11, in 
  File "/home/gojimmypi/.local/lib/python2.7/site-packages/tinyprog/", line 325, in main
    with active_port:
  File "/home/gojimmypi/.local/lib/python2.7/site-packages/tinyprog/", line 66, in __enter__
    self.ser = serial.Serial(self.port_name, timeout=1.0, writeTimeout=1.0).__enter__()
  File "/home/gojimmypi/.local/lib/python2.7/site-packages/serial/", line 240, in __init__
  File "/home/gojimmypi/.local/lib/python2.7/site-packages/serial/", line 268, in open
    raise SerialException(msg.errno, "could not open port {}: {}".format(self._port, msg))
serial.serialutil.SerialException: [Errno 5] could not open port /dev/ttyS15: [Errno 5] Input/output error: '/dev/ttyS15'
Makefile:3: recipe for target 'upload' failed
make: *** [upload] Error 1
Also note that I was unable to use the riscv32ic, riscv32ic, or riscv32icm toolchains resulting in errors like this:
  pass 198, 1 sharedm
  pass 199, 1 shared.
  pass 200, 1 shared.
fatal error: failed to route
Makefile:10: recipe for target 'hardware.asc' failed
make: *** [hardware.asc] Error 1
So unless you know you need those other toolchains for other RISC-V stuff, perhaps best to not waste time (and disk space!) compiling them all.

Note that afterwards, I found that even riscv32i was not working! Upon reinstalling the icestorm toolchain, this was resolved - and I was able to use all 4 of the riscv32i[c][m] flavors.

As RISC-V is new to me, so is the suffix information. For reference, the values are found in the The RISC-V Instruction Set Manual, specifically Table 22.1 summarizing the standardized subset names:
RISC-V : Standard ISA Subset Naming Convention 
Indeed once the icestorm and RISC-V toolchains are installed, Luke's (deceptively simple) command does it all:
cd  ~/workspace/TinyFPGA-BX/examples/picosoc
make upload
But the reality - is that a crazy amount of rather complex things are accomplished: Creating a soft CPU?!? On an FGPA!? Using Free and Open Source tools?! Then compiling C code for the newly implemented CPU! From the comfort of my home?! Just incredible. My thanks to everyone that has put in so much effort to make this possible: Particularly Luke Valenty (aka @tinyFPGA) and Clifford Wolf (aka @oe1cxw), along with many others. We are living in amazing times.

Resources, Inspiration, Credits, and Other Links:

Please leave comments, ideas, suggestions below (moderated, sometimes delayed) or send me a message at gmail, or find gojimmypi on twitter.

No comments:

Post a Comment

comments are welcome, but I prefer not to allow links to promotions or other unrelated services.

RISC-V on FPGA (the tinyFPGA) via WSL

In my previous blog , I wrote about the problems I encountered with programming the tinyFPGA BX in the Windows Subsystem for Linux (WSL). ...