PCIe Cards and bring-up

[Rucadi]

Hello,

I'm making the support for Alveo U280:
https://github.com/rucadi/vivado-risc-v (mostly the board/u280 folder)
but I made some changes that i wanted to discuss, to see if they are pertinent or not.

First, for some reason, I got the impression that the memory range [0x0000_0000-0x8000_0000] was not used.
This is because the mapping starts at 0x0000_0000, the lower addresses are used for internal devices or IO memory, so it was not able to be accessed. am I right?

I got away with modifying the Rocket.vhdl generation to add the following signals:

        signal offset : std_logic_vector(31 downto 0) := x"80000000";
        signal mem_axi4_araddr_buf : std_logic_vector(34 downto 0);
        signal mem_axi4_awaddr_buf : std_logic_vector(34 downto 0);

doing this assignments:

        mem_axi4_araddr <=  std_logic_vector(unsigned(mem_axi4_araddr_buf(34 downto 0)) - unsigned(offset(31 downto 0)));
        mem_axi4_awaddr <=  std_logic_vector(unsigned(mem_axi4_awaddr_buf(34 downto 0)) - unsigned(offset(31 downto 0)));

and mapping accordingly:

        mem_axi4_0_ar_bits_addr              => mem_axi4_araddr_buf,
        mem_axi4_0_aw_bits_addr              => mem_axi4_awaddr_buf,

Also, one thing that I saw is that the Bring-up using JTAG is not ideal, since it's rather slow.

First, I modified rocket-chip (patch on my repo) in order to be able to configure a Maskrom, which contains OpenSBI+Uboot embedded in it. So when the bitstream is loaded, it automatically loads it. (Currently there is a jump hardcoded on the bootrom, I expect to set there conditional compilation)

Since we now have an U-boot console, we can use QDMA Queues in order to copy our linux images or load it via ethernet or any other method, without requiring JTAG at all and easing the use of the system and making it way faster.

While is true that it consumes some FPGA resources, for most cases it's ok in larger FPGA and since it's configurable I expect it to not be a problem.

Currently I'm investigating a problem with the QDMA driver (that is failing at init for something to do with MSI-X interrupts).

What do you think about these changes?
I'm wrong about the unused memory?

Regards,
Ruben.

[eugene-tarassov]

First, for some reason, I got the impression that the memory range [0x0000_0000-0x8000_0000] was not used.
This is because the mapping starts at 0x0000_0000, the lower addresses are used for internal devices or IO memory, so it was not able to be accessed. I'm I right?

Yes, you are right. When RAM size > 2GB, first 2GB are not used. And subtracting 0x80000000 on AXI address buses is a good way to fix it. Actually, my latest version of Rocket.vhdl generator already does that, then adds RAM address (if it is not 0). It is not committed yet. It also has some other improvements. To avoid a merge conflicts, could you hold the generator changes until I commit?

First, I modified rocket-chip (patch on my repo) in order to be able to configure a Maskrom, which contains OpenSBI+Uboot embedded in it.

Another possibility is to use SPI flash memory on the board to store both bitstream and OpenSBI+Uboot. This would require adding SPI flash controller in the design to read the flash, and changes to the BootROM. I think this approach is somewhat better, because it does not need rocket-chip changes, but I'm not sure.

Since we now have an U-boot console, we can use QDMA Queues in order to copy our linux images

Yes, QDMA Queues could be useful, but I would rather use Ethernet: TFTP to load kernel, NFS for root fs. QDMA adds a lot of complexity and a lot of bugs. It would be nice to be able to run everything without it.

I'd prefer a boot sequence like this: FPGA boots from the flash, RISC-V BootROM loads OpenSBI+Uboot from the flash, Uboot loads Linux kernel over TFTP, Linux mounts root fs over NFS.

[Rucadi]

In my case I have a closed loop with 4 alveos U280 that can communicate via the QSFP but not with a Linux server. (And for the moment we don't expect to put a switch)

We temporally have access to the internet via a home-made simple driver that uses QDMA and a shared memory region, but we would prefer it gone.

So having a way to communicate via QDMA or XDMA could be useful, right now I'm creating u-boot fit images and use part of the memory as a ramdisk. I agree It adds complexity.

But I think that making it possible to communicate from the host to the rocket from the PCIe at fast speed it's a nice feature and maybe it's not so complicated to integrate.

I'll look about the possibility of using the flash memory, but I never used that myself,
what i see is that there seems to be 128mb or 1GB of spi nor flash memory, which should be more than enough, and probably we can just jump to the code there from the bootrom.

I didn't make a lot of changes to the rocketchip tho, most of them were in order to configure the maskrom name and location (and maybe I should merge-request that part into the rocketchip repo, since it didn't work at first, so it seems it's a little outdated)

[eugene-tarassov]

But I think that making it possible to communicate from the host to the rocket from the PCIe at fast speed it's a nice feature

I agree, it would be nice.

It also would be nice to be able to boot Linux without PCIe host, or with non-Linux PCIe host, or without XDMA drivers installed, or on a board without PCIe header.

Ideally, both methods needs to be supported.

I'll look about the possibility of using the flash memory, but I never used that myself,

BTW, your u280/Makefile.inc contains line CFG_BOOT ?= -loaddata {up 0x07000000 workspace/boot.elf}, which loads OpenSBI+Uboot into the flash if you run make flash. I started to implement boot from flash, but didn't have time to finish it. It would be great if you can make it to work.

[Rucadi]

BTW, your u280/Makefile.inc contains line CFG_BOOT ?= -loaddata {up 0x07000000 workspace/boot.elf}, which loads OpenSBI+Uboot into the flash if you run make flash. I started to implement boot from flash, but didn't have time to finish it. It would be great if you can make it to work.

I based the AU280 project from the AU250 support and didn't touch that line (because there was nothing to do at the moment)

However, from here I see that:
https://support.xilinx.com/s/article/71756?language=en_US

the address 0x0700_0000 is way above the protected area, and should begin from: 0x0100_2000 for all the alveo cards. Am I wrong? If there is as much space as I think, maybe it's not crazy to use the SPI as a permanent storage, with a simple linux fs and kernel.

If I looked at it correctly, we have from 0x0100_2000 to 0x0800_0000 [edited, if I'm not wrong we have 128MB or 1024Mb]
I'll try to bring-up U-boot this way, but if we manage to fit a Linux Image + fs there, maybe we can ditch U-boot and just use OpenSBI+Linux on alveo cards.

I'll start working on it on Monday, another thing that I made is to have two network interfaces, one for each QSFP.

I didn't test it yet, but, software-wise, the driver you made should work with two interfaces at the same time, right?

I'll inform here about any progress.
Have a nice weekend.