Why isn't there an assembler for the ESP32 version of uLisp?

#1

@dragoncoder047 wrote:

The ESP32 processors are designed with a “write XOR execute” design that makes on-the-fly machine code generation impossible

More precisely, it’s a decision taken in the design of the IDF (ESP32 IoT Development Framework):

This is caused by the memory protection of ESP-IDF: it assumes all executable code comes from the project binary, and that an attempt to re-write code or to execute data is someone maliciously trying to break into the system, and as such needs to be stopped.

It looks like the only way around it would be to recompile the ESP-IDF libraries; at this point I thought it wasn’t worth pursuing it.

See my full discussion on the ESP32 forum here:

How can I write machine code into RAM and then execute it?

Newbie, comparison to MicroPython
#2

Excellent.

I think I will get the Pico and give it a shot since I need one anyways.

Thanks!

#3

So were you able to find the assembler that ESP32’s use? I thought this wasn’t documented.

It sounds like you could set the partitions.csv file to utilize the maximum amount of flash space for app0, and then read and write from there. There may be some extra incantations necessary to turn off encrypted or readonly flags for that partition to allow writing and executing code.

https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/partition-tables.html

#4

So were you able to find the assembler that ESP32’s use? I thought this wasn’t documented.

For the original ESP32 I think this is the document:

Xtensa Instruction Set Architecture (ISA) Reference Manual

#5

Note that the ESP32-C3, ESP32-C5, and ESP32-C6 boards are based on a RISC-V processor, and if the ESP-IDF didn’t prevent execution of code in RAM the existing uLisp RISC-V assembler would be usable with these.

#6

Sorry, clarification:

The ESP-IDF allows you to put code into RAM at compile time and execute it at run time, but it causes an exception if you try to modify that code at run time.

#7

I asked Github Copilot to look at the ESP-IDF source code and tell me whether it was possible to turn the “invalid instruction fetch” fault into a no-op and allow it to fetch from data RAM, but it said that the W^X is done in hardware so there’s no way to turn it off. The whole IRAM thing is handled by the linker scripts and the ROM bootloader. Once the bootloader returns there’s no longer any way to write to IRAM.

#8

That’s interesting. The only remaining option is for the uLisp assembler to copy the machine code to flash and then execute it there. I wonder if that’s feasible?

#9

No, it said flash was also similarly protected. But even if it weren’t you would have to also deal with the cache, since even on the ESP32-PICO the flash is still technically considered “off-chip” and so the cache would have to be invalidated when you update the page with the assembler code in it, otherwise it would just fetch from the cache and run the old version. I don’t know if there’s any way to actually force a cache flush.