Ulisp-2.4-esp with 4 MByte PSRAM support


#1

Hi,
I added PSRAM support to ulisp-2.4-esp for the esp-32-WROVER modules (should work with other modules too, but I didn’t test it). It seems to work, even if it is not fully tested now.

This version incorporates a few patches to newsymbol (included in version 2.1 (?) to improve the use of Workspace-cells on systems with limited RAM) and compact-image (used by save-image) (see LARGE_WORKSPACE_SETUP symbol in the sources). Without these the extra memory made some operations of uLisp rather slow, because it was designed to run on systems with very limited RAM.

You can download this version of ulisp at https://github.com/Kaef/ulisp-esp .

Feedback is very welcomed.

Kindly regards,
Kaef


#2

Dear @Kaef, dear @johnsondavies,

I confirm that I could upload this to a ESP32-WROVER.

Seeing over 500K cells free sort of feels absolutely brilliant. - But what’s up with that alleged corruption of save-image? Is there any “safe” way to operate this variant, perhaps over some sort of string output? - And what is the matter with the ESP 8266 boards, why are they supposedly having 80k RAM but 3k cons cells … shouldn’t it be more in the range of 8K, given that the Due gets 10K?

Thank you for extending uLisp to new boards, things are getting really interesting, and I know of no other dialect that scales nearly 200x just like that! :)


#3

OK, what about this:

Would it be possible to print all definitions to sd-card like with pprintall, just with a stream object?

It then should be possible to read them back in by evaluating the string from the sd-card. Can pprintall be made to accept a stream?


#4

Hi Aeneas,
great that this version works on your system.
I’m working on adding sleep-modes to ulisp (sleep modes on ESPs behaves a bit different than on most other systems: after wakeup the system restart from beginning on ESPxx, while other systems continue to work on the next command after deepsleeping. Therefore the sleep-command from ulisp have a unexpected behaviour for most developers on ESP systems. Be aware that this is a ESP system behaviour and can not be change from ulisp.).

After that I want to expand the save-image to store images on sd-card if a filename is given to the command.

I did not do much tests with ulisp-2.4 on esp8266 systems, but I think the wifi-stack takes a lot of the RAM memory.

Do you have a dedicated reason to use esp8266 over esp32? (well, I have some esp8266 boards too and maybe I’ll use them with ulisp, but I think esp32 is much more interesting)

I think you could save the pprintall-output to a file on sd-card (I’m a beginner in programming lisp, but I think David could show how to do this). I think pprintall removes the name of the defuns (replace the name with ‘lambda’), so the name must be added to be able to read the saved file back. Again, I think this should be possible to solve with ulisp statements…

Kindly regards,
Kaef


#5

Currently pprintall doesn’t accept a stream, but pprint does, so you could use the function described in this thread to pprint all your definitions to a stream:


#6

@Kaef can confirm this, but as I understand it there’s no corruption; it’s just that the standard ESP version of uLisp is very slow doing a (save-image) when using all the memory in the ESP32-WROVER module.


#7

Thank you very much, @johnsondavies !

I discovered this version has some issue with the garbage collection (or at least the way this is printed) - note the diminishing number of cells:


#8

Yes, that’s not right. I haven’t tested your version with PSRAM support. Could you try some additional tests:

  • The standard 2.4 version with the code to address PSRAM, but without your speedup improvements.

  • Your version on a board without PSRAM.

Thanks, David


#9

corruption of save-image

I’m not sure in this point.
I think I had corrupted saved images, but I wasn’t able to reproduce this behaviour.
I didn’t look at this because I want to make it possible to save the images on sd-card. I’m not sure if this solves this problem, but I think it may have something to do with the eeprom-functions for esp32 (which uses the flash).

My version of save-image (better: compactimage () and subfunctions) should be a lot faster than the original version if not much lispcells are in use. On a heavy used system compactimage() takes a lot of time (several minutes) (I added the output of ‘.’ so you can see that the system is working).


#10

If you recompile uLisp after making changes, and then try to load in a saved image, it’s not guaranteed to work because the saved image isn’t position independent, and the address of the workspace may have changed. Could this explain what you observed?


#11

diminishing number of cells

This is because of my patch to newsymbol () and repl (). If you call (gc) in between all cells are marked as free (this was done in repl (), but the system-reponse-time increases a lot). The (gc) is automatically called if less than 20 lispcells are free, so don’t worry ;-).

If you didn’t like this behaviour you can comment out the beginning of newsymbol () (in my version lines 361 to 363 and line 369) or you can comment out the definition LARGE_WORKSPACE (line 200), which will also removes my patch to repl (calling gc) (of course you can remove my patch to repl () without changing newsymbol () or vice versa).
The downside of doing this is that the interface gets slower without that patches.

I’m think this will be very hard to solve completely (ie. the system is still fast if many lisp-cells used) – one way could be to use small ‘chunks’ for the workspace and add a new chunk if there’s too less memory free. But if you would use many lisp-cells (large programs) ulisp would get slower the larger the workspace is, so I think this solution is not good enough (and it would be complicated to realize).
Therefore I think we have to gc the chunks separately and only if anything in it changed since last call to gc, but (again) to realize this is rather complex (and it has very low priority on my personal TODO-list…)

I think the best solution is to live with it and (if you want a predictive system-runtime) call (gc) when you have time for housekeeping in your program.

Kindly regards,
Kaef


#12

Thank you very much! Indeed, “(gc)” brings me to a high level again (524016 - higher than at bootup, 523997).


#13

Fine.
That the number of free lisp-cells increases may have to do with the LispLibrary (I activated it to test this functionality).
After starting ulisp you can see the defined symbols using (globals).
You can disable this by comment out the line ‘#define lisplibrary’ in ulisp-esp.ino.
Of cause you can put your own functions into the LispLibrary (second TAB File LispLibrary.h) if you like. There’s an excellent description of it at the ulisp-page (ulisp.com/ -> Getting started -> Lisp Library): http://www.ulisp.com/show?27OV (I think David did a very good job in writing the documentation of ulisp – this is one reason why I love to play with it – thank you David! ;-) ).

The functions I put in the LispLibrary.h are mostly copied from the ulisp homepage to do some tests.

Maybe I’ll use the LispLibrary feature it to define some basic functionality (some kind of ‘mini-lisp-bootloader’) to read more lisp-files from a sd-card. In my mind there could be an enhanced function of loadProgram (which is able to load programs which are written in more than one line), an enhanced (emacs-like?) editor and many more (sensor function for example) (that would be a small lisp-machine one day…).
Using a ulisp-image is of cause also possible, either variants have some pros and cons.

But to realize this I have to learn a lot about lisp first… ;-) At the moment I’m going to support more esp32 specific functions like deepsleep and storing ulisp-images on the sd-card.
Later I’d like to build a ‘Tiny-Lisp-Computer’ like the ones David has already built but with esp32 and a larger (> 5’’) display…
I find it also very interesting to show what a powerful language lisp is and that it’s sometimes much easier to solve problems in lisp than in other languages.

So, there’re a lot of ideas, and for me it is a lot of fun’ playing’ with it.

Regards,
Kaef