Ulisp-2.4-esp with 4 MByte PSRAM support


#1

Hi,
I added PSRAM support to ulisp-2.4-esp for the esp-32-WROVER modules (should work with other modules too, but I didn’t test it). It seems to work, even if it is not fully tested now.

This version incorporates a few patches to newsymbol (included in version 2.1 (?) to improve the use of Workspace-cells on systems with limited RAM) and compact-image (used by save-image) (see LARGE_WORKSPACE_SETUP symbol in the sources). Without these the extra memory made some operations of uLisp rather slow, because it was designed to run on systems with very limited RAM.

You can download this version of ulisp at https://github.com/Kaef/ulisp-esp .

Feedback is very welcomed.

Kindly regards,
Kaef


#2

Dear @Kaef, dear @johnsondavies,

I confirm that I could upload this to a ESP32-WROVER.

Seeing over 500K cells free sort of feels absolutely brilliant. - But what’s up with that alleged corruption of save-image? Is there any “safe” way to operate this variant, perhaps over some sort of string output? - And what is the matter with the ESP 8266 boards, why are they supposedly having 80k RAM but 3k cons cells … shouldn’t it be more in the range of 8K, given that the Due gets 10K?

Thank you for extending uLisp to new boards, things are getting really interesting, and I know of no other dialect that scales nearly 200x just like that! :)


#3

OK, what about this:

Would it be possible to print all definitions to sd-card like with pprintall, just with a stream object?

It then should be possible to read them back in by evaluating the string from the sd-card. Can pprintall be made to accept a stream?


#4

Hi Aeneas,
great that this version works on your system.
I’m working on adding sleep-modes to ulisp (sleep modes on ESPs behaves a bit different than on most other systems: after wakeup the system restart from beginning on ESPxx, while other systems continue to work on the next command after deepsleeping. Therefore the sleep-command from ulisp have a unexpected behaviour for most developers on ESP systems. Be aware that this is a ESP system behaviour and can not be change from ulisp.).

After that I want to expand the save-image to store images on sd-card if a filename is given to the command.

I did not do much tests with ulisp-2.4 on esp8266 systems, but I think the wifi-stack takes a lot of the RAM memory.

Do you have a dedicated reason to use esp8266 over esp32? (well, I have some esp8266 boards too and maybe I’ll use them with ulisp, but I think esp32 is much more interesting)

I think you could save the pprintall-output to a file on sd-card (I’m a beginner in programming lisp, but I think David could show how to do this). I think pprintall removes the name of the defuns (replace the name with ‘lambda’), so the name must be added to be able to read the saved file back. Again, I think this should be possible to solve with ulisp statements…

Kindly regards,
Kaef


#5

Currently pprintall doesn’t accept a stream, but pprint does, so you could use the function described in this thread to pprint all your definitions to a stream:


#6

@Kaef can confirm this, but as I understand it there’s no corruption; it’s just that the standard ESP version of uLisp is very slow doing a (save-image) when using all the memory in the ESP32-WROVER module.


#7

Thank you very much, @johnsondavies !

I discovered this version has some issue with the garbage collection (or at least the way this is printed) - note the diminishing number of cells:


#8

Yes, that’s not right. I haven’t tested your version with PSRAM support. Could you try some additional tests:

  • The standard 2.4 version with the code to address PSRAM, but without your speedup improvements.

  • Your version on a board without PSRAM.

Thanks, David


#9

corruption of save-image

I’m not sure in this point.
I think I had corrupted saved images, but I wasn’t able to reproduce this behaviour.
I didn’t look at this because I want to make it possible to save the images on sd-card. I’m not sure if this solves this problem, but I think it may have something to do with the eeprom-functions for esp32 (which uses the flash).

My version of save-image (better: compactimage () and subfunctions) should be a lot faster than the original version if not much lispcells are in use. On a heavy used system compactimage() takes a lot of time (several minutes) (I added the output of ‘.’ so you can see that the system is working).


#10

If you recompile uLisp after making changes, and then try to load in a saved image, it’s not guaranteed to work because the saved image isn’t position independent, and the address of the workspace may have changed. Could this explain what you observed?


#11

diminishing number of cells

This is because of my patch to newsymbol () and repl (). If you call (gc) in between all cells are marked as free (this was done in repl (), but the system-reponse-time increases a lot). The (gc) is automatically called if less than 20 lispcells are free, so don’t worry ;-).

If you didn’t like this behaviour you can comment out the beginning of newsymbol () (in my version lines 361 to 363 and line 369) or you can comment out the definition LARGE_WORKSPACE (line 200), which will also removes my patch to repl (calling gc) (of course you can remove my patch to repl () without changing newsymbol () or vice versa).
The downside of doing this is that the interface gets slower without that patches.

I’m think this will be very hard to solve completely (ie. the system is still fast if many lisp-cells used) – one way could be to use small ‘chunks’ for the workspace and add a new chunk if there’s too less memory free. But if you would use many lisp-cells (large programs) ulisp would get slower the larger the workspace is, so I think this solution is not good enough (and it would be complicated to realize).
Therefore I think we have to gc the chunks separately and only if anything in it changed since last call to gc, but (again) to realize this is rather complex (and it has very low priority on my personal TODO-list…)

I think the best solution is to live with it and (if you want a predictive system-runtime) call (gc) when you have time for housekeeping in your program.

Kindly regards,
Kaef


#12

Thank you very much! Indeed, “(gc)” brings me to a high level again (524016 - higher than at bootup, 523997).


#13

Fine.
That the number of free lisp-cells increases may have to do with the LispLibrary (I activated it to test this functionality).
After starting ulisp you can see the defined symbols using (globals).
You can disable this by comment out the line ‘#define lisplibrary’ in ulisp-esp.ino.
Of cause you can put your own functions into the LispLibrary (second TAB File LispLibrary.h) if you like. There’s an excellent description of it at the ulisp-page (ulisp.com/ -> Getting started -> Lisp Library): http://www.ulisp.com/show?27OV (I think David did a very good job in writing the documentation of ulisp – this is one reason why I love to play with it – thank you David! ;-) ).

The functions I put in the LispLibrary.h are mostly copied from the ulisp homepage to do some tests.

Maybe I’ll use the LispLibrary feature it to define some basic functionality (some kind of ‘mini-lisp-bootloader’) to read more lisp-files from a sd-card. In my mind there could be an enhanced function of loadProgram (which is able to load programs which are written in more than one line), an enhanced (emacs-like?) editor and many more (sensor function for example) (that would be a small lisp-machine one day…).
Using a ulisp-image is of cause also possible, either variants have some pros and cons.

But to realize this I have to learn a lot about lisp first… ;-) At the moment I’m going to support more esp32 specific functions like deepsleep and storing ulisp-images on the sd-card.
Later I’d like to build a ‘Tiny-Lisp-Computer’ like the ones David has already built but with esp32 and a larger (> 5’’) display…
I find it also very interesting to show what a powerful language lisp is and that it’s sometimes much easier to solve problems in lisp than in other languages.

So, there’re a lot of ideas, and for me it is a lot of fun’ playing’ with it.

Regards,
Kaef


#14

Dear @Kaef ,

I have a Wemos ESP32-WROVER. Now, how do I use its SD-card? I am getting some sort of “SD card error”.

I see, you have defined in your code some MISO, MOSI etc.

Now, the SD-card area is marked as follows (on the back of the board):

SD
D2 - 12
D3 - 13
CMD - 15
CLK - 14
D0 - 2
D1 - 4

What do I need to change in your code to make SD card support possible on this board?

Kindly thank you in advance for your very obliging help.


#15

Hi Aeneas,

if your board is the one David describes at http://www.ulisp.com/show?2AJI – there is one solution shown on that page (patch the arduino file 'pins_arduino.h), but this will only work with my version of ulisp if you comment out the SDCARD_xx_yy defines at line 240.
I recommend to use the defines around line 240; they will be used in the SPI.begin() call (line 1657) :

#define SDCARD_SS_PIN    13 /* Kaef sdcard: 5 */
#define SDCARD_CLK_IO    14 /* Arduino standard: 18 */
#define SDCARD_MISO_IO    2 /* Arduino standard: 19 */
#define SDCARD_MOSI_IO   15 /* Arduino standard: 23 */

I inserted the values which should work for your board (of cause I didn’t test it because I don’t have the wemos board) (you have to enter ESP32-GPIO-numbers here, not PIN numbers!)

The sd-card must be formatted with fat-16 or fat-32 filesystem. On some boards you can only use “small” cards with 2GB or 4GB; I don’t know which sizes are supported by the wemos board.

You may copy some files to the card and enable the define SD_CARD_DEBUG (line 87). When starting ulisp the files from the sd-card should be listed (or you get an error-message).
If this works be sure that the filename given to (with-sd-card) starts with a leading “/”; if omitting the “/” you get the “SD card error” when try to read or write a file. (ie. (with-sd-card (str “/Greeting.txt” 2), not (with-sd-card (str “Greeting.txt” 2) )

HTH,
kindly regards,
Kaef

Here’s a log from my esp32:

uLisp 2.4a -- forked and extended by Kaef (https://github.com/kaef)
(c) by David Johnson-Davies - www.technoblogy.com
[...]

System information:
  reset reason: 1  wakeup cause: 0
  compiled: Nov 23 2018 09:21:43
  Features: resetautorun printfreespace serialmonitor sdcardsupport SD_CARD_DEBUG LARGE_WORKSPACE_SETUP
  initworkspace PSRAM 524281(512k - 7) cons allocated. 
SDCARD_SS_PIN = 5
SD Card Size: 1886MB
Listing directory: /
  FILE: /lisp.txt  SIZE: 48
  FILE: /foo.txt  SIZE: 13
  FILE: /hello.txt  SIZE: 11
[...]
done
  Free Heap: 233728

#16

Dear @Kaef ,

Thank you very much. Unfortunately, it still does not work. I enabled your debug-definition, and all I am getting is some sort of error telling me I did not mount the SD-card right. I then tried all possible values for the SS pin, i.e. 13, 12, 4 and 2, but still, error mounting the card upon boot-up. It is a known good card, successfully used in other Arduinos, too, and taken from another system just to be sure.

So… do I have any chance for SD card storage? :)


#17

Did you change the SDCARD_xx definitions (clk, miso, mosi)? It’s not enough changing the SS pin definition to get this work.


#18

Sorry, I didn’t read your post completely.
So you have change the definitions and it doesn’t work, rigght?
Please try an arduino example sketch and test if the card works with this.
You may also comment out my clk, mosi, miso definitions and try the way David suggested.
First, get the SD-card example working.


#19

Dear @Kaef ,

This is super annoying… Apparently I have a GENERAL issue with this board, the SD sketch does NOT work. Yes, I got that I need to change all pins, not just SS, and I found my pins_arduino.h for esp32, but no dice.

Well, I shall then order another board. Would it be possible to tell me which board EXACTLY you have? I want to get the same one as you, so debugging will be easier in the future. Is it this here, the one David recommended:


#20

Dear @Aeneas ,

I can confirm that your SD-card configuration doesn’t work!
I found a bug in my ulisp version (sometimes SD.begin() was called without parameters) – sorry for that.
I fixed (and tested) it with my module and I think it will work now, I uploaded the version to my dev-deepsleep branch (I will merge it to master soon).

Thank you very much for finding this bug!
Please let me know if it is working for you now.

I use a ESP32 WROVER module I soldered on a adapter board from Eckstein-shop ( https://eckstein-shop.de/HIMALAYA-Basic-ESP32-Adapter-Breakout-Board-for-ESP-WROOM-32-Partial-Kit). I don’t remember where I bought the ESP32 WROVER module, but there are several distributors. I connect the SD-Card with some wires, you can look at the photo on my github page. I also tested the connections which the WEMOS board without problems.
I also got a DSTIKE WiFi Packet Monitor board, but I didn’t test it with ulisp – I’ll do this ASAP.

Kindly regards,
Kaef