Packages and persistent storage

[nanomonkey]

Now that the T-deck has sd-card support I’ve been using save-image and edit to create new functionality that is persistent. Unfortunately when I update the firmware these functions are no longer loadable. So I started exporting using the pprintall function to save a text version of the functions that I had written to a separate file:

(load-image)

(with-sd-card (str "image.txt" 2)
  (pprintall str))

When writing a function to load the stored functions from the text file, I realized that I was starting to write a package loader of sorts:

(defun load-package (filename)
  (with-sd-card (str filename)
    (loop
     (let (ln (eval (read str)))
       (unless ln (return nothing)))))))

Which lead me to wanting to write a function that given a package file name, and a list of symbols, would write these out to a file to be loaded later:

(defun save-package (filename lst)
  (with-sd-card (str filename 2)
   (dolist (f lst)
      (pprint f str))))

(defun add-to-package (filename list)
   (with-sd-card (str filename 1)
     (dolist (f lst)
         (pprint f str))))

Sadly pprint either prints only the name if they are quoted, or the lambda of the symbol if given a list of symbols

(save-package "packages.lsp" '(save-package add-to-package))

At which point I realized I would need to write out a describe function that returns the whole definition of a function, and I might as well have it do all symbols, which lead me to coming here and seeing if anyone has put any thought into this. Could this be build into edit. Do we need to include shadowing, gensym and other package management components, or should a package just load into the global space.

Thoughts?

[johnsondavies]

A package loader sounds an interesting idea.

Sadly pprint either prints only the name if they are quoted, or the lambda of the symbol if given a list of symbols

I think if you look at the definition of pprintall it may do some of what you want. It constructs the definition of a function by adding defun and the function name, and also handles defvars.

[nanomonkey]

I got some time to look over the pprintall function, and came up with the following additions, which I’ve added to my extensions file:

object *fn_sym_def (object *args, object *env) {
  (void) env;
  object *obj = first(args);
  pfun_t pfun = pstreamfun(cdr(args));
  #if defined(gfxsupport)
  if (pfun == gfxwrite) ppwidth = GFXPPWIDTH;
  #endif
  object *pair = findvalue(obj, env);
  object *var = car(pair);
  object *val = cdr(pair);
  pln(pfun);
  if (consp(val) && symbolp(car(val)) && builtin(car(val)->name) == LAMBDA) {
    superprint(cons(bsymbol(DEFUN), cons(var, cdr(val))), 0, false, pfun);
  } else {
    superprint(cons(bsymbol(DEFVAR), cons(var, cons(quote(val), NULL))), 0, false, pfun);
  }
  pln(pfun);
  ppwidth = PPWIDTH;
  return bsymbol(NOTHING);
}

const char string_sym_def[] PROGMEM = "symbol-def";

const char doc_sym_def[] PROGMEM = "(symbol-def symbol [str])\n"
"Prints the definition of a symbol (variable or function) defined in ulisp using the pretty printer."
"If str is specified it prints to the specified stream. It returns no value.";

{ string_sym_def, fn_sym_def, 0212, doc_sym_def },

This creates a ulisp function symbol-def which pretty prints the definition of a symbol that is currently saved within globals. With this I can now use the following ulisp functions to do some basic “package” management:

(defun load-package (filename)
  (with-sd-card (str filename)
    (loop
     (let (ln (eval (read str)))
       (unless ln (return nothing)))))))

(defun save-package (filename lst)
  (with-sd-card (str filename 2)
   (dolist (f lst)
      (symbol-def f str))))

(defun add-to-package (filename list)
   (with-sd-card (str filename 1)
     (dolist (f lst)
         (symbol-def f str))))

Note these loaded “packages” will overwrite any existing symbols, and does no attempt at creating a namespace, shadowing variable or anything that you’d normally expect from a Common Lisp package.

[AbleTheAbove]

Hello, I believe this code no longer works. Or I am having some issue with getting it to work!

I have added the not-lisp to the extensions file I made

load-package seems to complete but do nothing
save-package and add-to-package do not even complete or I provide the wrong arguments.

[nanomonkey]

Are you getting any error messages?

I assume that you’ve enabled the sdcard reader, and have tested that you can read and write to it.

You may want to read the section on adding extensions:
http://www.ulisp.com/show?19Q4

The C function definition, doc strings, and symbol names need to be added for the uLisp code will work. As will an entry in the symbol lookup table, which is probably what is missing from your code if you’ve only added what I placed above. You’ll need to add something like the following to the symbol lookup table:

{ string_sym_def, fn_sym_def, 0212, doc_sym_def },

[AbleTheAbove]

I have enabled sd card reader
I have closely followed the extensions sample
I have added the C function definition, doc strings, and symbol names

I do not get errors but load-package doesn’t seem to work
It fails to load silently

Which might be a problem of me not using save-package properly

Forgive me because of my lack of lisp knowledge but does your decls of save-package and add-to-package have typos related to list and lst

[AbleTheAbove]

Sorry for the double post but here is my code

https://git.ablecorp.us/able/tdeck-ulisp

[nanomonkey]

Yes, I believe the add-to-package uLisp code above has a typo, it should be lst throughout.

Note: filename is the file you want to write to, and lst is a list of uLisp functions to write (or append in the case of add-to-package) to the file.

(add-to-package "my_functions.lsp" '(new_function another_function))

[AbleTheAbove]

Wonderful! I believe saving works.

However when I use (load-package "hello.lsp") the hello fun is not loaded.

999853> (directory)
("funcs.lsp" "ULISP.IMG" "start.lisp" "hello.lsp" "ext.lisp" ".Trash-1000")

999853> (with-sd-card (str "hello.lsp")   (read str))
(defun hello nil (print "hello") )

999853> (load-package "hello.lsp")

999853> (pprintall)

(defun start nil (print "hinr"))

(defun add-to-package (filename list)
  (with-sd-card (str filename 1) (dolist (f list) (symbol-def f str))))

(defun save-package (filename list)
  (with-sd-card (str filename 2) (dolist (f list) (symbol-def f str))))

(defun load-package (filename)
  (with-sd-card (str filename) (loop
      (let (ln (eval (read str)))
        (unless ln (return nothing))))))

999853> 

[nanomonkey]

It looks like there is another typo, there should be two parenthesis around the let binding:

(defun load-package (filename)
  (with-sd-card (str filename) 
    (loop
      (let ((ln (eval (read str))))
        (unless ln (return nothing))))))

I don’t have the ability to test this right now…I’ll check my notes when I get home if this doesn’t work for you.

If you want to load just one function from a file use load:

(defun load (filename)
  (with-sd-card (s filename)
    (eval (read s))))

[AbleTheAbove]

The updated load-package function works on code I write to the sd card from another computer

save-package leaves a strange unprintable character in the dumped file which leaves us with

999852> (load-package "hello.func")
Error: undefined: 
999836> 

[AbleTheAbove]

@nanomonkey Ping reminder

[nanomonkey]

Hi, sorry I did not have a T-deck or esp32 handy to retest my extension file.

I’m unable to reproduce your error on my end. The save-package, load-package, and add-to-package functions work as expected for me.

I do see unprintable characters at the Arduino serial monitor, but I believe those are just carriage returns.

[dragoncoder047]

It would be worthwhile to note that my own patches that run the file main.lisp off the SD card upon boot, sometimes result in this error. With no SD card plugged in, I get a “failed to open file” error (as expected), but when the card is connected and main.lisp exists, it sometimes prints “Error: undefined:” after it runs the file, if I recall correctly. So it might be a problem with uLisp’s handling of EOF inside (read). I do not believe it is an unprintable garbage character. If it is, maybe it’s just the NUL character not being handled right?

[hasn0life]

I know that sometimes when I turn on the t-deck there’s some stuff in the buffer that evaluates to nonsense and I couldn’t figure out where it was coming from. I thought it was the keyboard sending random data but it could be something like an uninitialized or poorly initialized buffer or pointer. It happens kinda randomly

[nanomonkey]

Sorry, I wasn’t evaluating the save/load package on a T-Deck, I went ahead and tried it out there and found that the c function superprint is adding extra characters for highlighting purposes on the T-Deck.

Note: this also causes a bug with pprintall.

Until this is fixed you can use the following in your extension file:

void hyperprint (object *form, int lm, pfun_t pfun) {
  if (atom(form)) {
    if (isbuiltin(form, NOTHING)) printsymbol(form, pfun);
    else printobject(form, pfun);
  } else if (quoted(form)) {
    pfun('\'');
    hyperprint(car(cdr(form)), lm + 1, pfun);
  } else {
    lm = lm + PPINDENT;
    bool fits = (subwidth(form, PPWIDTH - lm - PPINDENT) >= 0);
    int special = 0, extra = 0; bool separate = true;
    object *arg = car(form);
    if (symbolp(arg) && builtinp(arg->name)) {
      uint8_t minmax = getminmax(builtin(arg->name));
      if (minmax == 0327 || minmax == 0313) special = 2; // defun, setq, setf, defvar
      else if (minmax == 0317 || minmax == 0017 || minmax == 0117 || minmax == 0123) special = 1;
    }
    while (form != NULL) {
      if (atom(form)) { pfstring(PSTR(" . "), pfun); printobject(form, pfun); pfun(')'); return; }
      else if (separate) { 
        pfun('(');
        separate = false;
      } else if (special) {
        pfun(' ');
        special--; 
      } else if (fits) {
        pfun(' ');
      } else { pln(pfun); indent(lm, ' ', pfun); }
      hyperprint(car(form), lm+extra, pfun);
      form = cdr(form);
    }
    pfun(')');
  }
}

object *fn_sym_def (object *args, object *env) {
  (void) env;
  object *obj = first(args);
  pfun_t pfun = pstreamfun(cdr(args));
  #if defined(gfxsupport)
  if (pfun == gfxwrite) ppwidth = GFXPPWIDTH;
  #endif
  object *pair = findvalue(obj, env);
  object *var = car(pair);
  object *val = cdr(pair);
  pln(pfun);
  if (consp(val) && symbolp(car(val)) && builtin(car(val)->name) == LAMBDA) {
    hyperprint(cons(bsymbol(DEFUN), cons(var, cdr(val))), 0, pfun);
  } else {
    hyperprint(cons(bsymbol(DEFVAR), cons(var, cons(quote(val), NULL))), 0, pfun);
  }
  pln(pfun);
  ppwidth = PPWIDTH;
  return bsymbol(NOTHING);
}

[johnsondavies]

the c function superprint is adding extra characters for highlighting purposes on the T-Deck

Sorry about that. It’s on my to-do list to fix it.

I think the simplest solution would be a compile option on the T-Deck (and Cardputer):

// #define screeneditor

which would only add the highlight characters for the screen editor if enabled. What do you think?

[dragoncoder047]

It COULD also be another parameter to superprint(), so that it can be used without the special characters when dumping with pprintall() to a stream-that-may-not-be-the-screen, and with the special characters when it is known to go to the screen.

[johnsondavies]

I agree that would be a cleaner solution, but I’m not sure that many people use the screen editor.

[nanomonkey]

Either option seems suitable. My first instinct was to just make it a parameter for superprint.

I may be wrong, but isn’t highlighting only used on the graphic’s display? So couldn’t you only apply the highlighting characters when streaming to the display, and leave them off for all other streams?

if (pfun == gfxwrite)...