What would you like to see in uLisp in 2024?

#31

dragoncoder047 has implemented a pretty full implementation of macros in uLisp if you need them:

Yes, I know about this and already use them, but it would be good to add it to the core language. Macros are a very important part of Lisp, and a Lisp implementation without macros is simply not a real Lisp in my opinion. Leaving this out of the language and making everyone apply the patches themselves gives a lot of unnecessary friction to be able to use uLisp.

It also makes any code that is written using macros not portable. I want to be able to use macros in my code and to share this code with others without having to distribute the code alongside a patched version of uLisp.

uLisp is not thread safe, and it would be a major rewrite to change this, so I can’t see how this can be done.

Could it be done in a non-threadsafe way and let the programmer worry about the safety of his own threads? Or even an implementation where the interrupts/multithread/separate core is running in its own namespace or completely separate process with no access to the main workspace would still be a valuable addon.

Also, a with-littlefs special form much like with-sd-card would be nice to have.

Good suggestion, and it would be relatively easy. I’ll look at it.

Nice, thanks!

#32

Macros are a very important part of Lisp

I agree that they are important in a full Common Lisp, but I’m not convinced they are necessary for the applications that uLisp is targeted at. Can you give some examples of applications that you need macros for?

Could it be done in a non-threadsafe way and let the programmer worry about the safety of his own threads?

I don’t see how this would work. I’m assuming from what you said (“not just the polling variant you suggested”) you want an interrupt to be able to run Lisp code, or even a Lisp function. This implies that eval has to be reentrant, and I’m not sure how that would be possible.

#33

I agree that they are important in a full Common Lisp, but I’m not convinced they are necessary for the applications that uLisp is targeted at. Can you give some examples of applications that you need macros for?

I believe macros are essential to be able to clearly and concisely formulate good programs, no matter what the application is. I don’t think the size or type of application makes any difference to this. Macros allow you to create your own “syntactical sugar” and create things like custom with-* blocks, or implement doarray similar to dolist and all sorts of things you just can’t do with plain functions. Sure, you can implement the functionality of doarray as a function, but you can’t make the resulting function take the arguments in the same way as dolist without it being a macro, so the code would be uglier and less readable without macros, and it could not be implemented in a consistent way.

It would also improve interop of Common Lisp code in uLisp. Since uLisp is mostly a compatible subset of Common Lisp a lot of code can already be reused directly from CL. Adding macros would increase the amount of code reuse/interop that is possible. Not only directly from supporting defmacro and allowing to reuse existing CL code that uses macros, but it would also make it possible to use macros to quickly implement CL language features that are missing from uLisp in order to use CL code that depends on some missing language features.

I think that you are either underestimating what people are using uLisp for, or not really understanding the power of macros. Any program that is not simply a blinking LED “hello world” example is sufficiently advanced that it could potentially benefit from macros, so I do not agree at all that macros are not necessary for the type of applications uLisp is targeted at. Also, this type of reasoning becomes self-fulfilling, since the type of applications people are using uLisp for might be limited by missing features such as macros and interrupts, not the other way around.

I think that macros (and quasiquote and unquote) are possibly the biggest bang-for-the-buck addition to uLisp currently. It requires very little effort from your part to add one of the existing implementations to the language, but it gives a very big improvement to the power and expressiveness of the language. I don’t see any downsides or reasons why it shouldn’t be included.

I don’t see how this would work. I’m assuming from what you said (“not just the polling variant you suggested”) you want an interrupt to be able to run Lisp code, or even a Lisp function. This implies that eval has to be reentrant, and I’m not sure how that would be possible.

I don’t know too much about threads and thread safety, and I’m not familiar with how things are implemented in uLisp so I might be missing something obvious here. Could you explain what you mean by “eval has to be reentrant”?

#34

Could you explain what you mean by “eval has to be reentrant”?

An interrupt can happen at any time, such as during the evaluation of a Lisp statement. If you want the interrupt to be able to evaluate a Lisp function (the interrupt service routine), it needs to call eval. Therefore, the eval that’s executing needs to be re-entered to evaluate the interrupt service routine.

#35

Okay, that makes sense. In that case I think some kind of compromise between your suggested polling variant and a real interrupt could be good enough, such that it is uLisp itself that does the polling, not the Lisp program.

I don’t know exactly how the eval loop is implemented, but I assume it runs one function/expression at a time. Maybe the eval loop could poll for interrupt events each time it has processed one expression and call the triggered interrupt functions if any? Or would this cause a lot of overhead?

Alternatively, if I remember correctly there is a special register (on AVR chips at least) that enables and disables interrupt handling. Maybe this could be enabled and disabled when entering/leaving eval? (I’m not entirely sure if this register will queue up events that happened while the register was disabled or not though)

#36

dragoncoder047 has implemented a pretty full implementation of macros in uLisp

Has anyone gotten this to work in their own uLisp codebase? I tried to port over the changes they made and kept getting errors. There are quite a few code convention changes that may have hung me up, but I was under the assumption that there was still work to be done.

(defmacro triple (x) `(+ ,x ,x ,x))

(triple 4) 
=> Error: 'defun' too few arguments
(macroexpand-1 triple)
=>(rest (x) (defun (+ (backquote x) (backquote x) (backquote x))))

Any thoughts?!

I have to agree, a lisp without macros feels more like a shell scripting language rather than a true lisp. Macros would allow the ability to get rid of many of the #IFDEFs used for configuring a system, allow for writing cleaner/readable code that does destructuring, error correcting or creates special forms that aren’t currently implemented in C.

As far as interrupts go, could the interrupt handlers be simple lambdas that are evaluated with their own limited environments so as to not mess with the greater REPL environment? Or streams can be generalized (currently I believe there are only two?) so that the interrupt handler can output to a dedicated stream.

#37

Has anyone gotten this to work in their own uLisp codebase? I tried to port over the changes they made and kept getting errors. There are quite a few code convention changes that may have hung me up, but I was under the assumption that there was still work to be done.

Yes, I managed to get it working, but I had to try several times before it worked. I think it is important to get stuff in the right order in the enums and the list of functions, and there were also some things that was not mentioned in the blog, but was in the code in his repo. Also, some things needed to be updated to work with newer uLisp version.

Take a look at my fork, where I’ve successfully applied it to the latest ulisp-arm. I put everything related to the macros in a single commit, so it should be pretty easy to pick out. Let me know if it works for you.

#38

Interesting, my update was essentially the same as yours, only with a slightly different ordering of the function definitions. I placed them similar to yours, and got rid of the forward declaration for is_macro_call and now get the following response:

21047> (defmacro triple (x) `(+ ,x ,x ,x))
triple

21025> (triple 4)
Error: 'unquote' not valid outside backquote

which appears to be the function

object* bq_invalid (object* args, object* env) {
    (void)args, (void)env;
    error2(PSTR("not valid outside backquote"));
    // unreachable
    return NULL;
}

Which is surprising, even though I see that the symbols unquote and unsplicing are linked to that function

  { stringunquote, bq_invalid, 0311, docunquote },
  { stringuqsplicing, bq_invalid, 0311, docunquotesplicing },

I would assume that these symbols would be replaced by macroexpand, and not evaluated as symbols.

Did you test unquote(,) and unquotesplicing (,@)? What happens when you run said macro?

#39

Hi David

I’d like to have:

  1. Upload new uLisp binary into device without erase the saved image/datas

Since I’d like to save some persist data in the device, I’d like them persist all the life time, otherwise, I need to find a way to backup data every time I want to burn a new uLisp version into the device and restore them later.

#40

Whether the saved image becomes invalid depends on the upgrade, and how saveimage is implemented on the platform, but it would be difficult to solve this in all cases.

The simplest way to make a backup of your data is to do:

(pprintall)

in the REPL, and then copy and paste the listing into a text file. You can then evaluate it after the upgrade to reinstall your data.

#41

Agreed, it’s a natural way in Lisp.

I tested upload new uLisp image into Lisp Badge LE, the saved image(data) remains loaded after reboot.

#42

3 posts were split to a new topic: Would it be possible to add “make-symbol”?

#45

A post was split to a new topic: How to output text only to the REPL

#46

I am the same dragoncoder047 from GitHub.

My sincerest apologies for any confusion caused by me not testing my own tutorial.

Well, I’m here now, ask me questions.

#47

As far as l know, to make things thread safe, you just need to add locks around all global variable writes.

FreeRTOS (on the ESP32) already has a built-in std::mutex implementation that plays nicely with its tasks, so this may be all that is necessary.

For Teensy, I found this library that provides std::thread, std::mutex, and std::lock_guard:

I can’t think of any other platform that would make sense to implement threads on, so I stopped looking after that.

#48

Some more things I’d love to see in the next version:

  1. A way for programs to quickly test if a particular extension or feature is available by name, and not have to rely on kludges or tricks. For example, to test if the platform has floating point:

    (string= (princ-to-string (/ 1 2)) "0.5")

    But to test if WiFi support is included:

    (not (eq nothing (ignore-errors with-client t)))

    These work but it’s not entirely clear what they’re testing for.

    s7 Scheme has (provided?) to test if a feature is available. The equivalent of the above two tests would be:

    (provided? 'floating-point)
(provided? 'wifi)

    The equivalent name in lisp speak would probably be featurep.

    s7 also has a global variable *features* that holds all of the currently-loaded features, which provided? uses age the user program can modify. On my particular fork, *features* would probably be set to (core floating-point wifi catch-throw macros).

  2. Ability to have an arbitrary number of extensions loaded, subject only to available memory. If each extension has a name, then those names could be used to populate the *features* list.

  3. Function overloading by extensions. This would allow extensions to handle new cases of existing functions, and still defer to the core function. I could conceivably see this used to allow the bigint and floating point extensions to actually be extensions and be loaded in with (2), and, defer to the core arithmetic functions that can only do integers. The bigint functions could just overload the existing arithmetic and turn regular numbers into bigints.

    This could be implemented by the reader always picking the last extension’s overload of the function, and then if there is a number-of-parameters mismatch, the function longjmp’s out to a special point, or it returns a special sentinel low pointer, the evaluator would try again with the next overload, up to and including the last one (there core implementation), and then finally throw an error if none of them work.

  4. Better streams. Provide (open-X) for all the (with-X) forms as well as (close). Allow multiple SD files to be open for reading at once. (peek-char).

    I previously discussed some implementation details here.

  5. Proper **&key**word arguments.

    At the very least, :if-not-exist on all of the stream-opening functions. In some cases, you want to open the stream to test if the stream exists (for example, I2C scanner) and you would want :if-not-exist nil, but other times you expect the target of the stream to exist and so :if-not-exist :error will be appropriate.

I drafted some code that might work to allow this and some other things here.

#49

@dragoncoder047 Are you able to run the macros that I suggested earlier and not get an error? I’m curious where my codebase has gone wrong. I think I translated your function binary flag macros correctly, and I’m not missing any functions. Perhaps the ordering of definitions is off.

#50

I get the expected output:

8977> (defmacro triple (x) `(+ ,x ,x ,x))
triple

8953> (triple 4)
12

8953> (triple (print 'foo))

foo 
foo 
foo 
Error in +: argument is not a number: foo

8953> (macroexpand-1 '(triple (print 'foo))) ; how macroexpand-1 is supposed to be used
(+ (print (quote foo)) (print (quote foo)) (print (quote foo)))

8953> (macroexpand-1 triple) ; how nanomonkey used it
(macro (x) (backquote (+ (unquote x) (unquote x) (unquote x))))

I’m not sure what’s going wrong on your version. Start again with a fresh clean stock uLisp, and then apply my patches exactly as described. If you get any errors that’s on me, sorry… Please let me know what errors you do get because I want to be able to fix them.

Do note that if you’re viewing my page on a smartphone, the highlighted lines may be wrong. It is a known bug in the syntax highlighting library I am using (the code itself text-wraps but the highlighting does not, leading to misalignment). edit: I disabled code line wrapping on my website, the highlights will be correct but you’ll have to scroll.

#51

Thanks for your list of “Some more things I’d love to see in the next version”.

1. A way for programs to quickly test if a particular extension or feature is available by name

Great idea! The Common Lisp way of doing this is with the *features* variable. You would simply be able to do:

(member :floating-point *features*)

to find if your version of uLisp has a particular feature. Initially I suggest populating it with features from this table:

Lisp for microcontrollers - Versions

2. Ability to have an arbitrary number of extensions loaded

My first attempt at the Extensions feature did allow an arbitrary number of extensions files, but it made it a lot more complicated, and there was an impact on uLisp’s performance. I suppose if there’s enough interest in this I could revisit it.

3. Function overloading by extensions.

It’s a nice idea, but do you think it would really be useful, apart from the bigint example?

Also, can’t the equivalent be achieved by redefining the built-in functions? For example:

(defun * (a b) ($* a b))

4. Better streams

Yes, we’ve talked about this before. I had a look at it but it was non-trivial to use peek rather than the way I do it at the moment with LastChar. I can have another look at it.

5. Proper &key parameters

As you know, I’m keen to keep uLisp compact, rather then letting it gradually grow towards a full Common Lisp, and this seemed a good point at which to draw the line.

Because uLisp is an interpreter, supporting &key parameters would have a performance impact, even when they’re not used.

Also, once uLisp supports &key parameters there would be an case for many functions to be extended; for example, all the sequence functions could be expanded to support the full set of keyword parameters such as :start, :end, :test etc.

I feel that there are other ways of achieving what you’re suggesting without having to add full support for &key parameters; for example, like I’ve done with make-array.

#52
  1. Yes, except I was intending for *features* to be something that gets pre-populated upon boot. Currently, it isn’t.

 

  1. That would work for the bigints, but I was thinking more of extensions that add new types, rather than just more numbers, where more extensive overloading would be necessary. A trivial example would be this:

    (defmacro + (&rest strings)
  `(concatenate 'string ,@strings))

    Now you can use + to concatenate strings, but you can’t add numbers anymore. If this was designed to work with both strings and numbers, it would get extremely long.

    TBH you probably don’t need this unless you manage (2), because if there is only one extension, it can explicitly defer to the built-in function.

    The only thing in here that needs to change for this to be supported is the reader, since it always picks the first entry for the symbol in the tables, not the last, and so an extension’s version will never be called if it used the same name as an existing function.

  2. Another argument here is that the “extensions” that have to use streams (WiFi and GFX), can’t be packaged as proper extensions in separate files, because the streams have to be included in the main uLisp source. If there was a way for extensions to declare what streams they provide, that would enable people to clearly turn on and off what features they want and don’t want, while also making core uLisp shorter so it’s easier to tinker with in the Arduino IDE (which I’ve found sometimes struggles on extremely large files).

    Currently the stream lookup functions are a bit of a mess and due for an overhaul, since they use a synchronized enum and string array to store the stream names (this reminds me of the enormous builtin_t enum of uLisp 4.3), but there is no table for the gfuns/pfuns — gstreamfun(), a weird function that is basically a lookup table made out of conditional statements, is used instead.

  3. Okay, since keyword arguments are a little complicated I understand your concern. However, I think that some of the more powerful platforms (ESP32, Teensy) have enough horsepower to handle proper keyword arguments. I don’t see how keyword arguments can be packaged purely as an extension, so could they be put behind a #define switch?

From what I can see, a lot of these limitations would be removed if the extreme reliance on enums that are used as indexes into a table is eschewed — why not just use a pointer directly to the entry? Especially with the built in functions and special forms. Currently, getting the table entry involves many operations: get index -> multiply by element size -> get 0th entry’s address -> add -> dereference pointer. If the pointer is stored directly, it becomes: get pointer -> dereference pointer. With that in place, there may even be enough time saved to allow (2).