This micro-framework fell out of my Trinkey work and is super-helpful. It allows you to schedule stateless functions to run on a recurring basis while handling all the input/output without resorting to global storage. You can even share state across functions.
Check it out:
I’m looking for a method of scheduling tasks that don’t require the use of a busyloop or delay. How are folks managing this in their code currently without interrupts?
For example, say you want to set a pomodoro timer, that notifies you when 45 minutes have past, and then go back to editing a function/document in the uLisp editor. How would that be accomplished as minimally as possible?
I think one could create a C function (that runs after the repl function in the main loop) which checks a global association list with the time as the key and the function as the value. The function would need to be a thunk, and have all of it’s parameters supplied by the calling function.
Any other ideas?
You’re getting into OS like functionality here. There’s the two kinds of multitasking; preemptive and cooperative. uLisp doesn’t provide preemptive multitasking functionality afaik. Otoh, cooperative can be really good because you can have coroutines and closures pretty easily in uLisp. Then you’d want a framework to call them on time, which would be something like what the OP posted. I usually go with the latter style, but I haven’t done anything involving scheduling in uLisp surprisingly.
Yes, and no. Yes, I would like to extend the “LispMachine” concept to be a functional lisp OS. But mostly I’m trying to work around the limitations of not having the event driven capabilities that are built into most languages. In bare metal microcontroller C, I would have interrupt handling, same with Scheme or Common Lisp.
The functionality I’m looking for is not much different than adding break to a running REPL, and evaling a thunk, and then returning to the original REPL eval. Or trampolining between two tail call optimized recursive functions, one of which is checking for a specific event, and then finally running the tail call (thunk) when it evaluates true.
Any ways, I was just wondering if anyone had a method of getting around this limitation in uLisp. We have millis, for-millis, and delay, but no way of concurrently waiting without locking up the REPL with a busy-wait.
I thought a bit about this. I believe a very simple kind of cooperative multitasking could be done using a scheduler as the main infinite loop in the repl and encapsulating all other functionality within closures. This should save the context of the “tasks” and make them independent. The drawback would be that all functions of a task need to sit within that closure - but for smaller programs, why not? For larger programs this might create blocks that are hard to read, though, but the overhead would still be rather small.
Another thing I’ve been thinking of is the timer interrupt functionality of processors like the 2350. It should easily be possible to extend uLisp with small functions that set a global “time’s up” flag whenever the timer interrupt fires. This could be used by cooperative tasks as some kind of signal it’s time to give way.
All that would need to be strictly cooperative since there would be no means of protection (apart from variables within closures), and the whole idea obviously fails as soon as blocking functions like (delay) are used.
Bottom line: I think it’s possible and useful for several purposes, but on the other hand there’s not that much difference to a “main loop” that calls functions whenever a timespan measured with “millis” has passed. A scheduler as described would have very restricted means of control.
But then again I don’t think it’s a bad thing to circumvent a real OS, because IMHO this quickly leads to some kind of hard to track black box behaviour/annoying abstraction levels and the usual bureaucracy overhead I’m glad -not- to be forced to deal with using uLisp.
You can see what I did when I attempted my operating system experiment uos uLisp Operating System as far as keeping programs inside closures and passing control between various applications. I don’t think it would be too hard to add a scheduler to that framework, but I could be naive. The whole difficult aspect is writing everything else in a manner that cooperates with the framework.
My ultimate goal was to write a much more full featured REPL (than the default uLisp one), but it was surprisingly hard, even harder than writing an editor. But once you have a REPL that can support getting “interrupted”, the scheduling part should be easy afaik (just call millis every time you check for a keypress and compare it to a list of tasks, or whatever scheduling algo you want to use).
Yes, I think adding a scheduler to the REPL would be require doing continuation passing style evaluation, where each lambda has a context that it has captured responds to instead of one ENV stack for eval. Continuations are often done with lexical closures, so perhaps this is what you’re talking about. Sadly the term closure is a bit overloaded in math and functional programming.
A simpler method may be to have a C function, that is called before or after the REPL function’s call in the main loop, which checks a task/job queue to see if it’s time has come up, and then either runs it directly or drops it into the REPL to run next. The later would allow for “interactive” expressions that require feedback from the user to run, as opposed to just running thunks that update global variables alone.
This would of course require that the REPL has finished running, so infinite or long running programs would block the scheduler in similar ways that garbage collection is blocked, and has to be called manually. This wouldn’t be much of a problem for multi core micros such as the ESP32, which run FreeRTOS, as you could just pin the scheduler to a RTOS Task on another core.
New scheduled tasks could be added to the task queue by passing the time and function to run similar to the above managed state event loop.
I feel like you might want to give a more specific example of how exactly this should play out. Like when you say “For example, say you want to set a pomodoro timer, that notifies you when 45 minutes have past” how exactly would the notification work? Will it write something to a screen? or a buffer? Does it need to share any data with the REPL? Does the user need to confirm it to continue using the REPL?
Because if its something like a popup, you’d want something resembling a windowing system for example, and then said windowing system could be responsible for how state is stored in various windows. Otoh if it just prints a message in the REPL then you might not need to save state at all.
I’m actually trying to send out LoRA and BLE radio broadcasts at different allotted times, but that is irrelevant. There are a variety of use cases, the pomodoro timer being a simple one. If you wanted to set up a notification to pop up on the window, you’d still hope that the window manager doesn’t have to block the REPL that is running elsewhere.
Ok I played with this issue a bit and I think I see why this is tricky. I was doing multitasking stuff on devices with integrated keyboards and writing my own (non blocking!) functions for text editing. I’m guessing you’re trying to do multitasking with the standard serial input.
So what you’re probably running into is that serial input is blocking. Likewise, functions like read, read-line and co. are also blocking. So what you might have to do is either:
read-line etc. and pass whatever strings you get into eval (effectively making your own REPL in ulisp), running the REPL input concurrent with a millis event loop like the OP’s library. Here’s an extremely basic example(loop
(let ((input (read-nonblocking))) ;; read-nonblocking would return nil unless there's an atom or list
(when input (eval input)))
(when (> event1-time (millis))
(do-event1)
(update-event1-time))
(when (> event2-time (millis))
(do-event2)
(update-event2-time)))
Does that make sense or help at all, or am I missing the mark?
I’m currently not running into any problems, because I just write the solution in C, but I’d like to have the user be able to utilize the lisp REPL to write non blocking code that can respond to events without having to rely on a busy loop like what you are doing.
If I’m not mistaken, this busy loop blocks the REPL from returning. If you go down that route you have to constantly poll the status of millis or volatile variables, and garbage collection is never handled by the system, so you have to call (gc) periodically.
It would be much nicer if one could spawn separate async tasks that park their operation and wait for incoming events. The events could be appended to a uLisp stream buffer by a interrupt handler so that interrupts can be dealt with succinctly and events aren’t missed when the REPL is dealing with other calculations.
Clojure deals with this sort of multi-tasking with go-blocks and channels. A channel is a message queue on which one could stream incoming and outgoing events messages. The go blocks are bits of code that get compiled down into an independent state machine that can deal with incoming messages, and otherwise are parked.
In essence, it would be like your above loop structure, that are independent, and only get called when an event triggers them (interrupts).
So far I’ve just been using the FreeRTOS tasks and timers to do async functionality on the two cores of Esp32s:
TimerHandle_t theTimer = NULL;
void TimerCallback(TimerHandle_t xTimer) {
... whatever you want to run when the timer calls...
}
void setup() {
theTimer = xTimerCreate(
"theTimer", // Timer name
1000 / portTICK_PERIOD_MS, // 1s period
pdTRUE, // Auto-reload (periodic timer)
NULL, // Timer ID
TimerCallback // Callback function
);
if (theTimer == NULL) {
Serial.println("Failed to create timer!");
while (1);
}
xTimerStart(theTimer, 0); // Start timer immediately
}
or async tasks pinned to alternate cores that talk over a queue:
TaskHandle_t sendTask;
TaskHandle_t recieveTask;
struct packet {
unsigned char auth[AUTH_LEN];
unsigned char message[MESSAGE_LEN];
};
QueueHandle_t queue;
void setup() {
//create a task that will be executed in the Task1code() function, with priority 1 and executed on core 0
xTaskCreatePinnedToCore(
sendTask, /* Task function. */
"SendTask", /* name of task. */
10000, /* Stack size of task */
NULL, /* parameter of the task */
1, /* priority of the task */
&sendTask, /* Task handle to keep track of created task */
0); /* pin task to core 0 */
xTaskCreatePinnedToCore(
recieveTask, /* Task function. */
"RecieveTask", /* name of task. */
10000, /* Stack size of task */
NULL, /* parameter of the task */
1, /* priority of the task */
&recieveTask, /* Task handle to keep track of created task */
1); /* pin task to core 1 */
queue = xQueueCreate( 10, sizeof( struct packet ) );
if(queue == NULL){
Serial.println("Error creating the queue");
}
}
void sendTask( void * pvParameters ){
for(;;){
struct packet myPacket;
myPacket.auth = "Bob"
myPacket.message = "Hello World!"
xQueueSend(queue, &myPacket, portMAX_DELAY);
}
}
void recieveTask(void *parameter) {
for (;;) {
struct packet recievePacket;
if (xQueueReceive(queue, &recievePacket, portMAX_DELAY)) {
Serial.printf("Recieved %s from %s\n", recievePacket.message);
}
}
}
void loop() {
if(queue == NULL){
Serial.println("Queue was not created");
return;
}
...
}
Something I’ve been thinking about for running programs concurrently is splitting the main memory into independent segments, and have the interpreter be able to pick where to run depending on various conditions. They can even have separate garbage collections happening. And if you have multiple cores then they can run in parallel as well. Then you’d have some kind of message passing scheme between them. I don’t know if it’ll be useful in practice though.
Otherwise yea, you’d want some kind of fundamental concurrency construct.
There does appear to be a memory arena size that the garbage collector can traverse quickly, that provides enough space for most computations. One could easily provide for multiple REPLs, each with their own memory arena and gc.
I believe some lisps allocate groups of lisp cells on a page by page basis (4kB pages), so that you can garbage collect over pages that are dirty, and skip the rest. I believe they also have an indicator on each page index that says what the cell size is so that larger heaps can be allocated to a page that is broken out into larger cells structs. But that’s another complication to manage.