Proposed solution to serial buffer problems

#1

A very handy feature of uLisp has always been the ability to enter uLisp programs through the serial port, such as by pasting them directly into the input field in the Arduino IDE’s Serial Monitor window.

However, with some platforms pasting long programs can overflow the serial buffer, resulting in a string of error messages due to the dropped characters. The solution was either to paste the program in smaller sections, or increase the size of the buffer before compiling and uploading uLisp to the board.

Neither of these solutions is ideal. Having to paste the program in sections is annoying, and increasing the size of the buffer takes valuable RAM away from uLisp’s workspace.

The proposal

My suggestion is to provide a new echo function that allows you to turn off or on the echoing of serial characters. Before pasting a long listing you would do:

(echo nil)

and the text you then paste in will not get echoed to the Serial Monitor (although Lisp output will still appear as normal). To return to the normal REPL mode you would type:

(echo t)

It will work equally well if you include (echo nil) and (echo t) at the start and end of the listing before copying and pasting it.

Does this sound like a good solution, or is there a better one?

#2

The main question I have is whether it will fully solve the problem - at least it seems to me that it would still be possible to overflow buffers when sending multiple top-level forms, if uLisp takes long enough to execute one of them. It’s almost certainly a step in the right direction. The best solution would seem to be proper flow control, but various bits and pieces I have seen indicate that neither hardware nor software flow control is adequately supported by most USB serial devices, which is a shame.

If it wasn’t for the Arduino Serial Monitor, I would suggest using the DC3 and DC1 control codes instead of a function call; it seems almost wasteful to add a symbol to control this. I don’t think Arduino reads control characters, though, so that would be a serious usability hindrance.

#3

it seems to me that it would still be possible to overflow buffers when sending multiple top-level forms, if uLisp takes long enough to execute one of them.

In general yes, but the most important case to solve is when the forms are either defun or defvar.

it seems almost wasteful to add a symbol to control this

Yes, I agree. I’ve tried to think about whether it would be possible to do it automatically. Some ideas:

  • Turn off echo if a comment is encountered, since you don’t normally type comments in the REPL.

  • Turn echo back on automatically if there is no activity on the serial port for some time interval, such as a second.

#4

Absolutely. I’m thinking about stuff like the tests, which I still haven’t automated properly in my preferred working environment.

That might work, but could cause other issues if it can’t be turned off at runtime, which gets us back to the same issue of what method to use. I’ve been thinking that it might make sense to define a serial protocol for the REPL, but that kind of requires an IDE that’s specifically built for uLisp, as has been discussed before. In particular, it’s not friendly to the Arduino monitor.

#5

The main use of this will be to provide uLisp listings that people can safely copy and paste in without having to work around buffer problems. For example, the uLisp assemblers are currently too long for most buffer settings.

So there’s no need for an actual Lisp command to turn echo off. What we need is a directive that can be put in listings.

Here are a couple of suggestions:

  • A reader macro such as:

    #-echo

    that you put at the start of the listing, and another one, such as:

    #+echo

    you put at the end.

  • A special format of comment, such as with three semicolons, that can be used at the start of a listing which will have the side effect of turning echo off, such as:

    ;;; Infinite precision arithmetic

    and another one to turn it back on at the end:

    ;;; End of Infinite precision arithmetic

It’s a pity to have to have the matching directive at the end to restore normality. As mentioned earlier, I could perhaps do this automatically after a one second delay.

#6

Reader macros seem like a good idea. I’m wary of using #+ and #-, since that would add an incompatibility with CL, but I definitely see the attraction. It might be possible to use something like #;echo+ and #;echo-.

I think you need to at least have the ability to flip modes manually, even if it’s done automatically by default.

#7

I like #;echo+ and #;echo-. I’ll see how that works out.

#8

Is the problem really AT ALL the echoing? - Or is it rather that “the microcontroller operates while input is coming in, thus missing a part of the input”?

If the latter, I would rather suggest that you do something like “(buffer-on)” / “(buffer-off)”, i.e. where you can turn uLisp into some sort of “character acceptance-mode”, which is signalled in some form to begin, and if this beginning is noted, then:

collect all following characters until either (i) some maximum capacity per platform is reached, or (ii) some termination sequence is encountered.

Once the characters have been collected, they can be “spooled out” to the “evaluator” from the collection buffer. This way, no matter HOW long an evaluation will take, you will have “caught” the entire input.

Just an idea, but this is how I would do it (and how I DO do it when I use a wireless-to-serial bridge in order to get text for an old typewriter which I use as a printer).

#9

And if you were to be curious about the sequences, I would propose something “self-evaluating”, e.g.:

'on-buf

'of-buf

  • i.e. you might check for VERY unlikely (nonsensical) sequences or symbols, whose “lack of evaluation” in uLisp would be of no consequence, and if they “define nothing at all”, then they can be copy-pasted in “normal” Lisp code without having any effect whatsoever.
#10

Yes, it’s definitely the echoing. uLisp can process the incoming input fast enough not to cause problems. What causes problems is the extra delay of echoing each line, which is at the same relatively slow baud rate. If I turn echoing off I don’t get buffer errors, even with default small buffers (eg 64 bytes).

I think I’ve found the ideal solution. A comment turns off echo, and it is turned back on automatically after a delay of one second with no input. This means that you’ll be able to copy and paste all existing listings without any change (assuming they’ve got a comment at the start, which most have). You can’t actually type a comment into the REPL - it hangs up - so we’re not losing a useful feature with this.

If anyone would like to try this out I’ll make a patched version of 3.5 available.

#11

Wait, what?

#12

Well not exactly hangs up; it’s waiting for an opening and then closing bracket.

#13

Ah, okay. That makes more sense.

Which reminds me to ask: I presume you’re talking specifically about the echoing of characters being input, rather than disabling output in general. Is that a correct assumption?

#14

Yes, you’ll still see all the output from uLisp, so the name of each function that you’ve evaluated, and any errors that occur.

#15

(I, for one, am EXTREMELY happy about the LispLibrary possibility, i.e. to just “flash uLisp together with the program”: that, in practice, really solved the problem for me, “how to get a longer program onto a uLisp environment”. Hence, while it is great you implement your suggestion, in reality, you have already solved the issue for like 99% of the cases.)

#16

It’s useful, but doesn’t cover all bases - in particular, you need to have the code ready when you flash the program, so can be problematic when you’re trying to develop code live.

On the flipside of the coin, there are contexts where not having uLisp do echoing at all would be entirely fine - any ‘proper’ IDE for it, for instance, would presumably use local echoing and line editing buffers, rather than having uLisp handle that.

#17

An automatic solution to serial buffer problems, that doesn’t require any action on the part of the user, is now implemented in uLisp 3.6 on all platforms.

#18

Hi David

Lisp Badge LE + uLisp 4.4

(read-line), (read-byte) return immediately without read any keys from hard key pad, how can I read keys from key pad?

(defun test-read ()
   (let ((key (read-line)))
   (format t "~a~%" key)))

1942> 
(test-read)
nil

I’m not sure if it relate to the serial buffer management, need to investigate further.

Best regards
Walter

#19

Hi Walter,

Your function calling (read-line) returns immediately because it thinks it has read a blank line. You can get it to work by doing:

(loop (print (read-line)))

If you want to read individual keys there are the extensions keyboard and check-key built into the Lisp Badge LE to allow you to read the keyboard; see Lisp Badge LE extensions.

#20

Hi David

With uLisp4.4a on the Lisp Badge LE, I still can’t past a large chunk of code into the serial.

How I opened the serial:
tio -o 3 -b 9600 /dev/serial/by-id/usb-FTDI_FT232R_USB_UART_AK066ZFL-if00-port0

Code chunk:

(defun sq (x) (* x x))

(defun led-off nil
  (digitalwrite :led-builtin nil))

(defun led-on nil
  (pinmode :led-builtin t)
  (digitalwrite :led-builtin t))

(defun clrd nil
  (mapc makunbound (globals)))

(defun lga nil
  (let nil
    (princ
      (nth
        (random (length (globals)))
        (globals))))
  (delay 10)
  (lga))

The output when I past them to the serial:

1799> (defun sq (x) (* x x))
sq

  (digitalwrite :led-builtin nil))
led-off

(defun t)
Error: 'defun' too few arguments
1799> (delay 10)
10

1799>

Another try with out the -o 3 (delay 3ms for each char)

tio -b 9600 /dev/serial/by-id/usb-FTDI_FT232R_USB_UART_AK066ZFL-if00-port0

tio -b 9600 /dev/serial/by-id/usb-FTDI_FT232R_USB_UART_AK066ZFL-if00-port0 
[15:02:17.628] tio v2.8
[15:02:17.628] Press ctrl-y q to quit
[15:02:17.630] Connected
uLisp 4.4a 
1794> (defun sq (x) (* x x))
sq

  (digitalwrite :led-builtin nil))
led-off

  (digitalwrite :led-builtin t))
led-on

  (mapc makunbound (globals)))
clrd

(defun lga nindom (length (globals)))
lga

1809>         (globals)
(sq led-off led-on clrd lga blink flash)

1809> )
Error: incomplete list

Is there any information I missed?

Thanks
Walter