Proposed solution to serial buffer problems

#3

it seems to me that it would still be possible to overflow buffers when sending multiple top-level forms, if uLisp takes long enough to execute one of them.

In general yes, but the most important case to solve is when the forms are either defun or defvar.

it seems almost wasteful to add a symbol to control this

Yes, I agree. I’ve tried to think about whether it would be possible to do it automatically. Some ideas:

  • Turn off echo if a comment is encountered, since you don’t normally type comments in the REPL.

  • Turn echo back on automatically if there is no activity on the serial port for some time interval, such as a second.

#4

Absolutely. I’m thinking about stuff like the tests, which I still haven’t automated properly in my preferred working environment.

That might work, but could cause other issues if it can’t be turned off at runtime, which gets us back to the same issue of what method to use. I’ve been thinking that it might make sense to define a serial protocol for the REPL, but that kind of requires an IDE that’s specifically built for uLisp, as has been discussed before. In particular, it’s not friendly to the Arduino monitor.

#5

The main use of this will be to provide uLisp listings that people can safely copy and paste in without having to work around buffer problems. For example, the uLisp assemblers are currently too long for most buffer settings.

So there’s no need for an actual Lisp command to turn echo off. What we need is a directive that can be put in listings.

Here are a couple of suggestions:

  • A reader macro such as:

    #-echo

    that you put at the start of the listing, and another one, such as:

    #+echo

    you put at the end.

  • A special format of comment, such as with three semicolons, that can be used at the start of a listing which will have the side effect of turning echo off, such as:

    ;;; Infinite precision arithmetic

    and another one to turn it back on at the end:

    ;;; End of Infinite precision arithmetic

It’s a pity to have to have the matching directive at the end to restore normality. As mentioned earlier, I could perhaps do this automatically after a one second delay.

#6

Reader macros seem like a good idea. I’m wary of using #+ and #-, since that would add an incompatibility with CL, but I definitely see the attraction. It might be possible to use something like #;echo+ and #;echo-.

I think you need to at least have the ability to flip modes manually, even if it’s done automatically by default.

#7

I like #;echo+ and #;echo-. I’ll see how that works out.

#8

Is the problem really AT ALL the echoing? - Or is it rather that “the microcontroller operates while input is coming in, thus missing a part of the input”?

If the latter, I would rather suggest that you do something like “(buffer-on)” / “(buffer-off)”, i.e. where you can turn uLisp into some sort of “character acceptance-mode”, which is signalled in some form to begin, and if this beginning is noted, then:

collect all following characters until either (i) some maximum capacity per platform is reached, or (ii) some termination sequence is encountered.

Once the characters have been collected, they can be “spooled out” to the “evaluator” from the collection buffer. This way, no matter HOW long an evaluation will take, you will have “caught” the entire input.

Just an idea, but this is how I would do it (and how I DO do it when I use a wireless-to-serial bridge in order to get text for an old typewriter which I use as a printer).

#9

And if you were to be curious about the sequences, I would propose something “self-evaluating”, e.g.:

'on-buf

'of-buf

  • i.e. you might check for VERY unlikely (nonsensical) sequences or symbols, whose “lack of evaluation” in uLisp would be of no consequence, and if they “define nothing at all”, then they can be copy-pasted in “normal” Lisp code without having any effect whatsoever.
#10

Yes, it’s definitely the echoing. uLisp can process the incoming input fast enough not to cause problems. What causes problems is the extra delay of echoing each line, which is at the same relatively slow baud rate. If I turn echoing off I don’t get buffer errors, even with default small buffers (eg 64 bytes).

I think I’ve found the ideal solution. A comment turns off echo, and it is turned back on automatically after a delay of one second with no input. This means that you’ll be able to copy and paste all existing listings without any change (assuming they’ve got a comment at the start, which most have). You can’t actually type a comment into the REPL - it hangs up - so we’re not losing a useful feature with this.

If anyone would like to try this out I’ll make a patched version of 3.5 available.

#11

Wait, what?

#12

Well not exactly hangs up; it’s waiting for an opening and then closing bracket.

#13

Ah, okay. That makes more sense.

Which reminds me to ask: I presume you’re talking specifically about the echoing of characters being input, rather than disabling output in general. Is that a correct assumption?

#14

Yes, you’ll still see all the output from uLisp, so the name of each function that you’ve evaluated, and any errors that occur.

#15

(I, for one, am EXTREMELY happy about the LispLibrary possibility, i.e. to just “flash uLisp together with the program”: that, in practice, really solved the problem for me, “how to get a longer program onto a uLisp environment”. Hence, while it is great you implement your suggestion, in reality, you have already solved the issue for like 99% of the cases.)

#16

It’s useful, but doesn’t cover all bases - in particular, you need to have the code ready when you flash the program, so can be problematic when you’re trying to develop code live.

On the flipside of the coin, there are contexts where not having uLisp do echoing at all would be entirely fine - any ‘proper’ IDE for it, for instance, would presumably use local echoing and line editing buffers, rather than having uLisp handle that.

#17

An automatic solution to serial buffer problems, that doesn’t require any action on the part of the user, is now implemented in uLisp 3.6 on all platforms.

#18

Hi David

Lisp Badge LE + uLisp 4.4

(read-line), (read-byte) return immediately without read any keys from hard key pad, how can I read keys from key pad?

(defun test-read ()
   (let ((key (read-line)))
   (format t "~a~%" key)))

1942> 
(test-read)
nil

I’m not sure if it relate to the serial buffer management, need to investigate further.

Best regards
Walter

#19

Hi Walter,

Your function calling (read-line) returns immediately because it thinks it has read a blank line. You can get it to work by doing:

(loop (print (read-line)))

If you want to read individual keys there are the extensions keyboard and check-key built into the Lisp Badge LE to allow you to read the keyboard; see Lisp Badge LE extensions.

#20

Hi David

With uLisp4.4a on the Lisp Badge LE, I still can’t past a large chunk of code into the serial.

How I opened the serial:
tio -o 3 -b 9600 /dev/serial/by-id/usb-FTDI_FT232R_USB_UART_AK066ZFL-if00-port0

Code chunk:

(defun sq (x) (* x x))

(defun led-off nil
  (digitalwrite :led-builtin nil))

(defun led-on nil
  (pinmode :led-builtin t)
  (digitalwrite :led-builtin t))

(defun clrd nil
  (mapc makunbound (globals)))

(defun lga nil
  (let nil
    (princ
      (nth
        (random (length (globals)))
        (globals))))
  (delay 10)
  (lga))

The output when I past them to the serial:

1799> (defun sq (x) (* x x))
sq

  (digitalwrite :led-builtin nil))
led-off

(defun t)
Error: 'defun' too few arguments
1799> (delay 10)
10

1799>

Another try with out the -o 3 (delay 3ms for each char)

tio -b 9600 /dev/serial/by-id/usb-FTDI_FT232R_USB_UART_AK066ZFL-if00-port0

tio -b 9600 /dev/serial/by-id/usb-FTDI_FT232R_USB_UART_AK066ZFL-if00-port0 
[15:02:17.628] tio v2.8
[15:02:17.628] Press ctrl-y q to quit
[15:02:17.630] Connected
uLisp 4.4a 
1794> (defun sq (x) (* x x))
sq

  (digitalwrite :led-builtin nil))
led-off

  (digitalwrite :led-builtin t))
led-on

  (mapc makunbound (globals)))
clrd

(defun lga nindom (length (globals)))
lga

1809>         (globals)
(sq led-off led-on clrd lga blink flash)

1809> )
Error: incomplete list

Is there any information I missed?

Thanks
Walter

#21

The problem is that the Arduino serial interface, that uLisp relies on, doesn’t implement flow control. So when you paste in a long listing, the information echoed to the terminal using serial write causes the serial read to miss characters sent on the stream.

As a solution, the semicolon character turns off echo to the terminal; this is explained here:

http://www.ulisp.com/show?3L#semicolon

A comment line turns off echo so that by starting a long listing with a comment, it can safely be pasted in to the Arduino IDE’s Serial Monitor without overflowing the serial buffer. Echo is turned back on automatically after a one second delay if there is no activity on the serial input.

Please try pasting in a listing starting with a comment line, prefixed by a semicolon, and I think that should solve the problem. For example:

; Test program

(defun sq (x) (* x x))

(defun led-off nil
  (digitalwrite :led-builtin nil))

(defun led-on nil
  (pinmode :led-builtin t)
  (digitalwrite :led-builtin t))
Working around serial buffer overflow with Emacs
#22

Thanks David, It works perfectly.