Working around serial buffer overflow with Emacs


#4

Thanks, I actually already tried Super-Tyson on Linux as I seems to understand all the dependencies are available there. It builds fine on Mint but aborts with a core dump when I launch the program:

SuperTyson: /home/paolo/apps/super-tyson/src/main.cpp:10: int main(int, char**): Assertion `argc == 2' failed.
Aborted (core dumped)

Do I need to pass an argument? The README doesn’t mention any but perhaps I need to pass the serial device.

As for reusing code, I’m not fluent enough in C.


#5

Despite some other attempts Emacs still displays extra ^Ms in the inferior Lisp and sending expressions longer than the serial buffer hangs the session.

Emacs is not much usable with uLisp as is. Aside from the Serial Monitor, terminal emulators don’t support line editing. In addition, sending code to uLisp requires copypasting which introduces friction and potential mistakes. This limited environment doesn’t scale to typical medium-large Lisp code beyond short expressions one or two lines long.

The Cardputer keyboard is not an option for anything longer than a line. I have no practical way of interacting with uLisp over serial and sending code longer than the serial buffer. I’m stumped.


#6

I don’t know if you’re aware, but the serial buffer overflow problem is probably not anything to do with Emacs. The ESP32-S3 has native USB and it does the serial port via software, and currently it has a particularly nasty bug where if the receive buffer fills up too fast the ESP32-S3 will crash and disconnect. This also isn’t a problem with uLisp either — it’s with the Arduino core.


#7

sending code longer than the serial buffer

What serial buffer are you referring to?

To enter programs into uLisp I do:

  • Select a whole Lisp listing of several pages, such as the Mini text adventure game.

  • Do Select All, Copy, then Paste it into the Serial Monitor input field, and press Return.

This works reliably for me.

What microcontroller board are you using? There are problems with the serial interface software on the Arduino Core for some boards.


#8

@dragoncoder047 Yes I’m aware.

@johnsondavies I’m referring to the serial buffer related to the USB issue @dragoncoder047 explained. I assumed the Serial Monitor was somehow as limited as other terminal emulators but I’ll check it out in more depth with my M5Stock Cardputer.

Thanks both!


#9

Note that the Cardputer is based on the ESP32-S3 which I don’t recommend for use with uLisp because of an issue with the serial interface implementation:

As discussed on the page ESP32-S2, ESP-S3, and ESP-C3 boards, there is an outstanding issue with the ESP32-S3 built-in USB support that makes it hang up if you try to enter a program of more than a few lines from the Serial Monitor on a Mac computer.

Do you have another board you can try, such as the Raspberry Pi Pico or Pico 2?


#10

I read about that ESP32-S3 USB issue but thought it applied only to Macs whereas I use a Linux PC. The only board I have is a Cardputer.


#11

Here is the Serial Monitor connected to the Cardputer after copy-pasting the full source of the mini text adventure game and evaluating (adventure). I later evaluated two more expressions, 'abc and 123. Nothing happens, the Serial Monitor just echoes to the output pane any additional input.

The same occurs with input shorter than the adventure game source but longer than the serial buffer. It looks like the ESP32-S3 USB issue affects Linux PCs too, not just Macs.


#12

To test whether the issue is there you could try my MRE at the start of this post:

Native USB CDC hangs up on ESP32-S2 after about 320 characters

It affects the ESP32-S3 too.


#13

By the way, I believe that this issue only affects the use of the built-in USB support (CDC) in the ESP32-S2, ESP32-C3, or ESP32-S3.

If you used one of the widely available USB to Serial cables connected to the appropriate Rx and Tx ports on the StampS3 module, I believe the problem won’t occur (although I haven’t tried it).

For example, I had no problems using the ESP32-S3-DevKitM-1 with the UART connection; see:

ESP32-S2, ESP-S3, and ESP-C3 boards - ESP32-S3-DevKitM-1


#14

To test for the issue I need to flash the sketch in this post, right? At this point there are little doubts though.

I’m connecting a cable to the built-in USB-C port of the Cardputer, i.e. the one of the S3 on the right side of the Cardputer. As a software guy I’m not much familiar with hardware or assembling electronic components, so I’m not sure where else the UART connection is and what cable it requires.


#15

To test for the issue I need to flash the sketch in this post, right?

Yes, and then perform the test described there from the Serial Monitor.

I’m not sure where else the UART connection is and what cable it requires.

The cable would be something like this:

FTDI Serial TTL-232 USB Cable

I assume it would connect to Rx, Tx, and GND on the StampS3 module; perhaps someone else can give more help about this.


#16

I uploaded the sketch and performed the test. As expected the Cardputer is affected by the issue as the output of the test is:

64
128
192
256
320

Bummer. I’ll probably wait for Arduino to fix the issue but, since it has apparently been known for a couple of years, I’m not holding my breath.

Thanks anyway.


#17

I subscribed to the ESP32 USB issue, let’s hope it eventually goes forward.


#18

I subscribed to the ESP32 USB issue , let’s hope it eventually goes forward.

You could post there that you’ve tested my test program on a Linux PC and you can confirm that the issue happens there too, making Serial unusable on an ESP32-S3.

It might help …


#19

Done thanks.


#20

A user posted some code to the arduino-esp32 issue but I’m not sure whether it’s a fix or just a test case.


#21

A more general emacsy question, for setting up slime.
The info i’m seeing is that .emacs is not the best anymore and instead, on linux to put it here -
M-x config/emacs/init.el

When i do that and restart emacs it doesn’t ‘work’ / know where to look. Previously i had a .emacs in home.

Should i just revert to this or should i uses the config/emacs/init.el? how do i make that work?
Thanks


#22

This is my current workaround for sending an expression or region to an ESP32S3 from Emacs using a serial-term. I split the input into lines and send one line at a time with a delay in between to keep the ESP32’s serial buffer from overflowing. It may be inelegant, but it seems to work. I can send entire long ulisp buffers without hanging the serial port. (Tested with a LilyGo T-deck at 9600 baud. I reckon the delay could be reduced if you increase the baud rate.)

I tried doing a similar thing with an inferior lisp/comint buffer, but that required overriding comint-send-string and comint-send-region (which I tried to do with advice-add), and I ran into problems with weird double-echoing and ^M’s in the screen output, which my casual elisp knowledge has so far proven unable to solve.

;; based on ulisp forum posts by j-keck Nov. 18 in "Emacs for programming"

(defvar ulisp-term-port "/dev/ttyACM0")
(defvar ulisp-term-speed 9600)

;; send a string to the ulisp port.
;; send a line at a time, and delay after each to keep ESP32S3 serial buffer from hanging 
(defun ulisp-send-string (s)
  (with-current-buffer ulisp-term-port
     (end-of-buffer)
     (let ((lines (split-string s "\n")))
      (dolist (l lines)
        (insert l)
        (term-send-input)
        (sleep-for 0.2)))))

(defun ulisp-eval-last-expression-in-term ()
 (interactive)
 (let ((expr (buffer-substring-no-properties  
                    (save-excursion (backward-sexp) (point))
                    (point))))
   (ulisp-send-string expr)))

(defun ulisp-eval-region-in-term ()
 (interactive)
 (let ((expr (buffer-substring-no-properties  
                    (region-beginning)
                    (region-end))))
   (ulisp-send-string expr)))

(defun ulisp-connect-term ()
 (interactive)
 (split-window-below)
 (other-window 1)
 (serial-term ulisp-term-port ulisp-term-speed)
 (term-line-mode)
 (other-window 1))

(defun ulisp-disconnect-term ()
  (interactive)
  (pop-to-buffer ulisp-term-port)
  (delete-window)
  (kill-buffer ulisp-term-port))

(global-set-key (kbd "C-c e") 'ulisp-eval-last-expression-in-term)
(global-set-key (kbd "C-c r") 'ulisp-eval-region-in-term)
(global-set-key (kbd "C-c c") 'ulisp-connect-term)
(global-set-key (kbd "C-c d") 'ulisp-disconnect-term)

#23

Putting your initialization code in .emacs in your home directory still works, it’s just not as general. .config/emacs/init.el follows the latest convention for where to put init files. The advantage is that you can have multiple initialization files (to be included by init.el) all kept together in one place. But if all your init code is in one file, .emacs will work just fine.