Working around serial buffer overflow with Emacs


#8

@dragoncoder047 Yes I’m aware.

@johnsondavies I’m referring to the serial buffer related to the USB issue @dragoncoder047 explained. I assumed the Serial Monitor was somehow as limited as other terminal emulators but I’ll check it out in more depth with my M5Stock Cardputer.

Thanks both!


#9

Note that the Cardputer is based on the ESP32-S3 which I don’t recommend for use with uLisp because of an issue with the serial interface implementation:

As discussed on the page ESP32-S2, ESP-S3, and ESP-C3 boards, there is an outstanding issue with the ESP32-S3 built-in USB support that makes it hang up if you try to enter a program of more than a few lines from the Serial Monitor on a Mac computer.

Do you have another board you can try, such as the Raspberry Pi Pico or Pico 2?


#10

I read about that ESP32-S3 USB issue but thought it applied only to Macs whereas I use a Linux PC. The only board I have is a Cardputer.


#11

Here is the Serial Monitor connected to the Cardputer after copy-pasting the full source of the mini text adventure game and evaluating (adventure). I later evaluated two more expressions, 'abc and 123. Nothing happens, the Serial Monitor just echoes to the output pane any additional input.

The same occurs with input shorter than the adventure game source but longer than the serial buffer. It looks like the ESP32-S3 USB issue affects Linux PCs too, not just Macs.


#12

To test whether the issue is there you could try my MRE at the start of this post:

Native USB CDC hangs up on ESP32-S2 after about 320 characters

It affects the ESP32-S3 too.


#13

By the way, I believe that this issue only affects the use of the built-in USB support (CDC) in the ESP32-S2, ESP32-C3, or ESP32-S3.

If you used one of the widely available USB to Serial cables connected to the appropriate Rx and Tx ports on the StampS3 module, I believe the problem won’t occur (although I haven’t tried it).

For example, I had no problems using the ESP32-S3-DevKitM-1 with the UART connection; see:

ESP32-S2, ESP-S3, and ESP-C3 boards - ESP32-S3-DevKitM-1


#14

To test for the issue I need to flash the sketch in this post, right? At this point there are little doubts though.

I’m connecting a cable to the built-in USB-C port of the Cardputer, i.e. the one of the S3 on the right side of the Cardputer. As a software guy I’m not much familiar with hardware or assembling electronic components, so I’m not sure where else the UART connection is and what cable it requires.


#15

To test for the issue I need to flash the sketch in this post, right?

Yes, and then perform the test described there from the Serial Monitor.

I’m not sure where else the UART connection is and what cable it requires.

The cable would be something like this:

FTDI Serial TTL-232 USB Cable

I assume it would connect to Rx, Tx, and GND on the StampS3 module; perhaps someone else can give more help about this.


#16

I uploaded the sketch and performed the test. As expected the Cardputer is affected by the issue as the output of the test is:

64
128
192
256
320

Bummer. I’ll probably wait for Arduino to fix the issue but, since it has apparently been known for a couple of years, I’m not holding my breath.

Thanks anyway.


#17

I subscribed to the ESP32 USB issue, let’s hope it eventually goes forward.


#18

I subscribed to the ESP32 USB issue , let’s hope it eventually goes forward.

You could post there that you’ve tested my test program on a Linux PC and you can confirm that the issue happens there too, making Serial unusable on an ESP32-S3.

It might help …


#19

Done thanks.


#20

A user posted some code to the arduino-esp32 issue but I’m not sure whether it’s a fix or just a test case.


#21

A more general emacsy question, for setting up slime.
The info i’m seeing is that .emacs is not the best anymore and instead, on linux to put it here -
M-x config/emacs/init.el

When i do that and restart emacs it doesn’t ‘work’ / know where to look. Previously i had a .emacs in home.

Should i just revert to this or should i uses the config/emacs/init.el? how do i make that work?
Thanks


#22

This is my current workaround for sending an expression or region to an ESP32S3 from Emacs using a serial-term. I split the input into lines and send one line at a time with a delay in between to keep the ESP32’s serial buffer from overflowing. It may be inelegant, but it seems to work. I can send entire long ulisp buffers without hanging the serial port. (Tested with a LilyGo T-deck at 9600 baud. I reckon the delay could be reduced if you increase the baud rate.)

I tried doing a similar thing with an inferior lisp/comint buffer, but that required overriding comint-send-string and comint-send-region (which I tried to do with advice-add), and I ran into problems with weird double-echoing and ^M’s in the screen output, which my casual elisp knowledge has so far proven unable to solve.

;; based on ulisp forum posts by j-keck Nov. 18 in "Emacs for programming"

(defvar ulisp-term-port "/dev/ttyACM0")
(defvar ulisp-term-speed 9600)

;; send a string to the ulisp port.
;; send a line at a time, and delay after each to keep ESP32S3 serial buffer from hanging 
(defun ulisp-send-string (s)
  (with-current-buffer ulisp-term-port
     (end-of-buffer)
     (let ((lines (split-string s "\n")))
      (dolist (l lines)
        (insert l)
        (term-send-input)
        (sleep-for 0.2)))))

(defun ulisp-eval-last-expression-in-term ()
 (interactive)
 (let ((expr (buffer-substring-no-properties  
                    (save-excursion (backward-sexp) (point))
                    (point))))
   (ulisp-send-string expr)))

(defun ulisp-eval-region-in-term ()
 (interactive)
 (let ((expr (buffer-substring-no-properties  
                    (region-beginning)
                    (region-end))))
   (ulisp-send-string expr)))

(defun ulisp-connect-term ()
 (interactive)
 (split-window-below)
 (other-window 1)
 (serial-term ulisp-term-port ulisp-term-speed)
 (term-line-mode)
 (other-window 1))

(defun ulisp-disconnect-term ()
  (interactive)
  (pop-to-buffer ulisp-term-port)
  (delete-window)
  (kill-buffer ulisp-term-port))

(global-set-key (kbd "C-c e") 'ulisp-eval-last-expression-in-term)
(global-set-key (kbd "C-c r") 'ulisp-eval-region-in-term)
(global-set-key (kbd "C-c c") 'ulisp-connect-term)
(global-set-key (kbd "C-c d") 'ulisp-disconnect-term)

#23

Putting your initialization code in .emacs in your home directory still works, it’s just not as general. .config/emacs/init.el follows the latest convention for where to put init files. The advantage is that you can have multiple initialization files (to be included by init.el) all kept together in one place. But if all your init code is in one file, .emacs will work just fine.


#24

@dennisd Thanks, your code works well. I really appreciate it as it makes uLisp usable on the Cardputer.

I wonder whether it would be possible to turn off echoing of the expressions when sending them to uLisp. According to the Emacs manual, echoing in the terminal emulator is the responsibility of the subshell.


#25

I added (setq-local comint-process-echoes nil) at the beginning of ulisp-send-string and (setq-local comint-process-echoes t) at the end but the code sent to uLisp is still echoed in the terminal buffer.


#26

@amoroso, the serial-term isn’t a comint buffer (it’s more primitive), so the comint commands won’t do anything. I tried adding delays to the comint code, but I haven’t figured out how to get that working yet.

Per the “; (semicolon)” entry in the Ulisp Language Reference,

A comment line turns off echo, so that by starting a long listing with a comment 
it can safely be pasted in to the Arduino IDE's  **Serial Monitor**  without
overflowing the serial buffer. Echo is turned back on automatically after 
a one second delay if there is no activity on the serial input.

So you could try putting a ; line above your code and including it in the region to be sent. But I just tried it and I had weird results in the serial-term, I think because the (insert l) function actually inserts the text into the serial-term buffer, so Emacs may be trying to display it itself every time it receives characters from the ulisp side (as it will for every definition, where ulisp will print out the symbol defined and a prompt). So we may have to live with the echo for now.


#27

Since we don’t yet know how to turn off echo, I tried to at least make the sent code prettier in the terminal rather than spreading it out all over the buffer.

I modified ulisp-send-string to first insert a newline to align the input to the beginning of the line, restore the original newline when sending each line, and terminate the input with a final newline. Sending t is supposed to consume a dummy expression and align the prompt.

I experimented with various alternatives but things don’t improve much.

(defun ulisp-send-string (s)
  (with-current-buffer ulisp-term-port
    (end-of-buffer)
    (insert "t" ?\r ?\n)
    (let ((lines (split-string s "\n")))
      (dolist (l lines)
        (insert l ?\r ?\n)
        (term-send-input)
        (sleep-for 0.2)))
    (insert "t" ?\r ?\n)))