Seek and position in with-sd-card


#1

I have the need to open a file on an sd-card and seek to a specific position and read 128 bytes of data. Then possibly seek ahead to a different position on the file.

I see that the SD.h library has the file.seek() and file.position() functionality available. I’m curious if we could use the unused address parameter of the with-sd-card stream to achieve this functionality, or if there is a better way to add file-position in ulisp, such as a function that could work on the stream itself to set the position.


#2

I think you’d have to provide them as two new functions that take the SD card stream as a parameter.


#3

(file-position stream) and (file-position stream position) would be how it’s done in Common Lisp, so that sounds reasonable. I’m not sure how to get access to the SDgfile from a passed stream, and I’d rather not repeatedly flush the buffer to get to the desired position.

I guess since SDgfile is a global variable, one could just run SDgfile.seek(position) directly. Perhaps after checking that the stream is the correct type, and that the position given is less than the size of the file that is currently open. Something like the following:

object *fn_file_position(object *args, object *env) {
  (void) env;
  #if defined(sdcardsupport)
  if (isstream(first(args))>>8 != SDSTREAM) error2("invalid SD file stream");
  if (second(args) != NULL) {
    unsigned long pos = (unsigned long)(checkinteger(second(args)));
    if (pos <= SDgfile.size()) {
       SDgfile.seek(pos);  //TODO: check for write mode, use SDpfile instead
       return number(pos);
    }
  } else {
    return number(SDgfile.position()); //TODO: check for write mode?
    }
  #endif
  return nil;
}

const char string_file_position[] PROGMEM = "file-position";
//...
const char doc_file_position[] PROGMEM = "(file-position stream)\n (file-position stream position)\n Sets with-sd-card stream's file position, or returns it current value";
//...
{ string_file_position, fn_file_position, 0212, doc_file_position },

Unfortunately the above code fails to return a file position when no second argument is given…and doesn’t always work when you give a position. I think I’m messing up the passing of position to and from lisp objects, as it’s expecting unsigned longs.

If you have any suggestions, I’d love to hear them. Otherwise I’ll try and work on this more when I have time.


#4

OK, I’ll be happy to have a look at your code some time in the next day or so.


#5

PS What platform are you working on? I’ll try and test it on the same platform.


#6

I’m using the T-Deck currently.

I believe that the problem has to do with my lack of knowledge at how to pass unsigned long position variables to and from the uLisp form and the file.seek() and file.position() C++ methods. I’m currently using checkinteger() and number() to do these transformations, but I believe it’s reducing the numbers to fewer bytes.

An example of how I’m testing:
(with-sd-card (str "test.txt" 1 ) (dotimes (x 100) (print x str)))

Then try and seek through the file:
(with-sd-card (str "test.txt") (dotimes (x 100) (print x)(file-position str x)(princ (read-line str)))) <-- this works, but I may need to have a better idea of how to pass the file position.

(with-sd-card (str "test.txt") (dotimes (x 10) (princ (file-position str)) (princ (read-line str)))) <-- this fails horribly.


#7

I think the only small error in your function is the test for a second argument. In C the test:

if (second(args) != NULL) {

isn’t safe because, if there isn’t a second argument, it’s looking beyond the extent of the list into unknown territory. Your function would work provided you called it with:

(file-position str nil)

to read the file position. The correct test is:

  if (cdr(args) != NULL) { // Second argument
    SDgfile.seek(checkinteger(second(args)));
    return second(args);
  } else {
    return number(SDgfile.position());
  }

Also, your code doesn’t seem to do anything if you try to seek beyond the end of the file. I did a few tests and SDgfile.seek() seems quite happy to position the file pointer beyond the end of the file, so I’m not sure what the correct action should be. In any case Lisp should return something.

Regards, David


#8

YES! That seems to fix the problem! Thanks so very much.

I am doing a check to see that the position given is smaller than the total file size before doing the seek:

unsigned long pos = (unsigned long)(checkinteger(second(args)));
    if (pos <= SDgfile.size()) {

I’ll do some checks to see if these are in the same units. I seem to remember seeing code that used file.seek(file.filesize()) to get to the end of the file in the past on other projects.

Thanks again for your help.


#9

My point was that your fn_file_position() function doesn’t return anything if the pos <= SDgfile.size() test fails. All uLisp functions need to return a value.


#10

Huh, I assumed that the final return nil; at the end of the function would catch that condition.


#11

Yes, sorry, I missed that.


#12

@johnsondavies Is there any way to determine the mode that a with-sd-card form is being used (READ, WRITE, APPEND), so that one could use the .seek() functionality for both WRITE mode along with READ, ie switch between using SDgfile and SDpfile. Is there a means of determining this from the stream variable?

I don’t currently have this need, but I could see it being useful in the future. Thanks again for all your help.


#13

In uLisp a new stream is created with stream(streamtype, address), where address is an 8-bit address to allow you to have multiple streams of the same type, such as I2C streams where address is the I2C address.

Currently with SD streams the address is always 1, but the address could be set to the mode (READ, APPEND, WRITE) in sp_withsdcard(), which would provide the information you need. I don’t believe this would affect any other aspects of the SD card interface.


#14

I finally got some time to look at this again yesterday, here is what I ended up with, for those following along that want this functionality:

/* 
  (file-position stream) returns current file postition
  (file-position stream position) seeks to position given
*/

object *fn_file_position(object *args, object *env) {
  (void) env;
  #if defined(sdcardsupport)
  int stream = isstream(first(args));
  if (stream>>8 != SDSTREAM) error2("not an SD file stream");
  int address = stream & 0xFF;
  File *file = (address >= 1) ? &SDpfile : &SDgfile;
  if (cdr(args) != NULL) {
    unsigned long pos = (unsigned long)(checkinteger(second(args)));
    if (pos <= file->size()) {
       file->seek(pos);
       return number(pos);
    }
  } else {
    if (*file) {
      unsigned long position = file->position();
      return number(position);
    } else { error2("problem reading file position");}
  }
  #endif
  return nil;
}

This will work for both read and write, although it requires some updates to the with-sd-card C function to pass along the stream’s mode.

In the main ino file (i.e. ulisp-tdeck.ino), function sp_withsdcard change the address of 1 to mode, which has already been set.

object *pair = cons(var, stream(SDSTREAM, mode));

Unfortunately, the write modes supplied don’t allow for seeking to a specific position and editing data. They all append to the files last position, blocking the seek functionality. One leaves the existing data, appending only to the end; and the other mode clears the whole file, appending to the beginning. For the esp32, the modes are posix compliant (https://pubs.opengroup.org/onlinepubs/009695399/functions/fopen.html), a slight deviation from other arduino file operations, I believe.

There is a third option r+, that allows one to read and write to the file where the seek functionality is not blocked.

Going forward, I’m not sure if sticking to the enumerated integer values for file modes makes sense, and add 3, or to just pass the string values themselves, in which the options would be:

"r" Open file for reading.
"w" Truncate to zero length or create file for writing.
"a" Append; open or create file for writing at end-of-file.
"r+" Open file for update (reading and writing).
"w+" Truncate to zero length or create file for update.
"a+" Append; open or create file for update, writing at end-of-file.

Some of these seem redundant. I’m not sure what the use cases for a+ and w+ differ from a and w. I wish they’d used u instead of r+.

In this case the ulisp would look like:

(with-sd-card (str "test.txt" "r+")
   (file-position str 35)
   (write-string "new text" str))

#15

Here is the updated C function that you can pass along current enumerated modes, including 3:

object *sp_withsdcard (object *args, object *env) {
  #if defined(sdcardsupport)
  object *params = checkarguments(args, 2, 3);
  object *var = first(params);
  params = cdr(params);
  if (params == NULL) error2("no filename specified");
  builtin_t temp = Context;
  object *filename = eval(first(params), env);
  Context = temp;
  if (!stringp(filename)) error("filename is not a string", filename);
  params = cdr(params);
  SDBegin();
  int mode = 0;
  if (params != NULL && first(params) != NULL) mode = checkinteger(first(params));
  const char *oflag = FILE_READ;
  switch (mode) {
    case 0:
      oflag = FILE_READ; //"r"
      break;
    case 1:
      oflag = FILE_APPEND; //"a"
      break;
    case 2:
      oflag = FILE_WRITE; //"w"
      break;
    case 3:
      oflag = "r+";
      break;
  }
  if (mode >= 1) {
    char buffer[BUFFERSIZE];
    SDpfile = SD.open(MakeFilename(filename, buffer), oflag); 
    if (!SDpfile) error2("problem writing to SD card or invalid filename");
  } else {
    char buffer[BUFFERSIZE];
    SDgfile = SD.open(MakeFilename(filename, buffer), oflag);
    if (!SDgfile) error2("problem reading from SD card or invalid filename");
  }
  object *pair = cons(var, stream(SDSTREAM, mode));
  push(pair,env);
  object *forms = cdr(args);
  object *result = eval(tf_progn(forms,env), env);
  if (mode >= 1) SDpfile.close(); else SDgfile.close();
  return result;
  #else
  (void) args, (void) env;
  error2("not supported");
  return nil;
  #endif
}

#16

Just to clarify - I assume your updated version of sp_withsdcard() hasn’t implemented this support for mode as a string? You’ve just added a fourth option, mode=3.

Do you think this should become the standard for uLisp’s SD card support, with the addition of file-position? Is it backwards compatible? Do you know if it’s supported on the other platforms?


#17

You’re correct. I only implemented adding the third (3) option above. The string option would have required updating the argument checks and fiddling around.

I’m not sure what is the best option. It just felt a bit convoluted to be switching from integer values to characters that are possibly easier to remember.

I believe outside of the esp32, the flags are as follows:

uint8_t const O_READ = 0X01;
uint8_t const O_WRITE = 0X02;
uint8_t const O_CREAT = 0X10;
uint8_t const O_APPEND = 0X04;

where something like

#define FILE_WRITE (O_READ | O_WRITE | O_CREAT | O_APPEND)

I figure you’re a better judge of how you’d like the language to go forward.

Honestly, it’s not entirely a safe thing to overwrite a file in place mid file…most people should write to a new file and then remove the old file and rename the new file. But I thought that I would finish this off, since I needed the file-position seek functionality for reading.