Refactor to multiple files

[bobach]

Hello,

This is proposal for a refactor of the entire 8/16 bit codebase to be useful with Arduino-mk and be split among multiple files with some form of abstraction.

bobach/ulisp

I read and understand Why are the uLisp sources in a single file?

but I really feel that having structured project can be really beneficial, and price to pay (harder to develop with Arduino IDE) is worth paying, mainly because:

• Arduino IDE is really limiting

• Developers already have multitude of useful tools for software development like debuggers, git, makefiles and there is not really a reason to reinvent everything inside JVM application

• It would be possible to merge all architectures under one project, without code duplication and platform specific code could be handled with makefiles.

• It could spur more of core development with more accessible codebase.

• Compilation times will be improved (compiling only the changed files)

• Since Arduino IDE does lot of preprocessing (mainly forward declarations), current codebase is not actually correct C++ code

Currently not all decisions are necessarily good, but it compiles, uploads and runs ATmega1284. And its a start.

Also for me this is a learning exercise, so please point to me any glaring errors that I overlooked.

Next to do:

• Improve abstraction of hardware (currently in streams.h, i2c.h, and image.h), so its easier to add new low-level support for protocols and ports on other platforms.

• add more comments to make codebase more accessible.

• reformat project to be includable as arduino library. Than it would be almost as easy for the users to use uLisp. (double clicking .ino file is pretty high bar, but we can come close).

Another option is to create release file, that is all files merged into one.

Hope you find my contribution worthwhile.

[johnsondavies]

Thank you for contributing this - I think it will be very useful, extending the reach of uLisp to other IDEs.

One function my Lisp “makefile” program performs is to merge the sources for the functions to be included on a particular platform, and build the corresponding lookup tables of function names and entry points. Is this something you’ll be able to automate?

• add more comments to make codebase more accessible.

I’d be happy to help with this. I have made a start on commenting the functions in my source database, but currently don’t include the verbose comments when I build for a platform. Do you have a preferred format for comments?

Regards, David

[odin]

I think this is, in the long term, going to be extremely useful. Specifically, splitting the mostly platform-independent core of uLisp away from the hardware-dependent code makes it much easier to keep things in sync. It also makes it much easier to move uLisp towards being based on a lower level, which is something that I would really like to do.

Incidentally, I would hope that we can keep track of which parts of the code are C++ and which are actually simply C. Avoiding C++ makes it much easier to interface with the code from assembler.

Edit: To clarify this last bit, my hope is that eventually much of uLisp can be written in uLisp. It will be much easier to make that happen if we only have to interface with the fairly standardised C ABI.

[Aeneas]

I am a strong proponent of the ONE file approach. If you get a project so up the ivory tower that you need to be a computer scientist to use it — you factually kill it. Arduino is a good start, as it is “beginner friendly”. If you now do something that is totally NOT beginner friendly, that is a bit of a venire contra factum proprium. Then why not start with flashing ROMs on hardware programmers and whatnot … nobody actually DOES that. Case in point: the Lisp Badge. An AWESOME piece of beauty, realized perhaps in actuality by a dozen people or fewer.

[odin]

I would counter with the structure of CircuitPython. I don’t think you would argue that that’s not a beginner-friendly system, and yet building the firmware itself is a decidedly non-trivial operation. Also, there are quite a few computer scientists who can’t function without a hand-holding IDE, so I don’t think “you need to be a computer scientist to use it” means quite what you seem to be implying. :)

Edit:

venire contra factum proprium

Also, speaking of an ivory tower and then flinging latin phrases around is … a little hypocritical.

[Aeneas]

I counter by saying nobody “builds” CircuitPython. :) They buy boards with it — and that’s it. Or MicroPython. The CircuitPython example more proves my point than counters it. — A further example is RunCPM, which is AWESOME — and not much used. Another example really is Linux vs. BSD: the BSD guys were always more RTFM-oriented, and Linux … “won”. Work for the classes, live with the masses, work for the masses, live with the classes: it is true not only for painters, but also for software. EDIT: as to Latin — if you split it into multiple files, you may as well turn the comments into Latin.

[odin]

And nobody ever updates it on their boards?

[Aeneas]

Fewer people than you would think. 😉 I got indirect evidence of this as I once tried to flash CircuitPython on a (comparatively inexpensive) Arduino Zero: their file was defective, and the process failed. I told them and they corrected it swiftly. But it proved to me that for MONTHS, apparently nobody had cared to put it on an Arduino Zero, or the problem would have been corrected previously.

[bobach]

I am not really an expert, or actually have ever done Continuous integration(CI) system, but I think it would actually be very possible to create understandable system that would:

For each commit,
for each platform

1. assemble source for given platform,
2. if possible compile it
3. if possible upload it(to real connected hardware, or simulator)
4. run some tests over serial
5. report results
6. package as arduino library

This is inspired by https://jenkins.io/solutions/embedded/

But before all that, I think all repos must be united, and hardware dependencies split in libraries with common interfaces. Basicaly that means 3 basic layers.

1. hardware level
2. ulisp runtime layer (implements eval and interacts across fixed interface to hardware level)
3. lisp functions wrappers

Currently in this fork, levels 1 and 2 are at the same level.

So for example runtime layer would include need
permament_storage library.

then for each architecture we implement “drivers” for EEPROM, SDCard, maybe external EEPROM?

and then runtime includes “permament_storage.h”, and build system chooses appropriate implementation of that library. Be that SD card or interal EEPROM. Similar how Arduino IDE chooses appropriate libraries based on core selected.

@Aeneas this is Building of circuit python) So don’t really know what you are implying? That structuring projects as every programming project ever is structured to be easier to wrap your head around is not as good because it will be harder to upload? Ok I can give you that, but it is really possible to just combine all files and release that if you don’t really wan’t to do development inside core files, whats wrong with that?

Also everything is hard before someone makes it easy, who says building and uploading a system to the board should be hard?

And as for latin Lingua latina non est forsit. :)

Edit: as for forcing C it would be really helpful, mainly because PROGMEM in avr-gcc is actually deprecated, but it is replaceable with nicer attribute __flash, but that is, and never will be supported in C++. So I am all for it.

[Aeneas]

Then we can agree to disagree: I think the easiest is to make a Ctrl-C, Ctrl-V and simply “get” the software into the editor.

I think each one of us has stated his point clearly enough. It was my aim to demonstrate that modularisation is NOT an unopposed view — this I did. What you all do is up to you, and I shall not be spiteful, but simply wish you good luck. :)

[odin]

But it proved to me that for MONTHS, apparently nobody had cared to put it on an Arduino Zero, or the problem would have been corrected previously.

Huh. I have Zero-compatible boards and never would have thought to run CircuitPython on 'em. That might be part of the issue, but point taken.

However, I don’t think those people would consider Arduino easier than updating CircuitPython on UF2 boards. I’d aim for building uLisp UF2 images.

[odin]

I just remembered something that’s possibly relevant to this discussion.

A while back, I was looking to figure out more about exactly what the Arduino IDE does during its build process. I eventually found configuration files that spelled out exactly what I was looking for, but while doing it I found out that they have been doing some pretty hefty alterations behind the scenes. Specifically, the build process itself is now being delegated to code written in Go, which in turn is also available packaged as a command-line interface.

I haven’t used this code much, but it does appear to provide a path towards making a build system that absolutely ensures that we can concatenate our multiple files and build the whole thing in the Arduino IDE - because it itself runs the same code to do the build. It also appears to be simpler to setup for ARM than Arduino-mk.

There also seems to be a fairly simple way to generate UF2 files that could be uploaded to any of the Adafruit Express boards in the same way as a CircuitPython update is. I’ve just done exactly that for an ItsyBitsy M4 Express, flashing it from an Arduino IDE-uploaded blink sketch to the latest uLisp beta with my own additions by dragging a file onto a USB drive.

[odin]

One function my Lisp “makefile” program performs is to merge the sources for the functions to be included on a particular platform, and build the corresponding lookup tables of function names and entry points.

Would you mind showing the Lisp build system? It sounds interesting.

[johnsondavies]

Would you mind showing the Lisp build system? It sounds interesting.

I’ll send it to you in a message.

[jurov]

Why is the requirement to have one .ino file, or to have a build system that generates that one file? Recent versions of Arduino IDE are okay with having multiple C++ source files along with .ino file, just put them together in the same directory. No custom build system necessary.

I believe having standard C/C++ code (regardless of whether monolithic or modular) and stub .ino file with only setup() and loop() would be a boon for portability and interop. I actually had to do it myself to be able to use ulisp with PlatformIO, this is not mine but similar: https://github.com/ahmetus/esp32-ulisp-with-platformio

[johnsondavies]

There’s no requirement - it’s just that I prefer working with a single file.

See: Why are the uLisp sources in a single file?

I welcome the creation of a multi-file version if people prefer to work with that.