opaque types

[dragoncoder047]

I had another idea: user-definable opaque types. You could add a void* pointer to the car and cdr of the object union, and then have a variable OpaqueCounter which is the highest registered opaque type which must be less than the start of the workspace to avoid confusing it with a cons cell.

You would have to fix the markobject routine so it wouldn't start marking wild pointers into out-of-workspace memory -- a simple fix would be to, before marking or recursing on a pointer, check to see if (obj >= &Workspace[0] && object <= &Workspace[WORKSPACESIZE-1]). That way, the opaque type could use Lisp lists/trees to store garbage-collected variable-size data structures.

The idea of this is that you could implement a

uint32_t OpaqueCounter = PAIR;
void NextOpaqueID () {
  uint32_t temp = OpaqueCounter;
  if (temp >= &Workspace[0]) return 0; // Bail, out of IDs
  OpaqueCounter += 2;
  return temp;
}

void printopaque (object* obj, pfun_t pfun) {
  pfstring(PSTR("<opaque-"), pfun);
  pint(obj->type, pfun);
  pfstring(PSTR(" at 0x"), pfun);
  pintbase((uint32_t)obj, 16, pfun);
  pfun('>');
}

object* makeopaque (uint32_t id, void* arg) {
  if (id < PAIR || id >= &Workspace[0]) error2(PSTR("invalid opaque id"));
  object* obj = myalloc();
  obj->type = id;
  obj->cdr = (object*)arg;
  return obj;
}

void* checkopaque (uint32_t id, object* obj) {
  if (obj == NULL || obj->type != id) error(PSTR("expected opaque type"), obj);
  return (void*)obj->cdr;
}

void markobject (object *obj) {
  MARK:
  if (obj == NULL) return;
+ if (obj < &Workspace[0] || object > &Workspace[WORKSPACESIZE-1]) return; // Wild opaque pointer
  if (marked(obj)) return;

  object* arg = car(obj);
  unsigned int type = obj->type;
  mark(obj);

- if (type >= PAIR || type == ZZERO) { // cons
+ if (type >= &Workspace[0] || type == ZZERO) { // cons
    markobject(arg);
    obj = cdr(obj);
    goto MARK;
  }
  /* SNIP */
}

I probably haven't thought of the rest of it (edits to the printobject() routine, etc) or how to display a pretty name for the type, but I am open to suggestions.

The one big problem with this is I can't figure out a simple API for how an extension should be notified when an opaque object is garbage collected and the external memory should be freed, to avoid memory leaks. You could just put a big warning that your program needs to keep its own references and free them later somehow, or allocate objects statically or on the stack, but that doesn't sit quite right with me.

[technoblogy]

I'm not familiar with the concept of opaque types. I'll have to read up about it. What are they useful for?

[dragoncoder047]

Oops, sorry about that. I meant to write that but I never finished my sentence.

The idea is that if you have some objects that aren't primitive values, but it wouldn't make sense to have them available in a context-manager (with-XXX) block, you can stuff them in an opaque object and then pass the object around to uLisp extension functions that do stuff with it. I was thinking like the stuff with the ESP32's bluetooth API. It's not simple enough that you could put it in a (with-bluetooth) block. That wouldn't make much sense, anyway, because there are many different aspects of bluetooth (BLE, characteristics, properties, different protocols, cliets, servers, scanning for devices, connecting, disconnecting, receiving advertisement data, receiving property change notifications, writing values, etc) that it makes more sense in my mind to make opaque object wrapping the ESP32 classes and pass them into functions.

[dragoncoder047]

The one big problem with this is I can't figure out a simple API for how an extension should be notified when an opaque object is garbage collected and the external memory should be freed

Combined with #71, I think I figured it out -- the only time that memory needs to be freed is when a garbage collection occurs. So here is my solution:

Each extension could keep references to all its opaque pointers, and designate a garbage collector hook function (typedef void (*gchook_t)(void);) in its mtbl_entry_t. The garbage collector could call each of those after it is done sweeping. The hook function would then be in charge of scanning the Workspace to look for pointers to any of that extension's opaque objects. If it finds none, it should free the object. I think this should be left up to extension authors to write, because the implementation may depend on the platform -- on something like an ESP32 where you have coroutines, you can do the scanning in a separate thread (on the same core) and only set a "dirty" flag from the GC-hook function.

[technoblogy]

OK, so I think I get it. Opaque types are a way of having Lisp objects that point to memory that's not in Lisp's workspace. For example, you might want to write a Lisp extension that has Lisp symbols that point to C structs.

Each extension could ... designate a garbage collector hook function

Couldn't it still use Lisp's garbage collector to decide whether the object is accessible, and then call a garbage collector hook function to actually reclaim the storage, such as with free if it was allocated with malloc.

[dragoncoder047]

Couldn't it still use Lisp's garbage collector to decide whether the object is accessible, and then call a garbage collector hook function to actually reclaim the storage, such as with free if it was allocated with malloc.

Well, C++ complicates this in its use of delete which calls the destructor vs. just a bare free(), which doesn't. uLisp would need to be made aware of a) all of the opaque objects so it can get to the unmarked ones, b) what type to cast the opaque pointers to (so the right destructor can be called), and c) how to mark/unmark them, which I don't see any easy way of doing.

I think it would best be left up to extension authors to deal with this by sweeping their own arrays, marking their own objects, etc.

p.s. in Lisp contexts is it more appropriate to mark code using **bold** versus `monospace` text?

[technoblogy]

In Lisp contexts is it more appropriate to mark code using bold versus monospace text?

I've seen both used. I always use bold for in-line Lisp words because I think monospaced text looks ugly in the middle of a sentence in a proportional font.

[technoblogy]

To be realistic, I don't think I'm going to get around to implementing opaque types - I think they go a bit beyond what I'm intending for uLisp - but if you decide to have a go at implementing them on your fork of uLisp I'd be very interested to see how they work and try them out.

[dragoncoder047]

I think the new streams idea and opaque objects are very similar in some respects, so I'll hold off on trying opaque objects until I see your new streams version.