I will talk a little bit about the most straightforward HSV to RGB conversion algorithm, and I will try to iterate on it, for better or worse. I don’t think the result of my iteration has any significance, because:

• Most cases where the performance of HSV to RGB conversion matters probably involve GPU shader-based implementations.
• There are already well known alternative HSV to RGB implementations that I assume are much more suitable for GPUs than the approach I am iterating on.

So this post is just documenting a programming puzzle I spent time on, for fun. I will mostly discuss using pseudo-code. But I might have a closer look at performance implications of various implementations in follow-up posts.

Intro

Common algorithms for HSV to RGB conversion is documented in HSL and HSV Wikipedia page.

I spent some time exploring the first algorithm. It is simple, and it involves an interesting switch that makes me want to replace with a different approach.

There are two aspects to it:

• Calculating a set of values to be assigned to R, G, and B components.
• Distributing those values to each component.

Latter aspect shows up in the description as:

(C, X, 0) if 0 <= H' < 1
(X, C, 0) if 1 <= H' < 2
(0, C, X) if 2 <= H' < 3
(0, X, C) if 3 <= H' < 4
(X, 0, C) if 4 <= H' < 5
(C, 0, X) if 5 <= H' < 6

Here, H' corresponds to a particular factor of hue, depending on how we represent hue.

If H is represented in degrees in range [0, 360), then H' is:

H' = H / 60

If we were to represent H in range [0, 1], then:

H' = H * 6

C is chroma, where V is value, and S is saturation:

C = V x S

X doesn’t directly map to a term I know of. It is the second largest component, and its formula is:

X = C x (1 - |H'mod2 - 1|)

Once the right component ordering is found, each of those values will then be summed with V - C to get the final RGB values.

Example

Let’s manually try this with the HSV values below, all values being in [0,1] range:

H=0.6
S=0.8
V=0.7

Here is what the calculation would look like:

H' = 3.6
C = 0.8 * 0.7 = 0.56
X = 0.56 * ( 1 - |1.6-1|) = 0.56 * 0.4 = 0.224

Since H' is 3.6, base values of components is: (0, X, C). Using this, we can calculate RGB:

R = 0.7 - 0.56 + 0     = 0.14
G = 0.7 - 0.56 + 0.224 = 0.364
B = 0.7 - 0.56 + 0.56  = 0.7

Since the ordering of the X, C, and 0 values for each case doesn’t follow a very trivial pattern, the implementations I see usually have an actual switch statement. In fact, two variables are used instead of X, representing the two possible expansions of the absolute-value term.

Let’s assume F to be the fractional part of H':

X0 = C * F
X1 = C * (1 - F)

For this variant, distribution of values becomes:

(C , X0, 0 ) if 0 <= H' < 1
(X1, C , 0 ) if 1 <= H' < 2
(0 , C , X0) if 2 <= H' < 3
(0 , X1, C ) if 3 <= H' < 4
(X0, 0 , C ) if 4 <= H' < 5
(C , 0 , X1) if 5 <= H' < 6

Revising the previous example:

H' is 3.6, base values of components is now: (0, X1, C). Everything else being the same:

X1 = 0.56 * ( 1 - 0,6 ) = 0.224

In this form, the pattern of how these 4 possible values are ordered for different values of H' is even more obfuscated.

Revising the Algorithm

I was trying to find a way to rewrite this in a way to have some kind of ordering pattern. Specifically, a pattern that’s simple enough that I can implement without a switch. One approach I found is this:

Instead of assuming C as a constant, and selecting either X0 or X1, we can rewrite the distribution of components so that, both X0 and X1 always gets assigned to a component, but in each case, either X0 or X1 equals C. Now, there is a clear cycling ordering of values: X1, X0, and 0:

(X1, X0, 0 ) where X1 = C, if 0 <= H' < 1
(X1, X0, 0 ) where X0 = C, if 1 <= H' < 2
(0 , X1, X0) where X1 = C, if 2 <= H' < 3
(0 , X1, X0) where X0 = C, if 3 <= H' < 4
(X0, 0 , X1) where X1 = C, if 4 <= H' < 5
(X0, 0 , X1) where X0 = C, if 5 <= H' < 6

To achieve this, we can redefine X0 and X1 as below, where H_ODD is true if integer part of H' is odd:

H_ODD = H'mod2 >= 1
X0 = C * ( H_ODD ?     1 : F )
X1 = C * ( H_ODD ? (1-F) : 1 )

Adapting previous example again, the ordering is (0, X1, X0), where:

H_ODD = true
X0 = 0.56 * 1 = 0.56
X1 = 0.56 * ( 1 - 0.6 ) = 0.224

Since we now have a clear pattern, we can assign each value to indices that are calculated based on arithmetic. Here is a C implementation of this approach:

void
 hsv_to_rgb( double const * const hsv, double * const rgb )
{
    const double h = ( hsv[ 0 ] - floor( hsv[ 0 ] ) ) * 6;
    const int h_int = ( int ) h;
    const double h_frac = h - h_int;
    const unsigned int h_shr1 = h_int >> 1;
    const int h_odd = h_int & 1;
    const double chroma = hsv[ 1 ] * hsv[ 2 ];
    const double svf = chroma * h_frac;
    rgb[ ( h_shr1 + 0 ) % 3 ] = hsv[ 2 ] - h_odd * svf;
    rgb[ ( h_shr1 + 1 ) % 3 ] = hsv[ 2 ] - !h_odd * ( chroma - svf );
    rgb[ ( h_shr1 + 2 ) % 3 ] = hsv[ 2 ] - chroma;
}

When I did some naive benchmarking, I did observe better performance with this approach compared to the switch based implementation. However, I am pretty sure other popular HSV to RGB approaches will be faster than this. (I also haven’t really tested this implementation for edge cases.)

One alternative is actually documented in the same Wikipedia page. But I haven’t spent much time to understand that algorithm yet. I might do that in a future post.

That’s all for today. Thanks for reading! If you find technical errors, please report in the blog’s Issues page.

[ Check out all posts in “lisp” series here. ]

Short post today.

I wrote about mopr yesterday. I mentioned that USDS module supported generating attribute values for the Default timecode, but not for timesamples.

I added the initial support for timesamples today.

I had to write a mini C wrapper for UsdTimeCode, since that’s an object you need to pass as an argument while setting timesampled values on an attribute.

While at it, I also added support for authoring attributes of type timecode, which happens to be of cpp type SdfTimeCode. The underlying data is the same for both: a double. UsdTimeCode just encodes a couple sentinel values (Default and EarliestTime) as well.

Thanks for reading! If you find technical errors, please report in the blog’s Issues page.

[ Check out all posts in “lisp” series here. ]

I was trying to hit the first milestone in my latest project, called mopr, so I couldn’t write a post for the last few days.

I just open-sourced the code of this project, but it is heavily experimental, buggy, untested, unsafe, and completely undocumented. The experimental ideas I will describe in this post are similarly unsafe. So I really don’t recommend trying any of this.¹ I am just sharing it as an intellectual activity.

The repository is here. It is currently under a BSD-3-Clause license.

The acronym mopr is based on a reference to the economics term: “means of production”. I found the name suitable, since I am using mopr for experimenting with visual production related ideas.

This is a monolithic personal framework for my own VFX related experiments. So its characteristics will evolve over time, and it will likely remain heavily experimental, buggy, untested, and unsafe.

What Can It Do

Not a lot of interesting things just yet. It has the foundations that allow me to do OpenUSD-focused prototyping, mostly using Common Lisp:

• Very minimal and incomplete C and Lisp access to USD APIs like Layer, Stage, Prim.
• Support for generic data transfer from native Lisp arrays to USD’s VtValue containers.
• Boilerplate setup to allow integrating the code into OpenUSD plugins. This kind of integration is currently specific to Embeddable Common Lisp.

But this effort already enabled one prototype, that I call USDS.

USDS - Boring Parts

I do a lot of testing with USD content, and composition of content, usually to observe the outcome, confirm my intuition etc. I value being able to quickly prototype test data.

So the one actual feature I started implementing, is a declarative scene description language I call USDS, that allows me to populate a USD layer from native S-Expression (s-exp) forms, without explicitly calling USD APIs. The information stored is similar to usda format, but the representation allows symbolic manipulation. So it is more like what you would have if USDA was based on a generic serialization format.

I have support for scene graph tree building, and most attribute types. No metadata, relationship, or timecoded attribute support yet. I also have an extension that allows more concise syntax, but it comes with its own limitations, so I will skip that for now.

data/test module has some examples. 02.lisp looks like this:

((:tree
  ("hello"
   ("world")))

 (:prim ("hello")
        (:type Xform))

 (:prim ("hello" "world")
        (:type Sphere)))

When processed, this expression would produce usd that corresponds to usda below:

#usda 1.0

def Xform "hello"
{
    def Sphere "world"
    {
    }
}

Clearly, the usda form is more concise in this case. But I can directly manipulate/generate the s-exp form in Lisp.

One difference is, the USDS syntax separates the scene graph tree description (:tree forms) from the definition of prims (:prim forms), because I find it error-prone and tedious to write USD attributes inside 5 levels of nesting, when I write USDA by hand for testing.

I can also define “aliases” for prim paths. Here, x is an alias, that’s later used to refer to the prim instead of by path “/b/d/e”.

((:tree
  ("a" :spec :class)
  ("b"
   ("d"
    ("e" :spec :over :alias x)))
  ("c"))

 (:prim x
        (:type Sphere))

 (:prim ("a")))

This corresponds to a layer that consists of prims at: </a>, </b>, </b/d>, </b/d/e>, and </c>.

Also, multiple declarations targeting same entities is supported, which is useful when programmatically building parts of a prim or tree from independent functions.

Cooler Stuff

OK, let’s move on to slightly cooler stuff: I can register callables that I am able to trigger from within the USDS declarations.

Example: An expression like below will call the registered function test-gen-cubes with 30 as its only argument:

((:call :test-gen-cubes 30))

Obviously, what I am about to say would be a very unsafe thing to do, but if I were to then plug USDS into a file-format plugin, and opened the file that has the above form in usdview, I would see:

But again, this is a very bad idea. I only added support for callables to trigger calls from within a live programming context, where it is clear that some arbitrary execution might take place.

Anyway, callables are expected to be just usual Lisp functions that return USDS forms themselves, which then get processed normally. (But there is no mechanism in-place to sandbox the function being called, or prevent side effects.) Conceptually, the return value acts as a list that is spliced into the location of the :call form.

Example callable test-gen-cubes (called with arg N) produces a grid of NxN cubes, each looking like below:

def Cube "Prim_0002"
{
    double size = 0.5
    double3 xformOp:rotateXYZ = (0, 2, 2)
    float3 xformOp:translate = (0, 1, 0)
    uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:rotateXYZ"]
}

Here is an example that mixes hardcoded data with function calls at different levels: 09_call.lisp

((:call :test-tree-gen)

 (:prim :x
        (:type Sphere)
        (:call :test-gen-xform-info
               #1A (10 0 100)
               #1A (33 55 4))
        (:prop "radius"
         :double #0A 5.0d0))

 (:call :test-gen-cubes 2))

This corresponds to the USDA file here: 09_call.usda

In this last example, we have 3 call expressions. test-gen-xform-info is a “prim-level” call, and it takes 2 arguments, both are “one-dimensional arrays” (#1A).²

Architecture

The way I currently integrate with USD roughly looks like this:

The current set of key modules in repo are:³

• wrap: A very incomplete C API wrapper for some key OpenUSD modules, and minimal wrapper for miscellaneous classes.
• lisp: Common Lisp based implementation of various features building on the wrap module. This defines an ASDF system called mopr.

These 2 key modules currently build a generic API that aims to be Common Lisp implementation agnostic. So far, that’s the case for the 2 Common Lisp implementations I am testing with: SBCL, and ECL.

I am hoping to split this part to its own respository, once the OpenUSD API coverage is good enough (and tested enough) to be independently useful. For now, it only exposes the minimal API the rest of the framework needs.

lisp module mainly consists of submodules:

• ffi: Common Lisp foreign-function-interface for the API defined on wrap. This is using the ubiquitous CFFI system, and cl-autowrap library.
• mopr: An interface to interact with various USD concepts like the VtValue and VtArray, along with some with-* macros to implement RAII behaviour for foreign USD objects.
• plug: Placeholder to make the lisp system itself pluggable. It currently only registers a few hardcoded callables, so that the call expressions in the USDS declarations can trigger them.
• usds: This implements a list based declarative language to express scene description that can be used to populate an OpenUSD layer.

The remaining lisp submodules are not interesting.⁴

On top of this, there are also some modules to allow testing this lisp-based interface in a practical fashion. Specifically, I currently use Embeddable Common Lisp based Linux-native builds of the Lisp module to integrate it into the OpenUSD runtime using USD’s own plugin system:

• The boot module is built into a shared library that I use to initialize the mopr ECL modules. That becomes a dependency of the actual USD plugin library that will call into mopr lisp modules.
• plug module has an empty example USD “file format plugin” that integrates the boot module, and triggers some test code implemented in Lisp. So it doesn’t really do anything useful.

test module implements some initial Catch2 based testing. data module contains test files that I used to compare the usds forms to the reference USD ASCII format representation.

Why Common Lisp

I wrote about this a bit before. Not for any critical reasons, but:

• It is very suitable for the kind of flexible prototyping goals I have.
• It is a mature language ecosystem with some very cool properties like “homoiconicity”.
• Modern Lisp implementations find a good balance of dynamism and performance.

Goals

Here are a few things I am planning to do with this.

More Posts

Now that I have something that produces visuals, I am planning to write a series of blog posts with example procedural scenes first. (Even though this is not the most scalable (or resource-friendly) kind of proceduralism for scene content generation, it is proceduralism nevertheless.)

I am also planning to write up a bit about how I currently handle foreign objects etc.

I will add metadata etc. support soon. And hopefully, the C wrapper coverage will progressively increase. I am also planning to introduce an actual plugin interface to Lisp side.

Threading

So far, I spent no time to investigate how integrating CL-based computations into a USD plugin would work if Lisp gets called from multiple threads. I need to look into that.

I really don’t know if the OpenUSD-plugin - CommonLisp integration idea will hold well in practice.

If it does, I will likely move on to CL integration in other parts of USD, like in Hydra layer. Same fundamental data types are used on both layers, so I think it would be relatively straightforward to do similar interfacing in USD Hydra.

Data Mapping and FFI Improvements

Binary encoding of primitives is usually different between the Lisp world and the C/C++ world. So I currently don’t try to map or memcpy anything. I just iterate over the data blocks and let CFFI handle conversion. I didn’t worry about performance so far anyway. But specific optimizations might be possible, if the data passing the language boundary ends up being the bottleneck.

Also, note that I am not too experienced with FFIs. So there might be issues with my approach to bridging C++ to Lisp.

Other Implementations

Also, testing with other implementations: For now, I will continue focusing on ECL based integration. But I read that SBCL has early support for generating dynamic libraries. So I am planning to do some testing with that as well. Finally, once I am confident with the stability of various layers, I will return to investigating Clasp integration.

And since the USDS declarative syntax is basically standard s-expressions, I might test integration with a couple of Scheme implementations as well.

Sandboxed Callables

I mentioned that it is sufficient for callables to be just pure functions that produce more USDS forms. So I would like to see if it would be possible to integrate with WebAssembly, and use it as a sandbox for callables. Since WebAssembly has a text format that uses S-Expressions as well, there might be other synergy opportunities.

Other Plans

I am hoping to make the mopr framework a testbed for a few other ideas I have been investigating:

• I previously started looking into DWARF Debugging Standard. I would like to see how I can utilize DWARF for interesting runtime features, rather than just debugging.
• SBCL has a really cool concept called virtual operations (VOPs). This is another thing I have been collecting notes on, and I want to see if I can make creative use of that.

I am hoping to soon write about VOPs and DWARF anyway.

That’s all for today. I hope the post wasn’t too boring.

Thanks for reading! If you find technical errors, please report in the blog’s Issues page.

[ Check out all posts in “lisp” series here. ]

This morning, I was updating my OS packages, and I realized Chez Scheme had its v10.0 release! For those unfamiliar, here is the description of Chez from its Wikipedia page:

Chez Scheme is a programming language, a dialect and implementation of the language Scheme which is a type of Lisp. It uses an incremental native-code compiler to produce native binary files for the x86 (IA-32, x86-64), PowerPC, and SPARC processor architectures.

Links to Chez Scheme project page and source code are available in its webpage.

The release notes for v10.0.0 is available here. I have not used Chez a lot, but I have always been interested. The direction and amount of improvements really got me excited to give it a try with my current project, which I will write about soon.

There are a lot of nice improvements, from introduction of a portable bytecode to wrapper procedures and foreign interface improvements. Apparently, the compiler now generates code that locally unboxes floats. Great! But also, support for more architectures:

AArch64 (64-bit Arm), RV64G (64-bit RISC-V), and LoongArch64 architectures are supported. Threads are supported on architectures with weak memory models while preserving the safety guarantee described in the documentation for variables and primitive datatypes.

I read in forums that recently, Racket branch of Chez Scheme merged with the mainline Chez Scheme. Lisp dialects are not super mainstream nowadays, so this kind of consolidation is great to see.

Another scheme implementation I have always been interested is Gambit Scheme. Gambit has a compiler that compiles to C. I realize Gambit’s foreign interface is written in a way to also support new/delete based resource management of C++. That got me curious regarding the C++ interop experience in Gambit. Its wikipedia page also mentions that software written in Gambit-C can contain C++ and integrate with corresponding libraries.

I hope to soon try both implementations with some minimal FFI code and share my findings.

That’s all for today.

Thanks for reading! If you find technical errors, please report in the blog’s Issues page.

[ Check out all posts in “lisp” series here. ]

Common Lisp has a very powerful OOP system, called CLOS (Common Lisp Object System). To be clear, I am specifically talking about expressive power, and flexibility. CLOS is not cheap in terms of performance. I assume it will be fast enough for many tasks, but I would not rely on CLOS as I can rely on OOP concepts of C++, if performance is very important.

One of the interesting aspects of the CLOS model is the emphasis on the functions. Methods are not owned by classes, but by “generic functions”. Again, this is not the kind of generics statically typed languages like C++ has. It is just dynamic method dispatch. It is not a compile-time polymorphism solution, which I really wish existed in CL.¹

I think decoupling methods from the classes they operate on empowers the programmer to reason about the design in a more flexible manner. At least that’s my memory of my CLOS experience. CLOS also privides a standard method combination that has the concept of “auxiliary methods”: :before, :after, and :around. So a call can automatically resolve to a well-ordered set of function calls. It is a good feature, if used wisely.

Wikipedia page of CLOS has this nice diagram that visualizes the method application order defined by Standard Method Combination.

That’s all for today.

Thanks for reading! If you find technical errors, please report in the blog’s Issues page.

It has been one year since the 7.8 magnitude earthquake shook Türkiye and Syria. Description from the Wikipedia page:

On 6 February 2023, at 04:17 TRT (01:17 UTC), a Mw 7.8 earthquake struck southern and central Turkey and northern and western Syria. The epicenter was 37 km (23 mi) west–northwest of Gaziantep.[2] The earthquake had a maximum Mercalli intensity of XII (Extreme) around the epicenter and in Antakya. It was followed by a Mw 7.7 earthquake at 13:24.[3] This earthquake was centered 95 km (59 mi) north-northeast from the first. There was widespread damage and tens of thousands of fatalities.

Confirmed death toll in Türkiye is over 50,000.

Being Turkish, I have been following the events and announcements in Türkiye relating to the earthquake.

Hatay is the worst impacted city. A year after the event, many in Hatay are very angry.

There is anger towards the government. There is anger towards local authorities, including the mayor, who is a member of the main opposition party. Here is a post in English about the recent protests.

There is also 2024 Turkish local elections at the end of March. So there is obvious PR damage control, along with not-so-subtle political statements being made. Some of those statements leave me at a loss for words.

[ Check out all posts in “lisp” series here. ]

I made a brief intro to lambda lists (parameter lists) of CL in a previous post.

I wanted to mention another interesting feature of the lambda list of Common Lisp. We already mentioned that CL supports &optional, and &key parameters. It also allows both of these parameter groups to have default values.

Example:

(defun fn (a b &optional (c 5))
  (* a b c))

The optional parameter c has a default value of 5.

Let’s try to call this:

(fn 3 4 7) ; = 84
(fn 3 4)   ; = 60

Now, the cool part. You can also check whether the caller actually supplied an optional parameter, or if the value defaulted.

In languages like python, people sometimes default to an intermediate value, like None, that’s not a sensible value for that specific parameter, just to be able to distinguish between defaulted value vs a caller-supplied value that happens to match the default.

In CL, you don’t need that. You can supply a variable name to be bound to true or false, depending on whether or not an argument was supplied for the parameter during call.

(defun fn (&optional (x 0 x-supplied-p))
  (format t "Value of x: ~a~%Caller supplied x: ~a"
          x
          (if x-supplied-p "YES" "NO")))

Here is the output of various calls:

CL-USER> (fn 5)
Value of x: 5
Caller supplied x: YES
NIL
CL-USER> (fn 0)
Value of x: 0
Caller supplied x: YES
NIL
CL-USER> (fn)
Value of x: 0
Caller supplied x: NO
NIL

Thanks for reading! If you find technical errors, please report in the blog’s Issues page.

[ Check out all posts in “lisp” series here. ]

I made a brief intro to lambda lists (parameter lists) of CL in my previous post.

The lambda list of Common Lisp has an interesting feature that I haven’t encountered in other languages. It is called an &aux variable.

Basically, it allows you to define function variables in the parameter list. So it is not an actual parameter. You can’t provide an argument that will be bound to the auxiliary variable while calling the function. Rather, it is analogous to a variable binding that would be defined within the function body, using a let* form.

For reference: let and let* operators allow creating new variable bindings with a scope. let* variant performs the bindings sequentially, so each binding can refer to previously defined variables.

Let’s start with a simple function definition that doesn’t have an auxiliary variable:

(defun fn (a b)
  (* a b))

We just have a function fn that accepts parameters a and b. It returns the result of multiplying a and b. Calling (fn 2 3) would return 6.

Let’s modify this to have an &aux variable called c, bind it to a + b, and introduce c into the multiplication:

(defun fn (a b &aux (c (+ a b)))
  (* a b c))

And here is the corresponding function using the usual let* form instead:

(defun fn (a b)
  (let* ((c (+ a b)))
    (* a b c)))

As expected, (fn 2 3) will return 30 for both of these.

So why is &aux useful?

&aux is not a very popular feature of the language. The let* form above is how you would usually see local variables like this get introduced. Most uses of &aux are convenience, based on personal preference. For example, you can use it to reduce one level of indentation.

There is an old comp.lang.lisp discussion on &aux, where people shared a few other interesting uses.

Examples of convenience uses:

• Some use it for cases where they will possibly promote a local variable to a formal parameter later. That use allows the parameter “promotion” to happen without modifying the function body.
• Some people use &aux variables to emphasize an important local variable, like the variable that’s populated in function body, and eventually becomes the return value.
• Some use it for variables that are directly derived from function arguments, like the (c (+ a b)) example above. That makes the relationship obvious.

Examples of more important usecases (these will only be clear to people who already know Common Lisp):

• Non-trivial initialization of structure slots, which is also defined in the standard.
• If a macro introduces lexical bindings, introducing it via a let* form will cause possible declaration forms in the macro call to be placed in the wrong position in macro expansion. An &aux variable will avoid that.

I like &aux. I occasionally use it for each reason stated above.

Thanks for reading! If you find technical errors, please report in the blog’s Issues page.

[ Check out all posts in “lisp” series here. ]

Common Lisp parameter lists have some interesting features I haven’t encountered in other languages. But before I get to those, I will start by introducing the usual argument passing styles in CL.

Let’s start with a simple function definition:

(defun mul (a b)
  (* a b))

In CL, parameter lists are officially called “lambda list”. So (a b) is the lambda list here.

There are various types of lambda lists. The one used by functions is called “ordinary lambda list”. Macro definitions use an “extended lambda list”.

You have all the usual parameter styles that you would be familiar from languages like Python. You have required, optional, keyword, as well as a rest (catchall) style parameter. The way these are declared is slightly unconventional. The way they behave (especially the way combinations behave) is also different from other languages.

Each parameter declaration is called a “parameter specifier”. Specifiers of the same style are grouped together. There are “lambda list keywords” that describes the kind of each group.

Here is an example with optional positional parameters (... is where function body would go):

                __ required parameters
               /               __ optional parameters
           ___/          _____/
(defun fn (a b &optional c d e) ... )
                   \
                    \_ A lambda list keyword.

This can be called like:

(fn 1 2)     ; c = nil, d = nil, e = nil
(fn 1 2 3 4) ; c = 3, d = 4, e = nil

So everything you declare before the first lambda-list keyword are required parameters.

Lambda list keywords start with ampersand character, like: &optional.

Another example, this time using a &rest parameter:

                __ required
               /          __ all other positional arguments
           ___/      ____/   gathered into a list
(defun fn (a b &rest args) ... )

This can be called like:

(fn 1 2)             ; args = nil
(fn 1 2 3)           ; args = '(3)
(fn 1 2 3 4 5 6 7 8) ; args = '(3 4 5 6 7 8)

And the last one, using keyword parameters:

                __ required
               /        __ keyword arguments
           ___/     ___/
(defun fn (a b &key c d) ... )

This can be called like:

(fn 1 2)      ; c = nil , d = nil
(fn 1 2 :d 4) ; c = nil , d = 4

You can combine different groups but only a subset of combinations build an intuitive protocol.

I will continue describing some interesting aspects in future posts, but you can read a great description of details from the functions section of Practical Common Lisp.

Thanks for reading! If you find technical errors, please report in the blog’s Issues page.

[ Check out all posts in “lisp” series here. ]

Another short post today.

I have been exploring Embeddable Common-Lisp for a few weeks now. Here is my previous ECL post. I learned quite a lot since. I am now fairly familiar with:

• the general C API,
• ECL specific Lisp API,
• some key sections of the C codebase,
• the primary means of communicating between the C/C++ world and the Lisp world.

I also mentioned that I was trying to have a better grasp of the memory management aspect, which I started.

There is not much to say for objects that live purely in Lisp world.¹ Because it will just behave like what you would expect a garbage collected environment to behave.

However, the situation is different for foreign objects. Working with data across runtimes with different data models usually involves multiple layers of wrapper/proxy objects. So that already obfuscates things. On top of that, if the main memory management strategy of those runtimes are not similar, the topic of memory management and ownership becames a pretty complicated one to reason about.

I briefly talked about the cl_object type in my previous ECL related post. This value either encodes an immediate, or a pointer to a box object. The box can be set to contain “foreign data”, in which case it will point to a foreign data block. Here are a few subtleties that requires accounting for in the mental model, if you aim to transfer data across:

• There is a question of foreign data lifetime and ownership.
• There is a question of wrapper object lifetime and ownership.
• De-facto ownership expectations for the Lisp->C case is not quite the same as C->Lisp case. So there is a bit of asymmetry as well.
• Since the Lisp world utilizes garbage collection, there is the question of how references to allocations are tracked, and which memory regions are scanned for that.

At this point, there are gaps in my understanding, so I will defer elaborating further to another post. But it is an interesting topic, so I am hoping to return later with some real-world examples.

Thanks for reading! If you find technical errors, please report in the blog’s Issues page.