Perl 5.12 introduced the pluggable keyword API, which allows modules to define custom "keyword-headed" expressions (that is, your new custom keyword must be concerned with what comes after it, not before).
The idea of modifying Perl to introduce new keywords and operators goes back further, to the introduction of source filters in Perl 5.6 (almost a quarter century ago). The use of source filters is discouraged these days, mainly due to "only perl can parse Perl" (that is, the language has a single implementation and some difficult-to-specify dynamic behaviours - there is no static analyser or compiler. (There are other Perl implementations out there, but they may feature various compromises meaning they cannot work with arbitrary Perl source, or they may no longer be maintained)).
Source filters are akin to C preprocessor macros, in that they sit between the source code and
the parser - the parser receives fully modified source. Source filters frequently operate with
regex match and replace operations, which may be error prone when applied to a complete source
file. They may also have side effects depending on how they are implemented, such as clobbering
your __DATA__
section.
Another approach, now deprecated, was Devel::Declare
,
which hooked into the parser to inject keywords and other functionality. Do not use this - it
is naught but an interesting historical artifact.
The pluggable keyword API enables keyword creation at build-time, and can take complete advantage of the parser's lifecycle and language recognition features.
This post documents my own fumblings with pluggable keywords and works through an example keyword-based feature to explore the ins-and-outs of pluggable keywords (tl;dr: do not walk my path - I get close, but fail to attain cigar).
We should begin with a swift overview of the API itself (swift so I can wave my hands a lot, and paper over the parts I don't understand with vague terms).
The entry point to the API is a function pointer called PL_keyword_plugin
. When your
custom keyword plugin loads, it should set this pointer to run your code when a potential keyword
is encountered. Any existing value of the pointer should be stashed so you can call that extant function
if you're not interested in the keyword. That is, you set up a chain of execution where each plugin
calls the one loaded before until a keyword is recognised.
The function prototype is (mostly) (char *keyword_ptr, STRLEN keyword_len, OP **op_ptr)
.
The first parameter is a pointer to the source code text at the point the keyword was encountered.
The second parameter is the length of the keyword. The third is a pointer to the optree - a data
structure representing the source code being parsed, analogous to an abstract syntax tree.
Your function should "consume" as much of the source as is appropriate, and update the optree with the data and functionality required to do the keyword's work.
The pluggable keyword API as designed requires use of XS, a language which acts as an interface between Perl and C. As I am not familiar enough with XS to develop a new feature with it, let's for now discount it as an easy option (remember the title of this post - we're chasing easy for now).
Keyword::Simple
offers a wrapper for the keyword API.
It provides a define
function which takes two parameters, the name of your keyword, and a
callback to handle a reference to the source code from just after the point your keyword was used
(that is, the keyword is already consumed). Your callback should modify this reference to inject new
functionality into the source.
Immediately I think "Wait, isn't this just a source filter?".
Not quite. A source filter acts on the entire source file and often has to guess at the correct context. Here the parser has recognised a keyword and passed that to the callback, so we can be fairly certain the context is appropriate.
This allows for simpler parsing in our callback, as we don't have to account for other syntax features and can focus just on what's expected in the new syntax - that is, we don't have to check if we are making changes in the middle of a heredoc or a comment or otherwise outside the desired context.
As with source filters, we do have new source code injected for the parser - expect possible side effects, such as line numbers in error messages no longer lining up with the source file (Though some authors take steps to preserve line numbers from the original source).
Also, as we'll see shortly, there are tools to help control the complexities of parsing your keyword's parameters.
We can take a look at reverse dependencies of Keyword::Simple
for some clues on how best to exploit its
powers - taking one at random, PerlX::ScopeFunction
.
Looking at the source, the major thing to note is that it makes use of PPR
,
Damian Conway's Perl Pattern Recogniser. While I noted before that "only perl can parse Perl", PPR
can
match patterns and build a tree from Perl source just as effectively. I'll link to a talk below where Damian
outlines some of the advanced feats PPR
pulls off, as well as some of the arcane magick involved in its
construction.
Keyword::Declare
builds upon the simple approach offered by
Keyword::Simple
with some PPR
-powered niceties. New keywords are defined using keyword
, which takes the
name of your new keyword, a list of named PPR
entities, and a block which should return your replacement
source - the named entities are consumed from the source. That is, your returned string replaces the matched entities
you listed in the declaration.
This example will use Keyword::Declare
to (partially) implement multiple dispatch, or multimethods. Multimethods allow
for the definition of several variants of a method with the same identifier. Which variant to execute is selected
at run-time, based on the data types of the parameters. Don't think traditional types which offer compiler
cues for things like storage requirements, compile time validation of assignment, or casting. This is a run-time
process which inspects the content of parameters to see what they look like. There is no casting or autoboxing or
build-time validation.
Use of the new keyword multi
should look something like:
class Multiplier {
multi method multiply( Num $multiplier, Num $multiplicand ) {
$multiplier * $multiplicand;
}
multi method multiply( Str $string, Int $multiplicand ) {
$string x $multiplicand;
}
multi method multiply( OBJECT $obj, Int $multiplicand ) {
map { $obj->clone } 1..$multiplicand;
}
...
}
Here we have three different definitions for the multiply
method, one of which may be executed at runtime
depending on whether the first parameter is a number, string, or object instance. The second parameter should
also be validated to be a number of the appropriate form.
Since we have a fair idea of how the syntax will look, let's take a peek at a complete keword definition:
keyword multi (
/sub|method/ $sub,
Ident $method,
Attributes? $attribs,
List? $raw_params,
Block $code
) {
my $signature = _extract_signature( $raw_params );
my $param_string = join ',', keys $signature->%*;
my @types = values $signature->%*;
my $signature_name = join '_', map { $_ // 'undef' } @types;
my $target_method = "_multimethod_${signature_name}_$method";
$methods->{ $class }->{ $method } //= [];
push $methods->{ $class }->{ $method }->@*, {
signature => $signature, method => $target_method
};
_build_type_checkers( @types );
_inject_proxy_method( $class, $method );
"$sub $target_method $attribs ( $param_string ) $code";
}
This keyword should expect to be placed before a sub or method declaration, consisting of an identifier, optional attributes, an optional list of expected parameters, then a block of code (it occurs to me now reading this that we could likely leave the code block out of the keyword definition - it is not changed at all).
As our param list also contains new and unique syntax, we'll need to unroll that ourselves in
_extract_signature
:
sub _extract_signature( $raw_params ) {
$raw_params =~ s/^\(//;
$raw_params =~ s/\)$//;
+{
map {
my @param = split " ", $_;
$param[1]
? ( $param[1] => $param[0] )
: ( $param[0] => undef )
} split ",", $raw_params
}
}
This removes the parentheses from the parameter list text, then builds a hash of parameter => type. For example, the method declaration:
multi method hello( Int $foo, $bar );
...would result in the hash:
{
$foo => 'Int',
$bar => undef
}
Returning to the keyword definition, the next lines look like so:
my $param_string = join ',', keys $signature->%*;
my @types = values $signature->%*;
my $signature_name = join '_', map { $_ // 'undef' } @types;
my $target_method = "_multimethod_${signature_name}_$method";
The $param_string
variable will become useful when returning text from the keyword definition block.
The array @types
will contain a list of the types extracted from the signature in the order they
were encountered. This is used to build a unique method name, $target_method
, which acts as a
means to hack our multimethod definitions into the package/class namespace, where method names
must be unique.
Let's return to the keyword definition again - the next lines are:
die "Ambiguous signature in $method declaration"
if $class->can( $target_method );
$methods->{ $class }->{ $method } //= [];
push $methods->{ $class }->{ $method }->@*, {
signature => $signature, method => $target_method
};
Firstly a quick validation is performed - has a multi method with this signature already been declared?
The $methods
hashref is a file scoped stash for info about the current method -
the method's signature hash and $target_method
name. Next:
_build_type_checkers( @types );
_inject_proxy_method( $class, $method );
We start with _build_type_checkers
...
sub _build_type_checkers( @types ) {
for my $type ( uniq sort grep { $_ } @types ) {
next if $checkers->{ $type };
$checkers->{ $type } = sub( $datum ) {
state $ts_type = Types::Standard->can( $type );
$ts_type
? $ts_type->()->assert_valid( $datum )
: blessed $datum && $datum->isa( $type )
or die("$datum is not an instance of $type");
};
}
}
This function maintains another file-scoped stash, $checkers
, which contiains validation coderefs to check
passed data against the named type. If the type is supported by
Types::Standard
, its assert_valid
method is used,
otherwise the type is considered an isa
check - is this an instance of a specified object?
Next we _inject_proxy_method
...
sub _inject_proxy_method( $class, $method ) {
return if $class->can( $method );
my $meta = Object::Pad::MOP::Class->for_class( $class );
$meta->add_method(
"$method",
sub {
Multimethod::delegate( $class, $method, @_ )
}
);
}
Here we make use of the experimental Object::Pad
MOP (meta-object
protocol). A MOP is an API which allows for inspection and modification of a class' features - class members,
methods, roles, inheritance tree, and so on. The API opens up class internals and allows definition of new
roles and classes dynamically.
Note the explicit stringification of $method
- this is an instance of Keyword::Declare::Arg
.
The proxy method we inject into the class here is a redefinition of the multimethod name to delegate the
selection of, and calling of the appropriate method to Multimethod::delegate
, which looks like this:
sub delegate( $class, $method, $instance, @params ) {
my $delegates = $methods->{ $class }->{ $method };
my $delegate_method = _find_signature_match( $delegates, @params );
die "No delegate method found for ${class}::$method" unless $delegate_method;
$instance->$delegate_method( @params );
}
This starts by pulling the set of methods defined for the method name from the $methods
stash
(the potential delegates (noun) to delegate (verb) to - don't blame me, I only speak this language),
then calls _find_signature_match
to find the first matching method. If a matching method is
found, it is called, otherwise the program bails out. The _find_signature_match
function is
as follows:
sub _find_signature_match( $delegates, @params ) {
OUTER: for my $delegate ( $delegates->@* ) {
my @types = values $delegate->{ signature }->%*;
my $iter = each_array( @types, @params );
while ( my ( $type, $param ) = $iter->() ) {
next unless $type;
try {
$checkers->{ $type }->( $param );
} catch( $e ) {
next OUTER;
}
}
return $delegate->{ method };
}
}
This simply iterates over each delegate method and returns the first one for which the passed parameters pass all type checks stashed for that declaration, or nothing at all.
Looking back to the keyword definition, the final job is to return the now-uniquely-named method definition:
"$sub $target_method $attribs ( $param_string ) $code";
The code outlined here is run each time the keyword multi
is encountered.
Let's kick off a REPL:
$ reply
0> use Object::Pad
1> use Multimethod
Variable "$class" will not stay shared at lib/Multimethod.pm line 112.
An inauspicious start. Multimethod
's import function looks as follows:
sub import {
my $class = caller();
keyword multi ( ...
The $class
variable is used within keyword
's block. As this is not an anonymous sub, it does
not create a closure around $class. As the block may be called at any time, perl warns us of a potential
pitfall - this block is working with a copy of $class
. As we can be fairly certain the block is run
now, this warning should be safe to ignore.
Also, it makes no sense to use Multimethod
at this level - it needs a package name to work with in
caller()
.
Let's start again:
$ reply
0> use v5.38.0
1> use Object::Pad
2> class Foo {
2... use Multimethod;
2... multi method say_things( Int $int ) { say "Got an int : $int" };
2... multi method say_things( Str $str ) { say "Got a string : $str" };
2... multi method say_things( $thing ) { say "Got something else : $thing" };
2... }
Variable "$class" will not stay shared at lib/Multimethod.pm line 112.
$res[0] = 1
Nothing catastrophic so far. Here we see three variants of say_things
declared. It should hopefully be
obvious what each of these methods do and what data they respond to.
Let's instantiate the class and see how the namespace looks:
3> my $foo = Foo->new
Object::Pad::MOP is experimental and may be changed or removed without notice at /home/fuzzix/perl5/perlbrew/perls/perl-5.38.0/lib/site_perl/5.38.0/Data/Printer/Filter/GenericClass.pm line 96.
$res[1] = Foo {
parents: Object::Pad::UNIVERSAL
public methods (5):
DOES, META, new, say_things
Object::Pad::UNIVERSAL:
BUILDARGS
private methods (3): _multimethod_Int_say_things, _multimethod_Str_say_things, _multimethod_undef_say_things
internals: []
}
We can see the new proxy method say_things
, and the uniquely named methods for each variant of say_things
declared above. A thought occurs - this system calls nominally "private" methods from another
package. This is a definite wart.
The warning is from Data::Printer
's inspection of object internals using the experimental Object::Pad::MOP
.
Moving on, let's see if the multimethods work as expected:
4> $foo->say_things( 123 )
Got an int : 123
$res[2] = 1
5> $foo->say_things( 'abc' )
Got a string : abc
$res[3] = 1
6> $foo->say_things( {} )
Got something else : HASH(0x2f6a130)
$res[4] = 1
This looks good to me! (Merge it!)
Writing class definitions in a REPL isn't much fun, so I will write some basic tests and include them with the mlutimethod example source code.
While we ended up with a more-or-less functional mlutimethod implementation, it is not without problems (besides being woefully incomplete and lacking real validation, tests, etc.)
I think one major issue is the namespace pollution - we end up adding a chunk of mildly inscrutable private methods for each set of multimethod declarations. They may not have seemed so bad in the example above, but in even a modestly complex class, these methods may start to pile up.
There's also the fact that these nominally private methods are called from outside the class. Another
approach might be to add a private delegate method to the class, but this clutters the namespace further.
There's also AUTOLOAD
- we could create a package named
multi
with an AUTOLOAD
block which extracts a method name. That is, a proxy method named foo
would
call $self->multi::foo( @args )
, rather than Multimethod::delegate
.
The package multi
could then dispatch to the correct variant of foo
based on the content of @args
. See curry
for an example of this type
of thing. TMTOWTDI.
The generic isa
object check for types not defined in Types::Standard
is not going to work in 99.9% of
cases. The typical package name (using ::
OR '
package separators) really seems to confuse the signatures
parser. My expectation was that the signature would be parsed after Multimethod
rewrites it with a valid
signature, but something else happens here. I don't currently have more insight.
It should be obvious that the example presented here is far from production ready ... but also, that one
probably should not write this type of functionality themselves. While Types::Standard
is fantastic,
there is an ongoing effort dubbed 'Oshun', which aims to bring data checks into core.
This work could form the basis of multimethod support in
Corinna, the specification and exploration project behind core feature 'class'
.
Object::Pad
is an exploratory implementation of feature 'class'
which I used here for the MOP.
I think the last major issue I have is rewriting source code. While this is far more tighly controlled than
plain source filters, it feels error-prone - displaced error messages, a weirded call-stack ...
I need to look into the XS optree manipulation approach.
XS::Parse::Keyword
appears to offer some niceties to perhaps
help me along the path. This distribution also includes XS::Parse::Infix
to help support a relatively new infix operator API (oh yeah, this is the juice!) There may be a follow-up to this
post.
As for the opening question, how easy was all this? Quite easy! I don't know one end of a compiler from the other,
what I know about type systems could be transcribed to a postage stamp in crayon, and yet with a little dyanamism
and a lot of sticky tape I built something that worked, after a fashion. I think the accessibility of Keyword::Simple
and
Keyword::Declare
are a boon to the ecosystem. I can see myself making use of them again in future, albeit with
some more care and attention based on what was observed here.
I learned something - I hope you did too.
Despite attempts to pre-emptively absolve myself of any responsibility for accuracy with talk of vagueness, I do not wish to spread disinformation. If something I've said here is completely wrong, feel free to reach out, or hit me across the nose with a rolled up newspaper and say "BAD!"
While working on a distributed computing infrastructure project, I found myself wrangling with the problem of service discovery - how might my nodes find each other on the network without requiring a lot of manual configuration? As time was of the essence, I ended up with a custom, hand-rolled REST service which nodes used to advertise their presence and capabilities to a centralised name service.
Now that time is no longer of the essence, I have decided to revisit this system.
Or more accurately DNS-based Service Discovery (DNS-SD) with Multicast (mDNS). A full Zeroconf stack also includes IP allocation, with hosts attempting to claim and defend IPs in the link-local address space. Finding a network which doesn't use DHCP or static address allocation is rare, so having a link-local address is often a sign some configuration has failed.
The combination of DNS-SD and mDNS is known as by Apple as Bonjour (they came up with it, so they get to name it). Avahi provides a complete Zeroconf implementation, deployable on a Linux or BSD system which runs D-Bus. On Windows ... well, we'll get to Windows.
DNS-SD and mDNS are deployed widely a number of contexts, especially in home networking. The canonical example is automatic discovery of a networked printer. Running a mDNS browser on my home network, besides a printer I can see the speaker in my kitchen, a bunch of light bulbs, switches & dimmers, as well as any number of network services - SSH, Kodi RPC, file sharing ... the list goes on (and the sysadmin in me shudders). I have also used Bonjour explicitly in the past for RTP-MIDI.
DNS-SD is often conflated with mDNS because of its use in this home-networking context, but DNS-SD may just as easily be deployed using traditional unicast DNS, allowing for internet-scale service discovery and dynamic allocation.
DNS-SD (RFC6763) is built upon a number of extant DNS standards. PTR records, most commonly used for reverse-DNS, can enumerate relevant domains, advertise the presence of a directory of services on the domain, as well as providing this directory of services. Once a desired service has been identified, SRV records provide a list of hosts providing the service, with priority information to assist with failover and capacity planning. A TXT record may provide further basic information about the capabilities of the provided service - you don't want to send an A4 letter to an A0 plotter (this would be a mistake writ large). Finally the host's IP is found with a simple A or AAAA record lookup.
This sounds ideal for my purposes. I want my distributed computing services to be able to advertise their presence, while they browse the network for other components of the system. Dynamically retiring offline nodes and routing requests to new nodes without config changes is also compelling. In the unlikely event the system escapes a single broadcast domain, the DNS-SD component may be upgraded to use unicast DNS.
It may seem tempting now to overload the TXT record with additional service info about specific capabilities and capacity (outside priority provided in SRV records), but once a service is discovered, these specifics should be queried from the service itself.
Such a popular and widespread protocol suite is surely well described in a number of accessible and easy to read books, right?
To my knowledge there is precisely one book - Zero Configuration Networking: The Definitive Guide (2005) by Daniel H Steinberg and Stuart Cheshire. Perhaps there only needs to be one book - this is the definitive guide after all, and Stuart Cheshire is one of the authors of the mDNS and DNS-SD RFCs!
As far as it goes, this book is pretty good, giving a nice high-level overview of the various moving parts of Zeroconf. There is some code, describing an API with 2005 style concurrency primitives. I'm not sure how much this API has changed in the intervening decades, but with the existence of Avahi, and the Windows situation (I'll get to that, I promise), it is not a complete picture.
I thought Andrew S. Tanenbaum and David J. Wetherall's Computer Networks (2010) might provide further context - it has helped in the past. Of Multicast DNS it says:
224.0.0.251 All DNS Servers on a LAN
...I guess.
I think I'll take the Web Server Gateway Interface (WSGI) approach. WSGI provided a simple interface for web framework authors and web server programmers and administrators. This was a roaring success, and the specification has since been ported to a number of programming languages. Sometimes an idiomatic native implementation is the right solution.
There is a reserved TLD for mDNS, '.local'. On an appropriately configured system, you may look up .local A and AAAA records with any resolver library which delegates lookup to the system.
You may also be able to trick DNS tooling to do mDNS lookups. Older versions of dig
could be coerced into performing mDNS
lookups by querying the multicast domain explicitly:
$ dig -p 5353 @224.0.0.251 myrouter.local A
...
;; QUESTION SECTION:
;myrouter.local. IN A
;; ANSWER SECTION:
myrouter.local. 10 IN A 192.168.0.1
;; Query time: 0 msec
...
Though this does not extend to browsing for services:
$ dig -p 5353 @224.0.0.251 _http._tcp.myrouter.local SRV
...
;; connection timed out; no servers could be reached
I think subclassing, or otherwise extending a DNS library may be a good idea in order to take advantage of its response-parsing capabilities. The querying approach diverges from DNS. Instead of requesting a response from a single node, mDNS casts a wide net to capture all capable nodes on a domain, then those which feel themselves responsible for answering the request respond. This may take time, and may require a series of responses to be coalesced, or even a series of requests to be issued.
I think the following sections describe a reasonable approach to the implementation of mDNS responder services.
In the presence of an existing service, every effort should be made to integrate with it. Service advertisements should be propagated to and handled by the native system as much as possible. I don't yet know how I might do that with mDNSResponder (i.e. Apple) systems, though I have a path forward for systems running Avahi.
Those familiar with Avahi may know that is opens its UDP socket with SO_REUSEPORT
enabled. This means other services may bind to
UDP:5353 and advertise their capabilities. This may be disabled with the disallow-other-stacks
config option. The man page for
this option states:
disallow-other-stacks= Takes a boolean value ("yes" or "no"). If set to "yes" no other process is allowed to bind to UDP port 5353. This effectively impedes other mDNS stacks from running on the host. Use this as a security measure to make sure that only Avahi is responsible for mDNS traffic. Please note that we do not recommend running multiple mDNS stacks on the same host simultaneously. This hampers reliability and is a waste of resources. However, to not annoy people this option defaults to "no".
That is, other mDNS responders may set themselves up in an ad-hoc fashion and I guess we shouldn't stop them by default. The observation that it "hampers reliability" I think understates things a little.
Advertising services on systems using systemd.dnssd should also be possible using D-Bus.
SO_REUSEPORT
In the absence of an existing responder, I think the only option is to create one. Reading the above, it might seem that we can use
a hand-rolled responder alongside Avahi, but this excellent StackOverflow answer details some of the vagaries of SO_REUSEPORT
and SO_REUSEADDR
.
In short you may be able to reuse the IP and port combination, if your kernel allows it. More recent Linux versions implement "port hijacking" prevention - "All sockets that want to share the same address and port combination must belong to processes that share the same effective user ID".
Once you have bound to your port, again depending on your kernel, you may not actually receive all mDNS requests to the system - your kernel may decide to distribute requests across services bound to the same port.
Though this behaviour might be different if you bind to a multicast address (or rather join a multicast group) - some sources say this will propagate requests to all listeners. Time to hit the books again...
Advanced Programming in the UNIX Environment Third Edition (2013) by W. Richard Stevens and Stephen A. Rago should tell me everything I need to know about socket options. (I have the Second Edition on my bookshelf - check that out if you want to see a cover that has not aged well).
Its index does not mention multicast at all ... nor does it mention SO_REUSEPORT
. The section on SO_REUSEADDR
is part of a single page
and mostly consists of an example regarding a TCP service. So there's another rabbit hole.
For now, binding a hand-rolled responder to a socket and allowing for reuse seems to be the least-worst approach in the absence of a canonical responder service.
I'm still feeling my way around here, but Windows these days appears to ship with DNS-SD support in the Win32 API, though this blog post on mDNS security issues appears to suggest that services are responsible for implementing their own listeners. Either way, some experimentation is called for.
The path forward for now appears to be to use the existing DNS-SD infrastructure where it exists (i.e. Windows 10+), falling back to the hand-rolled responder where infrastructure is not available. Some installations may have Apple Bonjour installed, but I'm going to call that a rarity and not (yet) consider that setup for inclusion.
Nope! Consider this an experiment in feasibility studies. There's been some research, some exploration of the landscape as it exists, and some marginal experimentation (with much more to come). I know this sort of blog post usually appears when one has a shiny new thing to show off, so let's call this "Part One", with "Part Two" to appear in the next week or in 2026.
If I can shake an implementation out of the above without having to burrow down too many more rabbit holes, I'll be happy enough... Though that's not very likely, is it? ð
Perhaps this post belongs in a notebook instead, but I don't use a notebook.
Why don't I just use an existing zeroconf library? Because it doesn't completely exist in my ecosystem. There are pieces, and there are complete implementations (with Avahi integration) which won't work for me.
It's space year 2023. I don't want to write IO-bound infrastructure which blocks while awaiting input. I could kick my blocking service to another process, but then instead of a cross-platform multicast DNS headache, I have a cross-platform IPC and process management headache. I know which headache I want.
...and perhaps if I bring it into existence, it will help help enrich the ecosystem just the tiniest amount.
The original distributed computing project focussed on wrapping a distributed object framework in an abstraction. Most concerns were eventually hidden behind a single keyword, apart from the sticky business of network configuration. As mentioned in the intro, an ugly second system was bolted on to maintain the illusion.
While the system should accomodate as much flexibility (or rigidity) as required, I think its default operation should be as simple as plugging in a printer, with clients finding the new service with ease ("PC Load Letter" issues notwithstanding).
I plan to go into some detail on this project once I get the service discovery detail worked out. That is, in the next couple months or in 2027.
In conclusion, DNS-SD is a land of contrasts.
Forgive me if I got some detail wrong here - I tried to keep things high-level as this is exploratory work for a side-project. That is, my usual "I am far from an expert on this" disclaimer applies.
I should prefix this post with a note: I am far from an event systems expert. This post describes some fumbling about and getting to grips with event-driven systems, with a specific focus on MIDI to act as an example
While I continue working (somewhat slowly) on the Perl binding to RtMidi, I have been thinking about the general problem of integrating heterogeneous event systems.
A typical async event system operates using a set of file descriptors (fd), which may signal their readiness when data becomes available for processing. File descriptors may represent actual files, sockets, pipes, or even timers and generic system events. File descriptors can be gathered into a simple data structure, which is then passed to a system call like select
, kqueue
, or some variant of poll
. When one of the descriptors is ready (has data to be read - is read-y), this system call returns details on one or more events to be handled. Implementation details will differ across event systems and platform, but for our purposes this is an adequate description of event notifier internals.
While "Everything is a file" is a powerful tenet of the Unix paradigm, it also happens not to be true. Many things are
files, other things are signals. One may not select
or poll
for signals - they must be explicitly handled outside
normal program control flow. DJB's self-pipe trick pattern offers a simple way
for signals to notify the event system - a pipe is opened and watched by select
and friends, then signal handlers
write to this pipe. IO::Async::Signal provides signal handling within its
event loop - as I say, implementations vary.
RtMidi provides its own callback-based event system, which we can see in use in this callback example from the RtMidi documentation. Unlike some other event systems, there is no explicit loop runner and there isn't much we might attach a fd to. We might set up a callback to write to an eventfd or pipe watched by the program, but the linked example is instructive for another reason. The program's "loop" is effectively doing nothing. A callback example in Python's RtMidi binding demonstrates a similar pattern.
Not only will RtMidi not integrate with your event loop, it probably won't even coexist peacefully alongside it. A callback could be triggered at any point in your program's flow, and control may not be handed back cleanly - your call stack and event system state may now be classed as "deeply unhappy". That is, to be ready to process a RtMidi callback, your program should probably not be doing much of anything.
Knowing all of the above, how may we exploit the characteristics of RtMidi and IO::Async to have MIDI input be simply
another event notifier we can await
or attach an async callback to? A common approach when there is a requirement to meld incompatible event loops is to give each loop its
own thread, each with its own specialised event handling, with some communication between them to notify about specific events relevant to both loops (e.g. tell the GUI that 20% of a file has been processed).
An approach came to mind to allow the handling of events be programmed in a single event system (IO::Async in this case) - use IO::Async::Routine to start a process (or thread) which attaches a simple callback to RtMidi to pass all events back to the originating program via an IO::Async::Channel.
my $loop = IO::Async::Loop->new;
my $midi_ch = IO::Async::Channel->new;
my $midi_rtn = IO::Async::Routine->new(
channels_out => [ $midi_ch ],
code => sub {
my $midi_in = MIDI::RtMidi::FFI::Device->new( type => 'in' );
$midi_in->open_port_by_name( qr/LKMK3/i ); # LaunchKey Mk 3
$midi_in->set_callback(
sub( $ts, $msg, $data ) {
my $t0 = [ gettimeofday ];
$midi_ch->send( [ $t0, $ts, $msg ] );
}
);
sleep;
}
);
$loop->add( $midi_rtn );
The code block passed to the routine instantiates the MIDI device, opens a port on a connected MIDI keyboard, then sets a callback to send MIDI events from the keyboard back to the main program via the $midi_ch
channel. We also gettimeofday
to facilitate some instrumentation in the handler for the purposes of this example. Finally, the routine sleeps to allow RtMidi to effectively take control of it.
The handler may now simply await
MIDI messages from the keyboard:
async sub process_midi_events {
while ( my $event = await $midi_ch->recv ) {
say "recv took " . tv_interval( $event->[0] ) . "s";
say unpack 'H*', $event->[2];
}
}
The first line inside the while loop outputs roughly how long it took to be notified of and receive available MIDI data on the channel. The second simply outputs the hex representation of the received MIDI message.
The final step is to add some work to the loop so we can observe that it is not blocked by any of the above, then set it running:
my $tick = 0;
$loop->add( IO::Async::Timer::Periodic->new(
interval => 1,
on_tick => sub { say "Tick " . $tick++; },
)->start );
$loop->await( process_midi_events );
Sample output:
$ ./callback.pl
Tick 0
recv took 0.000653s
903c29
recv took 0.00056s
903c
Tick 1
Tick 2
recv took 0.000302s
903e2c
recv took 0.000324s
903e
Tick 3
recv took 0.000271s
904012
recv took 0.000288s
9040
Tick 4
...
We can see the ticks from our timer continue to display, interleaved with processed MIDI messages. The time between send and receive is sub-millisecond - this was run on a not-especially-high-spec virtual machine, so timings on real hardware are likely lower. A sub-millisecond delay is far below the limit of human perception, though it should be noted that every bit of latency added to a realtime system increases the chance of overall latency being perceived - that is, it all adds up.
The MIDI message itself is displayed as hex octets. Without diving into the MIDI spec, let's take a look at the first two messages:
Message 1 '903c29' - The byte 0x90
means a "note on" message on MIDI channel 1. The byte 0x3c
is the note that the "note on" event refers to - this translates to middle-C. The byte 0x29
is the velocity - how hard I pressed the key. This is a number between 0x00
and 0x7F
(or 0 and 127). This message means I pressed the middle-C key on the keyboard fairly softly.
Message 2 '903c' - This looks remarkably similar to the previous message - we are just missing the velocity value. A "note on" message with no (or zero) velocity should be interpreted as "note off". That is, I took my finger off the middle-C key.
The code for this example is on Github.
I can easily imagine using an approach like this for a program which performs dynamic MIDI processing while it receives realtime instructions from a performer, updates status visually, logs the performance to file, and so on. This might be more difficult working strictly within the RtMidi event loop, or attempting to pass aggregated statuses between more complete, though separate, heterogeneous event handlers.
There are a number of cases where this approach breaks down. For example if the routine-ified secondary event loop was designed to process a lot of data, passing that back and forth over a channel could be bad for guaranteeing timely responses. A similar situation could arise if your event handling has heavy CPU-bound processing. For integrating event systems with simple messages which don't work in a way that your existing event system expects, I can see myself using this pattern in future.
(Currently playing: Mission of Burma - Signals, Calls, and Marches)
With Perl version 5.36, released in May 2022, the signatures feature is no longer considered experimental and is enabled in that version bundle (i.e. if you use v5.36
, the signatures feature will be enabled).
use v5.36;
sub signatures( $hooray = undef ) {
say $hooray // 'Hooray!';
}
There is a simpler but related feature in Perl called prototypes, which allows you to declare the types of parameters though does not allocate names for these parameters. The default list @_
must still be unrolled. This feature is often used for compile-time parameter type checking, though it is more appropriately used for creating functions with builtin-style calling conventions. A common example is accepting a code block (rather than an anonymous sub) as a parameter, which allows calling without parentheses in the style of grep
or map
. This example will, perhaps unwisely, apply the block passed to transform
to the supplied list:
sub transform (&@) {
my $code = shift;
$code->( $_ ) for @_;
}
my @list = 1, 2, 3;
transform { ++$_ } @list; # @list now contains 2, 3, 4
If we naively upgrade this script by adding use v5.36
, this form of the prototypes feature will now throw a syntax error: "A signature parameter must start with '$', '@' or '%'
". So, how might one define a subroutine with a signature, which acts equivalently to a prototype? My very first thought was something like:
sub transform( &code, @list ) {
&code( $_ ) for @list;
}
... but as we see from the syntax error above &
is not one of the allowed sigils! This invocation also would not allow for the builtin-style calling convention without parentheses. You would need to call transform( sub { ... }, @list )
.
Thankfully an alternative mechanism exists for declaring prototypes, the prototype
attribute. A first pass at applying a prototype alongside a signature might look like this:
use v5.36;
sub transform :prototype(&@) ( $code, @list ) {
$code->( $_ ) for @list;
}
One problem remains. The @list
handled inside transform
is now a copy of the passed values, so operations in the block are not propagated to the contents of the original array. That is:
my @list = 1, 2, 3;
transform { ++$_ } @list; # @list still contains 1, 2, 3
We can use the prototype to coerce @list
to be an array ref - effectively a pointer to the passed array - and dereference this array ref within transform
. The complete code now looks like this:
use v5.36;
sub transform :prototype(&\@) ( $code, $list ) {
$code->( $_ ) for $list->@*;
}
my @list = 1, 2, 3;
transform { ++$_ } @list; # @list now contains 2, 3, 4
The old and the new are joined in harmony (Currently playing: Boards of Canada). I am certain this is all explained in documentation somewhere, but I need some excuse to update the site.
Update 5 Jan 2023 An update from Paul Evans on the possibility of parameter attributes in core states:
@fuzzix By default signature args are copies but there are vague thoughts ahead of one day doing an attribute for other behaviours. E.g.
sub transform :prototype(&@) ( $code, @values :alias ) { ... }
This is certainly interesting. It looks like attributes, rarely used outside frameworks like Catalyst, may be about to become a lot more powerful and useful.
Attributes are also being applied to fields in Corinna (Corinna is the ongoing effort to specify the upcoming feature 'class'
in core Perl). Paul is also working on the implementation for this and I am excited to see how it works out. If you want to play with these ideas now, check out Object::Pad.
Update 23 Aug 2023 A fork of clink by Chris Antos provides support for Windows 11, along with automagic updates and many additional quality-of-life features. The integration method outlined below still works, though you'll have to change the clink exe path to match the one installed on your system (in my case "C:\Program Files (x86)\clink\clink_x64.exe"
).
To my shame, I still haven't learned PowerShell... I still use CMD.EXE with the Readline features provided by Clink, along with a bunch of GNU tools in my path.
Integrating Clink into the Alacritty terminal emulator wasn't immediately obvious, so here'e my solution. Open %APPDATA%\alacritty\alacritty.yml
, find the shell section and set it to:
shell:
program: cmd
args:
- /k ""C:\Program Files (x86)\clink\0.4.9\clink_x64.exe" inject --profile %LOCALAPPDATA%\clink\"
These are some simple things you can do if tuning a static site for performance and download size.
There was a time when bundling js and css into minified blobs was a necessity for a performant site, but HTTP/2 multiplexing has made this redundant.
While encryption is not explicitly a requirement for HTTP/2, agents such as Firefox and Chrome only support HTTP/2 over HTTPS. If you don't have HTTPS configured, your first step may be to set up letsencrypt.
Enabling HTTP/2 then simply a matter of adding http2
to the listen directive in your static site config's server block:
listen 443 ssl http2;
listen [2a01:4f8:c2c:4783::c0:ffee]:443 ssl http2;
For text content, consider installing brotli alongside gzip. It offers higher compression ratios with similar CPU overhead to gzip. If you install nginx from FreeBSD ports, brotli should be a built-in config option. I also found a decent looking guide for installing ngx-brotli on Ubuntu. It differs from other guides in that it does not recommend rebuilding the whole of nginx.
As with HTTP/2, there is nothing inherent to brotli which requires HTTPS, but user agents tend not to request it on plain HTTP connections.
You'll need to enable the brotli modules in the main nginx.conf:
load_module "modules/ngx_http_brotli_filter_module.so";
load_module "modules/ngx_http_brotli_static_module.so";
Then enable brotli compression for the set of static filetypes you intend to serve:
brotli on;
brotli_static on;
brotli_comp_level 6;
brotli_types application/atom+xml application/javascript application/json application/rss+xml
application/vnd.ms-fontobject application/x-font-opentype application/x-font-truetype
application/x-font-ttf application/x-javascript application/xhtml+xml application/xml
font/eot font/opentype font/otf font/truetype image/svg+xml image/vnd.microsoft.icon
image/x-icon image/x-win-bitmap text/css text/javascript text/plain text/xml;
If gzip is not yet enabled, you should consider configuring that as well. It's as above, but s/brotli/gzip/g
.
You may have noticed the brotli_static on;
line in the above config. This enables optional on-disk compression of static assets. That is, if you are serving index.html and create index.html.br containing a brotli compressed version, this compressed file will be served. This has two advantages: nginx no longer has to compress content on-the-fly and you can use much higher compression ratios without impacting client performance.
I can't provide specific examples for static site generators, but the idea is that each time you update a file you should invoke a hook which creates compressed versions of that file. If this is a command-line job, the invocation you need is something like:
gzip -k -9 -f $file
brotli -k -Z -f $file
The -k
switch keeps the original - you'll need that for legacy agents. Maximum compression is set with -9
for gzip and -Z
for brotli (apparently it goes to 11, but this may be a funny joke). The -f
switch forces overwrite. I'm not sure how these tools operate from an atomicity perspective. That is, is there a risk you'll serve a partially compressed file? I don't worry about it.
The steps outlined here should help you serve a reasonably light and high-performance site from commodity hardware to modern user agents. If you want a starting point for further tuning, you can find out about configuring queue sizes, port ranges and other kernel-y bits in "Tuning FreeBSD for the highload" and "Tuning NGINX for Performance".
Have fun!
About time I got this site up and running again.
Writing my own static site generator was faster than learning to use Jekyll or whatever ;)
The code? It's a trade secret (i.e. shite).
Hey, does syntax highlighting work?
my $foo = 123;
sub bar( $foo ) {
die $foo;
}