Herding Camels

They say that no man is an island. Likewise, no software runs in a void. Well, except maybe for Voyager’s main control. But that’s not the same. And beside the point.

So, as I was saying, no software runs in a void. There are dependencies to think about. And depending of where you are in the overall stack, those can come in two flavors. There are, obviously enough, the dependencies that you are using, and there are the reverse dependencies; the other pieces of software that depends on your own.

Fortunately, testing is a very deeply ingrained characteristic of the Perl world. Modules come with their test suites, and the ever-vigilant, ever-running CPANtesters ensures that if a new release of a CPAN module breaks tests of another, authors are more likely than not to learn about it rather quickly.

That’s already mightily fine. But sometimes one needs more… custom arrangements. Recently I had such needs, and with the judicious use of already-existing tools I was able create a little setup that would not only allow me to test a selection of modules on my box, but also let me painlessly upgrade those modules when they’d change on CPAN.

My personal use case

As some of you might know, I’m part of the Dancer core crew. In the last few months, I’ve been cooking a new incarnation of its plugin architecture. Thoroughly new, very shiny, only smells faintly of madness — all in all, a spiffy affair. And then SawyerX moved in and added backward-compatible layer that should allow to transparently swap the old and squeaky Dancer2::Plugin with its new and improved version without any work from the plugin authors. Assuming this patch delivers on its promises, it would allow the gates of a glorious future to be boldly unlatched while the barn doors of backward-compatibility would remain wide open.

Awesome. But… does it?

One way to find out would be to push the changes to CPAN and see what happens. That’d be simple, but a mite… high-handed. Or we could release it as a trial release and ask people to test their plugins. Which would be nicer, but would require the concerted effort of all plugin authors.

No, really, in this case what we want is to test things locally, as often as we want, and with all the tweaks we might want to put in.

So let’s see how to do just that.

Drawing the list

First challenge: find all the modules of interest. For my specific case, since I know that all Dancer2 plugins will be within the Dancer2::Plugin::\* namespace, grepping through 02packages.details.txt (found in all good CPAN mirrors) would suffice. That’ll do it.

$ curl http://cpan.metacpan.org/modules/02packages.details.txt.gz \
    | zcat \
    | grep Dancer2::Plugin

Dancer2::Plugin::Ajax          0.200000  X/XS/XSAWYERX/Dancer2-Plugin-Ajax-0.200000.tar.gz
Dancer2::Plugin::AppRole::Helper 1.152121  M/MI/MITHALDU/Dancer2-Plugin-AppRole-Helper-1.152121.tar.gz
Dancer2::Plugin::AppRole::LogContextual 1.152121  M/MI/MITHALDU/Dancer2-Plugin-LogContextual-1.152121.tar.gz
Dancer2::Plugin::AppRole::LogContextualWarn 1.152121  M/MI/MITHALDU/Dancer2-Plugin-LogContextual-1.152121.tar.gz
[..]

It works. But it’s crude, limited, and, I’m sure you’ll agree, not fun.

For a more general approach, we can leverage the MetaCPAN database. It holds information on all CPAN modules, including nifty things like declared dependencies. Better, it’s ElasticSearch-powered and accessible via a REST interface. Betterer still: there’s even a module, MetaCPAN::Client, to interface with that interface.

For my plugins, I can use this script,

#!/usr/bin/env perl

use 5.10.0;

use MetaCPAN::Client;

my $result = MetaCPAN::Client->new->distribution( { name => 'Dancer2-Plugin-*' });

say $_->name while $_ = $result->next;

and it will gives me

$ perl get_plugins.pl

Dancer2-Plugin-ParamKeywords
Dancer2-Plugin-Ajax
Dancer2-Plugin-Queue-MongoDB
Dancer2-Plugin-SendAs
Dancer2-Plugin-Locale
Dancer2-Plugin-BrowserDetect
Dancer2-Plugin-Multilang
Dancer2-Plugin-Paginator
Dancer2-Plugin-Sixpack
Dancer2-Plugin-Chain
Dancer2-Plugin-Growler

Beautiful. And if I ever want to draw a different list, I only need to change my query. Want all the Dancer2 plugins I’ve authored in the last year? No sweat:

use 5.10.0;

use List::MoreUtils qw/ uniq /;
use MetaCPAN::Client;

my $result = MetaCPAN::Client->new->all('releases', {
    es_filter => {
        and => [
            { range => { 'release.date' => { 'gte' => '2015-01-01T00:00:01' } } },
            { term => { 'release.author' => 'YANICK' } },
        ]
    }
});

my @distro;

while( my $rel = $result->next ) {
    push @distro, $rel->distribution;
}

say for uniq sort grep { /^Dancer2-Plugin-/ } @distro;

Getting local copies

Now that we have the list of modules we want, we want to get a local copy of all of them. We could download their latest CPAN releases, but it’ll be a pain to keep in sync if our testing takes a while and the modules are updated on CPAN.

Instead, we’ll use Git::CPAN::Patch. This module adds a few git subcommands. Amongst them, git cpan clone, which creates a local Git repository with the best history it can get from a module: if a Git repository is given in the module’s META information, that’s what it will use, and if not it’ll build a history based on the CPAN releases of that module. Working along that first command is git cpan update, which is smart enough to know which type of source we used and detect/import updates.

So, to get my updatable mirror of all plugins, we do

$ cat d2_plugins | xargs git cpan clone

and watch all the repos come to life.

Well… that’s the basic version. As it happens, in the file d2_plugins listing all the plugins, I commented out some plugins causing problems. And I only care about the current CPAN version of the plugins. And I wanted xargs to soldier on even if some plugins couldn’t be cloned (xargs stops at the first error it meets), so I ended up doing

$ cat d2_plugins | \
    grep -v '#'  | \
    xargs  -IX -- sh -c "git cpan clone --norepository --latest  X || true"

Wrangling 'em

Next, it’d be nice to have all those repositories in a kind of group so that we can, for example, run a command for all of them.

For that, we’ll be using App::GitGot.

Assuming that we are in the directory where all the cloned plugin repositories are, loading them all into Got and tagging them as members of the d2plugins family is done via

$ got add --tag d2plugins -D --recursive .

And now that we have all those repos loaded and tagged in got, we can do whatever we want with them. Updating them all using the afore-mentioned git cpan update is done via

$ got do --tag d2plugins --command "git cpan update"

Of course, that’s something we could have done via xargs. But got gives us further niceties, like getting a status on all repositories (which will come in handy if we begin to patch plugin code), further refining tagging (like d2_read, d2_needwork, d2_hopeless), moving the repositories anywhere on our disk, etc.

Run the tests. All of them

We have the list of plugins, we have a copy of them locally. We’re set. Our final step: running all test suites. First with the CPAN-installed Dancer2.

$ got do --tag d2plugins \
    --command "echo -n (git config --get cpan.module-name) \
        ' '; prove -l t > /dev/null 2> /dev/null           \
        && echo PASSED || echo FAILED"  | perl -ne'print if /PASSED|FAILED/'

Dancer2-Plugin-Adapter  PASSED
Dancer2-Plugin-Ajax  PASSED
Dancer2-Plugin-AppRole-Helper  PASSED
Dancer2-Plugin-Articulate  PASSED
Dancer2-Plugin-Auth-Extensible  PASSED
Dancer2-Plugin-Auth-Extensible-Provider-DBIC  FAILED
Dancer2-Plugin-Auth-Extensible-Provider-Usergroup  PASSED
Dancer2-Plugin-Auth-HTTP-Basic-DWIW  PASSED
Dancer2-Plugin-Auth-OAuth  PASSED
Dancer2-Plugin-Auth-Tiny  PASSED
...

For the sharp-eyed and curious amongst you who wonder where that cpan.module-name git variable is coming from: git cpan clone populated it for us, because it’s just nice that way.

Now we can run all those test suites with the modified plugin module:

$ export PERL5LIB=/home/yanick/work/perl-modules/dancer/Dancer2/lib
$ got do --tag d2plugins \
    --command "echo -n (git config --get cpan.module-name) \
        ' '; prove -l t > /dev/null 2> /dev/null           \
        && echo PASSED || echo FAILED"  | perl -ne'print if /PASSED|FAILED/'

Dancer2-Plugin-Adapter  FAILED
Dancer2-Plugin-Ajax  FAILED
Dancer2-Plugin-AppRole-Helper  PASSED
Dancer2-Plugin-Articulate  PASSED
Dancer2-Plugin-Auth-Extensible  FAILED
...

Enjoying the bountiful harvest

The pipeline that we built here is a perfectly satisfying scratchpost for my initial need. I now have most of the Dancer2 plugins cloned locally, and I can gather the results of their test suites with different versions of the core Dancer2 libraries. Even better, everything is set in such a way that, when I return to this project in a few weeks, I’ll be able to upgrade those plugins to their latest versions instead of having to start all over from zero.

Is this a set-up that many of us will need verbatim? No, not really. For most of our needs, CPANTesters and Travis are still good enough. But for more finicky projects involving a disparate smattering of repositories, we now have a few toys — Git::CPAN::Patch, App::GitGot, MetaCPAN::Client, and of course Git itself — that can be taken out of the toolbox, duct-taped together, and get the jolly job done.

Tags: technology perl git