Title: Exodep : A Simple External Dependency Refresher

Author: Bob Schmidt

Date: 05 November 2019 18:22:46 +00:00 or Tue, 05 November 2019 18:22:46 +00:00

Summary: Pete Cordell introduces a library dependency tool for C++ projects.

Body: 

A while back (May 2016), an email on the ACCU-General mailing list lamented the lack of third-party library download tools for C++. Recent tweets have echoed this deficiency. Languages such as Python and Ruby have tools like pip and rubygems, which are closely integrated with their respective ecosystems. A few attempts have been made to do the same for C++, such as biicode (now defunct) and conan.io [1], but so far it seems their adoption has been weak. Git submodules have also been used in this context, but often the reported experience is underwhelming.

For my own purposes, I work on a number of projects and have common bits of code used in each. Often these are small bits of code, such as string ends_with() level functionality or the legendary left-pad [2]. These often slowly collect functionality as subsequent projects need slightly different features. I also work on my desktop and laptop. So rather than have common libraries on each machine, I like to include the relevant library in each project’s repo. This makes moving from machine to machine easier and also means I know exactly what my customer has working on their systems when it comes to dealing with bug reports.

For a long time I struggled with manually copying code from the library development area to the relevant project. This was tedious with .h and .cpp files being in different directories. There was also the nagging concern that during the manual copying process I might accidently copy code in the wrong direction, copying an old version of code from the project area into the library development area.

This and the ACCU-General mailing list discussion spurred me on to see if I could come up with a solution at least for my needs, but maybe also one that benefits others.

The result is something I have called ‘exodep’, and the code is on Github [3].

Design goals

Before I started developing a solution I identified a number a design goals the solution should attempt to satisfy. These included:

• No Special Server Software Working both on a desktop and a laptop meant that central repos such as GitHub, BitBucket and GitLab were ideal places for storing the library code I was interested in. I also wanted to make it easy for other people to make code available. Further, I recognised that some people may want to use the solution internally on their own private systems. That meant the server had to be simple, requiring no more than an HTTP server or just a file share.

• ‘Democratic’ I wanted to avoid having a central registry where components are registered, such as with Ruby Gems. I didn’t want to have the responsibility of running such a registry, nor did I want to run the risk of a central registry losing funding and going off air. I hope the latter will remove a barrier to people thinking of investing their time and code into the approach. I only have a partial solution to this, but what I have I will discuss later.

• Exodep isn’t all there is When you dig into dependency management, you find it can get quite involved. For example, there is a vast range in the amount of code that may need to be downloaded. It could involve a single header file or something as big as Boost. Making exodep cater for all situations would complicate it. Therefore, I made the focus of exodep very much at the lower end, downloading low dozens of files. Equally, there is no revert or undo facility. The version control system can be used for that.

• Minimal Magic and Dumb Software, Smart People As mentioned, the complexities of dependency management can get involved. Solutions that automatically perform dependency management at run time or build time need to be smart and they potentially restrict how developers can structure their code. To allow a simpler tool I decided to use the smarts in developers and require dependency updates to be a human supervised activity. For example, typically this might involve performing a pull, refreshing dependencies, checking and fixing the build and pushing back. The left-pad saga [2] satisfied me that this was an acceptable situation. It also means you are able to dictate when updates are made to your code rather than the third-party software provider. My experience with Windows Updates has confirmed to me that that this approach was more than justifiable!

• Multiple dependent versions are too hard One of the harder problems in dependency management is two dependencies that each use incompatible versions of a third dependency. My clever solution to this problem was to decide that this was ‘not my problem’ and it is an issue for the third dependency to sort out. For example, where incompatibilities come into play, the third dependency should either be given a new name or have some mechanism to configure it in such as way that the two versions can co-exist, perhaps via clever namespace management.

• Source Only Some dependency tools are able to download binary versions of a library, after working out which particular build you need. My experience providing binaries for multiple platforms is that it’s a fool’s game. There are multiple versions of gcc, clang and Visual Studio. They can be built in debug and release modes. Visual Studio, at least, can have STL debugging enabled or not, use static or dynamic runtime and so on. You quickly end up with a combinatorial explosion. Supporting only source code download avoids this and also means a developer has full visibility of the code they are using (if they care to look). A side advantage is that exodep can be used on any source code, not just C++.

• Visibility For a number of tools, the ‘recipes’ are stored on a central server and only the name of the recipe is included in the local configuration. This makes it harder a-priori to know what will be downloaded. In line with the dumb software, smart people policy, I wanted the full recipe to be visible on the local machine. In addition to visibility, aside from allowing the developer to tweak and tune the download, it allows easier debugging of the update process. A side issue of this is that, if the build is not working properly due to a dependency not downloading properly, the first action should be to check that the locally stored recipe is up to date.

  A side benefit of this is that if you find a piece of code on the Internet that doesn’t have an exodep recipe, you can create one yourself. In fact, you can create your own library of exodep files referring to other code out in the Internet.

• Easy Setup I wanted to make it easy to add a new dependency to a project, especially minimising the amount of typing of convoluted identifiers required. Making this aspect simpler has been a recurring theme as the software has evolved.

• Readily Configurable Even though a dependency should be easy to set up, that should ideally not be at the expense of the developer being able to structure the resulting code to their liking. This should be possible without having to directly modify the recipe file downloaded for the library.

Implementation Language

The principle action required for exodep is an HTTP download. I could have gone a long way with a solution written in Bash and using wget. But this gave me two problems: (1) the recipe and implementation mechanics would be highly intertwined if included solely in a Bash script, and (2) I work on Windows. I decided a scripting language (rather than a compiled language like C++) was the way to go as this would give me cross-platform support ‘out-of-the-box’ and performance was unlikely to be affected by being script based. I am a fan of Ruby, so that was an option, but in the end I opted for Python3 as I believe this is more widely deployed and developers have more experience with it.

Simple Beginnings

A Github URL for a specific raw file in a repository has the following structure:

  https://raw.githubusercontent.com/{owner}/
  {project}/{branch}/{path}{file}

Bitbucket and Gitlab have similar formats.

To avoid having to repeat the bulk of the URL, having a URL template and some variable substitution was an obvious way to go. This led to an exodep file format that looked something like:

  uritemplate https://raw.githubusercontent.com/
  ${owner}/${project}/${strand}/${path}${file}
  $owner codalogic
  $project exodep
  get exodep.py

The command $owner codalogic sets the variable owner to codalogic. When executing the exodep get command, the ${owner} and ${project} fields in the uri template are substituted with the contents of the respective owner and project variables. Requiring braces around the substitution variable name allows the variable to be immediately followed by an alphanumeric character. The ${path} variable defaults to the empty string, but can be set by the user. The ${strand} field has some magic to it, but defaults to master. The ${file} field comes from the get command.

In practice the Github uri template is the default, so there’s no need to specify it if that’s what you want. The commands hosting bitbucket and hosting gitlab can quickly set up the uri template for those other hosting platforms.

This leads to simple recipes. For example, for Phil Nash’s Catch2 unit test library, a basic exodep recipe might look like this:

  $owner catchorg
  $project Catch2
  get single_include/catch2/catch.hpp

In practice you may not want your files being placed where the dependency dictates and I’ll come back to that.

The next consideration is where to place the recipe, or, as it turns out, recipes, that exodep reads. My first pass at implementing exodep consisted of including a file called mydeps.exodep in a projects top-level directory. This could then contain exodep include commands that would cause library specific exodep files to be processed. This approach cluttered up the top-level directory, which I didn’t like, so I opted to define a directory called exodep-imports and include the mydeps.exodep file therein. I quickly realised that rather than having to edit the mydeps.exodep file to add include statements for all the exodep files I wanted read, I could just do a file glob of all the .exodep files in the exodep-imports directory. Consequently, to add another dependency, all you have to do is download its exodep file in the exodep-imports directory.

A project that wishes to be used as an exodep source would list its .exodep files in the exodep-exports sub-directory within its online repo. So, to use a library, I would copy the .exodep files listed in the library’s exodep-exports sub-directory into my project’s exodep-imports sub-directory. The exodep file would be named after the project. For example, the exodep file for Catch2 might be called Catch2.exodep or catchorg.Catch2.exodep.

Software structure of library

The intention is that exodep doesn’t restrict how a developer lays out their source code files. That said, some structure is useful, as much as anything to avoid similarly named files from different libraries overwriting each other.

Languages like C++ traditionally divide their code into include files and source files sub-directories. This leads to default file names such as include/libname/file.h and src/libname/file.cpp.

The significance of this is that the #include statements in the C++ files work best with exodep when they have a form similar to:

  #include "libname/file.h"

The compiler can then have the set of include directories it uses configured so it can find the needed files. (Exodep has a subst command if this is considered too restrictive, but I’m hoping it won’t be used often.)

Customising the behaviour

The get single_include/catch2/catch.hpp command in the example above stores the downloaded file in the same location as specified in the source, i.e. single_include/catch2/catch.hpp. This doesn’t allow any input from the developer in terms of customisation. I wanted to have the option for the user to say where they want the file to be stored.

We could change the command to:

  get single_include/catch2/catch.hpp ${inc_dst}

But the developer may not want all include files stored in the same location. It’s better to have the command say something like ${Catch2_inc_dst}. The user then needs a way to set this variable while also allowing a sensible default.

Bearing in mind we want to include the project name in the path of the target location, the catch exodep file can set a default value for the ${Catch2_inc_dst} variable by doing:

  default $Catch2_inc_dst include/Catch2/

The default command says, if the variable is not set already, then use this value.

Now I needed to allow the user to set the $Catch2_inc_dst variable to something different. This is supported by the magic file exodep-import/__init.exodep. This is run prior to globbing the files in the exodep-import directory and allows setting values like the above and overriding the defaults set by the recipes.

When it comes to the user setting their own values, they may like the option to work at different levels of granularity. For example, they may want to set variables for each file that is downloaded. Or at the other extreme they may want to say ‘all external files should go into the ‘external’ sub-directory’.

To accommodate this, the exodep file for a dependency would have to include something like the following:

  default $extern_dst
  default $inc_dst ${extern_dst}include/
  default $Catch2_inc_dst ${inc_dst}catch/

This allows the user to modify any of $extern_dst, $inc_dst and $Catch2_inc_dst in the __init.exodep file. Modifying $extern_dst or $inc_dst would have an impact on all processed exodep files.

But this is tedious to set up and error prone. I only had about 3 or 4 exodep files to work with when I got fed up playing around with this sort of thing.

Instead I added the autovars command. When used after the $project variable is set, this creates a whole bunch of variables similar to that above. There are sets for include files and source files, plus sets for equivalent files used as part of testing.

With this the catch file would look something like:

  $owner catchorg
  $project Catch2
  autovars
  get single_include/catch2/catch.hpp
    ${Catch2_test_inc_dst}

Fixing build errors

To update dependencies, exodep.py runs through the exodep files that you have downloaded, vetted and stored locally. This can present a problem if the remote library has had new files added to it and the set of files that your local exodep file downloads is insufficient for the library to build. The authority command addresses this. It downloads the referenced remote exodep file and compares it to the local copy. If there are any differences, it reports an error prompting you to review and update the exodep files needed to build your project. The `authority` command uses the various variables identifying the project and the active uritemplate, so an example command might be:

  authority exodep-exports/myproject.exodep

A library may also use other libraries. The exodep files for these dependencies would also conveniently be stored in the exodep-exports of the library that uses them. Additionally, a project’s exodep file can contain uses commands that show the URL of other libraries that this library depends on. Exodep itself does not use the URLs in the uses commands. It is left to the developer to go into detective mode and track down the other exodep files needed.

Conditionals

Sometimes you may wish to modify the behaviour of a recipe depending on the environment in which it is being run.

For example, you may wish to do different things when used on Windows or Linux. Or you may wish to change the behaviour depending on which directories exist on the target system.

To support this, exodep allows conditional operation of commands. The format of a conditional command is <condition> <command>. If the condition is true, then the command is executed.

For an OS dependent download you may wish to download a .sln for Windows and a Makefile for Linux. This can be achieved using something like:

  windows get my-project.sln ${build_dir}
  linux get Makefile ${build_dir}
  osx get Makefile ${build_dir}

For the PHP scripts I have, I use something like:

  ondir htdocs default $php_dst htdocs/
  ondir httpdocs default $php_dst httpdocs/
  ondir wwwroot default $php_dst wwwroot/
  default $php_dst ./
  default $my_project_dst ${php_dst}

The ‘Registry’ – searching for code

Most dependency downloaders like pip and rubygems have a central repository that allows you to search for code that you want to use. After all, a dependency downloader that doesn’t have anything to download isn’t much use. But, as mentioned earlier, I want to avoid a central repository.

My solution to this is to use Google (other search engines are available). The idea is to include in a project’s repo short description a special tag name that is a combination of the word ‘exodep’ and the language of the library. For example, for C++ it would be ‘exodep_cpp’. By including this in the description of your Github repo, Google would be able to index it. You then prefix your Google search with the tag and hopefully you will find what you want. For example, you might search for “exodep_cpp unit test”. As there are currently very few ‘exodep_cpp’ tagged projects I haven’t yet been able to determine whether this will actually work!

Windows friendly(er)

The simplest way to update dependencies using exodep is to run the exodep.py program on the command-line in a shell opened at the relevant directory. This is great for Linux, but less so for Windows. To make things easier for Windows users I have created a simple installation file that adds a registry setting that adds a right-click option to Windows Explorer. After installing this, updating dependencies is as simple as right-clicking in the folder in Windows Explorer and selecting the ‘Run Exodep here’ option.

Summary

I recognise that exodep is at the toy end of the spectrum when it comes to the world of dependency tools (and open source projects in general). But it scratches an itch I’ve had for a number of years now and it’s been fun to develop. For me, it represents quite a good return for a mere 800 or so lines of Python (some of which could probably do with being deleted by now!).

Being so small, it is easy to update and extend. If you think it might help you, but you’d like other features, feel free to fork it and have a play.

If you have any libraries that you think may be of use to others, please consider adding an exodep-exports directory with a suitable exodep file and tagging the project description with a relevant search tag (e.g. ‘exodep_cpp’). Or build your own library of exodep files and share them.

References

[1] Conan, the C/C++ Package Manager for Developers:https://conan.io/

[2] Chris Williams (2016) ‘How one developer just broke Node, Babel and thousands of projects in 11 lines of JavaScript’, at https://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/ (published 23 Mar 2016)

[3] exodep on GitHub: https://github.com/codalogic/exodep

Pete Cordell started with V = IR many decades ago and has been slowly working up the stack ever since. Pete runs his own company, selling tools to make using XML in C++ easier.

Notes: 

More fields may be available via dynamicdata ..