Title: Building C#/.NET, Go, and Ruby Programs with libCLImate – Part 1: Ruby

Author: Bob Schmidt

Date: 05 May 2019 23:33:51 +01:00 or Sun, 05 May 2019 23:33:51 +01:00

Summary: Matthew Wilson demonstrates command-line processing.

Body: 

This article, the fourth in a series about software anatomy, describes some of my ongoing efforts to provide simple, succinct, easy, and, most importantly, consistent command-line argument processing across multiple languages. It, and the next one, build upon the observations of the first three articles, which pertained to building command-line interface (CLI) programs in C and C++, illustrating how the topic may be handled for the languages C#/.NET, Go, and Ruby, all of which have featured heavily in my career in the intervening years. As well as introducing readers to the respective Go, .NET and Ruby variants of the libCLImate and CLASP libraries, I will discuss how the features of these younger languages offer increased expressiveness and flexibility over C and C++, including how to work around some of their limitations.

Introduction

Command-line interface (CLI) programs often take arguments from the operating environment in order to control their behaviour. The shell breaks up the command-line (based on whitespace, except where quotes are operative) and presents this sequence of argument strings to the program. There are several, sometimes contradictory, conventions for how to classify arguments, including by the IEEE [1], GNU [2], Go [3].

Consider the command-line:

  $ count_slocs "C++ code" -D --terse     --log-file=./log1 '*.cpp' '*.hpp'

In this case, the program count_slocs would be passed six arguments, based primarily on the separating whitespace. There’s a label argument "C++ code" which is enclosed within double-quotes, which instructs the shell to treat all characters within (but not including) the double-quotes as a single argument. There are two arguments containing wildcards – '*.cpp' and '*.hpp' – that are each enclosed within single quotes, in order to tell the shell not to expand them. Each of these three arguments is just a value.

The remaining three arguments are different, insofar as they are prefixed with hyphens. The arguments -D and --terse are flags. The argument --log-file=./log1 is an option.

The terms flag, option, and value are my names for these concepts, as described in my article ‘An Introduction to CLASP, part 1: C’ [4]. The basic rules are pretty simple:

• if the argument does not begin with a hyphen (-) then it is a value; else
• if the argument contains an equals sign (=) then it is an option (and the parts before and after the equals sign are known as the option name and option value, respectively); else
• it is a flag.

There are, of course, complications, including:

• After the first occurrence of the special flag --, all arguments should be treated as values (regardless of any leading hyphen(s));
• Some program(mer)s prefer to be able to specify options separated by a space, e.g. the above option would alternatively be specified as --log-file ./log1. Naturally, for this to be recognised as two (separate argument) parts of a single option, something in the program has to know that --log-file is an option name, and that the next argument will the option value;
• Some program(mer)s prefer to be able to specify multiple flags (that have a single hyphen and a single letter) in combination so that, for example, the following command-line consisting of such flags:

    $ myprog -c -D -x

  can be expressed alternately as

    $ myprog -cDx

  and obtain precisely the same behaviour.

(There’s also a common convention to use the special flag "-" to mean standard-input, but that doesn’t affect parsing so I won’t mention it further.)

Obtaining command-line arguments

Different languages have different mechanisms for obtaining command-line arguments:

• C and C++ are provided the argument pair argc : int , argv : char*[] to the program entry point main();
• C#/.NET are provided an array of System.String to the program entry point Main();
• Python and Ruby do not have program entry points in the same way, and instead obtain the sys.argv property (a list) and global variable ARGV (an array), respectively;
• While Go does have a program entry point, it takes no arguments. The program-arguments are obtained from the exported array ([]string) os.Argv.

CLASP

CLASP stands for Command-Line Argument Parsing and Sorting. In essence, CLASP libraries parse command-lines to recognise flags, options, and values, and then presents them in a convenient set of sequences as part of a data structure representing the command-line (along with other useful information, such as program-name, and so forth).

The CLASP library concept started out with a C/C++ library, cunningly called CLASP, which was introduced to the world in [5] and further discussed in the three preceding parts of this anatomies series [4] [6] [7].

CLASP has subsequently been implemented in a number of different languages, including C#/.NET, Go, JavaScript, Python, and Ruby (all of which projects are available under https://github.com/synesissoftware).

As is the way of these things, different amounts of attention applied at different times, along with the strong influence of language limitations and conventions, have led to some differences, but fundamentally all perform the parsing and sorting (into flags, options, and values).

I found the use of CLASP to be a huge boon to productivity, not only because it provides often missing functionality, but also because having the same model in different languages means thinking time is kept to a useful minimum.

libCLImate

Despite the considerable advantages afforded by using CLASP, there are still tasks that one finds oneself performing again and again and again across projects. The most obvious ones are handling of the de facto standard --help and --version. But there are plenty of others, including:

• checking whether any unrecognised flags/options are received;
• checking whether any required flags/options have been specified;
• checking whether too many, or too few, values have been specified;
• issuing contingent reports of a consistent form, in particular that they begin with program-name :;

…and many more.

The situation begged for a higher-level library, implemented in terms of CLASP, that provides most/all of this behaviour.

In the third part of this series, ‘Building C & C++ CLI Programs with the libCLImate Mini-framework’ [7], I described in detail the design and implementation of the original libCLImate library, which provide many of these extra facilities for C/C++. Since that time (2015), I have continued to use the notion of a library providing boilerplate facilities that it is implemented in terms of CLASP, and have so far built and released variants for Go, C#/.NET, and Ruby. (It is reasonably likely that that Python version will be released by the time you’re reading this.)

(NOTE: after an exhaustive process of consultation with the esteemed members of the ACCU, the name libCLImate was accepted so that “every pull request will result in CLImate change”. Arf. Arf.)

As described in [7], due to the limitations of the runtime libraries the C/C++ libCLImate includes additional features and dependencies for diagnostics. Furthermore, because of the inherent limitations of the language, and because the library supports both C and C++, setting up many elements for a given program are still quite verbose. I plan soon to use some C++-11/14/17 features to address the second problem, and am optimistic that it can all be made simpler and more succinct in client code. However, the dependencies on the diagnostic libraries will likely stay, because they solve a still-glaring omission of the C/C++ environment.

Conversely, with the other languages, the implementation of libCLImate is much simpler, and the use of the library in programs is very succinct indeed, as I will shortly demonstrate. I think this has been because libCLImate.Ruby was the first (after the original), and I’ve noticed that when I write libraries in Ruby that subsequently are ported to other languages that the high expressiveness common in Ruby tends to get carried across.

libCLImate.Ruby

As mentioned above, since this is the oldest port, it’s also the most fully featured (and widely tested). Let’s consider how we might implement the notional program count_slocs from the introduction.

Basic boilerplate

The basic boilerplate of a libCLImate.Ruby program is as shown below:

  #! /usr/bin/env ruby
  require 'libclimate'
  options = {}
  climate = LibCLImate::Climate.new do |cl|
    # customise here
  end
  r = climate.parse ARGV

Even without customising it in any way, we get a fair amount of useful functionality:

• Support for --help:

  $ ./count_slocs --help

  yields the output:

  USAGE: count_slocs [... flags and options...]
  flags/options:
  --help
    shows this help and terminates
  --version
    shows version and terminates

• Policing any unrecognised flags:

  $ ./count_slocs -D

  yields the contingent report (and non-0 exit code):

  count_slocs: unrecognised flag '-D'; use --help for usage

• Policing any unrecognised options:

  $ ./count_slocs --log-file=./log1

  yields the contingent report (and non-0 exit code):

  count_slocs: unrecognised option '--log-file=./log1'; use --help for usage

Specifying version

However, if we run with --version we get a nasty surprise, as shown in Figure 1.

[...].../gems/clasp-ruby-0.16.0/lib/clasp/cli.rb:88:in
`generate_version_string_': options must specify :version or :version_major [ + :version_minor [ + :version_revision [ + :version_build ]]] (ArgumentError)
  from [...].../gems/clasp-ruby-0.16.0/lib/clasp/cli.rb:282:in
`show_version'
  from
[...]libCLImate/libCLImate.Ruby/trunk/lib/libclimate/climate.rb:179:in `show_version_'
  from
[...]libCLImate/libCLImate.Ruby/trunk/lib/libclimate/climate.rb:321:in `block in initialize'
  from
[...]libCLImate/libCLImate.Ruby/trunk/lib/libclimate/climate.rb:466:in `block in parse'
  from
[...]libCLImate/libCLImate.Ruby/trunk/lib/libclimate/climate.rb:438:in `each'
  from
[...]libCLImate/libCLImate.Ruby/trunk/lib/libclimate/climate.rb:438:in `parse'
  from ./count_slocs:9:in `<main>'
			

This is because we haven’t specified any version information. This can be done as a string, or an array of strings or integers (or mixed), as in:

  climate = LibCLImate::Climate.new do |cl|
    # customise here
    cl.version = [ 0, 0, 1 ]
  end

Now running with --version gives:

  count_slocs 0.0.1.alpha-1

Specifying program-specific flags

Now let’s add support for the flags -D and --terse. This is done via the Climate#add_flag method, which takes a name, an optional number of options, and an optional block to be executed if the flag is specified on the command-line (see Listing 1).

climate = LibCLImate::Climate.new do |cl|
  cl.version = [ 0, 0, 1, 'alpha-1' ]
  cl.add_flag('-D', help: 'runs in debug mode') {
    options[:debug] = true }
  cl.add_flag('--terse', 
    help: 'minimal program output') { 
    options[:terse] = true }
end
			

When either of these flags is specified on the command-line its respective presence is recorded in the options hash, for consumption in the program proper. The help strings are used for the --help flag, which I’ll show in full later.

Specifying program-specific options

Adding support for the --log-file option follows in the same vein (see Listing 2).

climate = LibCLImate::Climate.new do |cl|
  ...
  cl.add_flag('--terse',
    help: 'minimal program output') {
    options[:terse] = true }
  cl.add_option('--log-file',
    help: 'specifies the log-file') do |o|
    options[:log_file] = o.value or \
      cl.abort "no log-file specified"
  end
end
			

As you can see, this is very similar to adding a flag, except that the block takes a parameter of the actual parsed option, the value of which is placed into the options hash. If o.value is nil, which would happen if the last argument of the command-line is --log-file (without an =) an abort is executed, as in:

  $ count_slocs -D --terse --log-file

which yields the contingent report (and non-0 exit code):

  count_slocs: no log-file specified

If you also wished to reject an empty log-file (e.g. is passed --log-file=) then you could do:

  ( options[:log_file] = ( o.value ||'')).empty? 
  and cl.abort "no log-file specified"

Constraining values

Let’s now consider the values. In the example we had three values: the label, the implementation files filter, and the header files filter. Now obviously in the real world it’s more likely that rather than two filters there’d be one or more filters, but for now let’s pretend we want exactly two. As the program stands right now you can specify any number of values and it won’t care.

One way to do this is to obtain the values from the parse result, r, as in:

  r = climate.parse_and_verify ARGV
  unless 3 == r.values.size
    climate.abort "must specify three values; use    --h-help for usage"
  end

But that’s just a generic message, and you’re having to manually check. Instead, you can constrain the values, and give meaningful names to the values by customising the Climate instance (see Listing 3).

climate = LibCLImate::Climate.new do |cl|
  ...
  cl.add_option('--log-file', help: 'specifies
    the log-file') do |o|
    options[:log_file] = o.value or \
      cl.abort "no log-file specified"
  end
  cl.constrain_values = 3
  cl.value_names = [
    'label',
    'implementation files filter',
    'header files filter',
  ]
  cl.usage_values = '<label> <impl-filter>
  <hdr-filter>'
end
			

Now, if you run with, say, one argument:

  $ ./count_slocs "C++ code"

You’ll get the contingent report (and non-0 exit code):

  count_slocs: implementation files filter not
  specified; use --help for usage

And with two:

  $ ./count_slocs "C++ code" '*.cpp'

You’ll get the contingent report (and non-0 exit code):

  count_slocs: header files filter not specified;
  use --help for usage

Enhancing usage

If you take the advice and specify --help then you’ll get the output shown in Figure 2.

USAGE: count_slocs [ ... flags and options ... ]
<label> <impl-filter> <hdr-filter>

flags/options:
  --help
    shows this help and terminates
  --version
    shows version and terminates
  -D
    runs in debug mode
  --terse
    minimal program output
  --log-file=<value>
    specifies the log-file
			

This is reasonably good, but it can be better. We can specify some informational lines as shown in Listing 4.

climate = LibCLImate::Climate.new do |cl|
  ...
  cl.usage_values = '<label> <impl-filter>
    <hdr-filter>'
  cl.info_lines = [
    'libCLImate.Ruby example for CVu',
    nil,
    :version,
    nil,
  ]
end
			

Now --help gives the output shown in Figure 3.

libCLImate.Ruby example for CVu
count_slocs 0.0.1.alpha-1
USAGE: count_slocs [ ... flags and options ... ]
  <label> <impl-filter> <hdr-filter>
flags/options:
  --help
    shows this help and terminates
  --version
    shows version and terminates
  -D
    runs in debug mode
  --terse
    minimal program output
  --log-file=<value>
    specifies the log-file
			

Flag aliases

With the current definitions, we have to specify -D and --terse separately. However, CLASP (in all its guises) allows the combination of single-hyphen single-letter flags. This can be easily achieved by adding an alias property -t for --terse, as in:

  climate = LibCLImate::Climate.new do |cl|
  cl.version = [ 0, 0, 1, 'alpha-1' ]
  cl.add_flag('--terse', alias: '-t', help:
  'minimal program output') {
    options[:terse] = true }
  end

Now the two can be combined, as in:

  $ ./count_slocs -Dt ... other args ...

And this acts exactly as if they’d been supplied separately.

Option aliases

Options can have aliases two. For example, we could add an alias for --log-file to, say, -f, just in the same way as done for the flag above. This would allow for the long form expression seen already, as well as for a short(er) form as in:

  $ ./count_slocs -l ./log1 ... other args ...

But there’s a much more interesting way of using aliases with options. Consider that we change the flag --terse to an option --verbosity that can take a set of verbosity values and that, further, we have the flag -t instead be equivalent to specifying --verbosity=terse. This is supported by all versions of CLASP, and that filters through to use in libCLImate.Ruby (see Listing 5).

VERBOSITIES = %w{ silent terse normal chatty
  verbose }
climate = LibCLImate::Climate.new do |cl|
  ...
  cl.add_flag('-D', help: 'runs in debug mode') {
    options[:debug] = true }
  cl.add_option('--verbosity', help: 'specifies
    verbosity of program output', 
    values: VERBOSITIES) do |o|
      options[:verbosity] and \
        cl.abort 'already specified verbosity'
      options[:verbosity] = o.value
  end
  VERBOSITIES.each do |v|
    # === cl.add_alias('--verbosity=terse', '-t')
    cl.add_alias("--verbosity=#{v}", "-#{v[0]}")
  end
  ...
end
			

This code specifies an option, --verbosity, along with description string, a values list, and several combinable flag aliases, which would mean that the -Dt (and so forth) argument is still valid. And all this information is useful for the usage when applying --help (see Figure 4).

libCLImate.Ruby example for CVu
count_slocs 0.0.1.alpha-1
USAGE: count_slocs [ ... flags and options ... ]
   <label> <impl-filter> <hdr-filter>
flags/options:
  --help
    shows this help and terminates
  --version
    shows version and terminates
  -D
    runs in debug mode
  -s --verbosity=silent
  -t --verbosity=terse
  -n --verbosity=normal
  -c --verbosity=chatty
  -v --verbosity=verbose
  --verbosity=<value>
    specifies verbosity of program output
    where <value> one of:
      silent
      terse
      normal
      chatty
      verbose
  --log-file=<value>
    specifies the log-file
			

Obtaining results

In a real program, you will likely use blocks for flags and options as I’ve shown. But if you prefer, you can instead process them from the collections supplied in the parse result. For now, we’ll just print them out thusly:

  puts "flags (#{r.flags.size}):"
  r.flags.each { |f| puts "\t#{f}" }
  puts "options (#{r.options.size}):"
  r.options.each { |o| puts "\t#{o}" }
  puts "values (#{r.values.size}):"
  r.values.each { |v| puts "\t#{v}" }

With the command-line

  $ ./count_slocs -Dt --log-file=./log1 "C++ code"  '*.cpp' '*.hpp'

This yields the output:

  flags (1):
    -D
  options (2):
    --verbosity=terse
    --log-file=./log1
  values (3):
    C++ code
    *.cpp
    *.hpp

Final code

All, up all this code (excluding the printing of the flags/options/values collections) comes in at under 45 lines of code, as shown in full in Listing 6.

#! /usr/bin/env ruby
require 'libclimate'
VERBOSITIES = %w{ silent terse normal chatty
  verbose }
options = {}
climate = LibCLImate::Climate.new do |cl|

  # customise here
  cl.version = [ 0, 0, 1, 'alpha-1' ]
  cl.add_flag('-D', help: 'runs in debug mode') {
    options[:debug] = true }
  cl.add_option('--verbosity', help: 'specifies
    verbosity of program output',
    values: VERBOSITIES) do |o|
      options[:verbosity] and \
        cl.abort 'already specified verbosity'
      options[:verbosity] = o.value
  end

  VERBOSITIES.each do |v|
    cl.add_alias("--verbosity=#{v}", "-#{v[0]}")
  end

  cl.add_option('--log-file', required: true,
    help: 'specifies the log-file') do |o|
      options[:log_file] = o.value or \
        cl.abort "no log-file specified"
  end

  cl.constrain_values = 3
  cl.value_names = [
    'label',
    'implementation files filter',
    'header files filter',
  ]
  cl.usage_values = '<label> <impl-filter> 
    <hdr-filter>'

  cl.info_lines = [
    'libCLImate.Ruby example for CVu',
    nil,
    :version,
    nil,
  ]
end
r = climate.parse_and_verify ARGV

			

Next time

In the next instalment I’ll provide a much more chopped down presentation of equivalent programs in C# using libCLImate.NET and Go using libCLImate.Go, and then will go into details about the different facilities in the respective languages that help with the expressiveness and flexibility of such libraries. More importantly, perhaps, I will also go into those aspects in the languages that impede such, and how I have gone about addressing them with somewhat advanced features - anonymous structures and reflection in C#; extensions and monkey-patching in Ruby; strong typedefs and type-switches in Go - in order to have a consistent programmer and user experience across the different technologies.

For now, here’re three teaser trailers for next time.

More powerful option value processing in Ruby

As a little teaser for next time, permit me to illustrate how the verbosity values can be made even than bit more useable, to both user and programmer, by allow option value initials to be specified on the command-line and then automatically tested and converted into Ruby symbols in the code. Consider the changes in Listing 7.

require 'libclimate'
require 'xqsr3/extensions/string/
  map_option_string'
...

VERBOSITIES = %w{ [s]ilent [t]erse [n]ormal
  [c]hatty [v]erbose }
...
  cl.add_option('--verbosity', help: 'specifies
    verbosity of program output',
    values: VERBOSITIES) do |o|
      options[:verbosity] and cl.abort 'already
        specified verbosity'
      options[:verbosity] = 
        o.value .map_option_string(VERBOSITIES)
        or \
        cl.abort "invalid verbosity '#{o.value}';
        use --help for usage"
  end
  VERBOSITIES.each do |v|
    v = v.gsub /[\[\]]/, ''
    cl.add_alias("--verbosity=#{v}", "-#{v[0]}")
  end
  ...
			

This uses one of the String class extensions provided by the Xqsr3 library (another of mine - https://github.com/synesissoftware/xqsr3) to allow string values to be interpreted in full or via a contraction (the part inside the square brackets) against a known set of values such as VERBOSITIES, obtaining as the result a Ruby symbol. If not matched, nil is obtained, and the abort is fired. For example, the command-line

  $ ./count_slocs --verbosity=s ... other args ...

cause the options[:verbosity] value to be :silent, whereas the command-line

  $ ./count_slocs --verbosity=j ... other args ...

precipitates the contingent report (and a non-0 exit code):

  count_slocs: invalid verbosity 'j'; use --help
  for usage

count_slocs in Go

Listing 8 contains the equivalent program in Go (written in terms of libCLImate.Go and CLASP.Go).

package main
import (
  clasp "github.com/synesissoftware/CLASP.Go"
  libclimate "github.com/synesissoftware/
    libCLImate.Go"
  "fmt"
  "os"
  "strings"
)
const (
  verbosities =
   "silent|terse|normal|chatty|verbose"
)

func main() {
  debug := false
  verbosity := ""
  fl_Debug := clasp.Flag("-D").
    SetHelp("runs in debug mode")
  opt_Verb := clasp.Option("--verbosity").
    SetHelp("specifies verbosity of program
    output").
    SetValues(strings.Split(verbosities, "|")...)
  climate, err := libclimate.Init(func 
      (cl *libclimate.Climate) (error) {
    cl.AddFlagFunc(fl_Debug, func () {
      debug = true })
    cl.AddOptionFunc(opt_Verb, func 
      (o *clasp.Argument, a *clasp.Alias) {
        verbosity = o.Value
    })
    for _, v := range(strings.Split(verbosities,
      "|")) {
        cl.AddAlias("--verbosity=" + v,
          "-" + v[0:1])
    }
    return nil
  })
  if err != nil {
    fmt.Fprintf(os.Stderr, 
      "failed to create CLI parser: %v\n", err)
  }
  _, _ = climate.ParseAndVerify(os.Args)
  
  // Program-specific processing of flags/options
  if 0 != len(verbosity) {
    fmt.Printf("verbosity is specified as: %s\n",
    verbosity)
  }
  if debug {
    fmt.Printf("Debug mode is specified\n")
  }
  // Finish normal processing
  return
}
			

count_slocs in C#

Listing 9 contains the C# version.

namespace CVu_example
{
  using global::LibCLImate;
  using Clasp =
    global::SynesisSoftware.SystemTools.Clasp;
  using System;
  static class Program
  {
    static int Main(string[] args)
    {
      bool debug = false;
      string verbosity = null;
      Climate climate = Climate.Init((cl) => {
        cl.AddFlag("--debug"
        , new { 
          Alias = "-d",
          Help = "runs in Debug mode" }
        , (f, a) => {debug = true;}
      );

        cl.AddOption("--verbosity"
        , new
        {
          Alias = "-v",
          Help = "specifies the verbosity",
          Values = new string[] { "silent",
            "terse", "normal", "chatty",
            "verbose" },
          Default = "chatty",
          Required = true,
        }, (o, a) =>
        {
          verbosity = o.Value;
        });
        cl.AddAlias("--verbosity=chatty", "-c");

        cl.SetInfoLines(
          "libCLImate.NET examples",
          null,
          ":version:",
          null
        );
        cl.SetUsageValues("<path>");
      });

      var r = climate.ParseAndVerify(args);

      if(null != verbosity)
      {
        Console.WriteLine(
          "verbosity is specified as: {0}",
          verbosity);
      }

      if(debug)
      {
        Console.WriteLine(
          "Debug mode is specified");
      }
      return 0;
    }
  }
}
			

References

[1] The Open Group Base Specifications Issue 7, 2018 edition, section 12: Utility conventions, at http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html

[2] GNU manual: http://www.gnu.org/software/libc/manual/html_node/Getopt-Long-Options.html

[3] The Go Programming Language: https://golang.org/pkg/flag

[4] Anatomy of a CLI Program written in C, Matthew Wilson, CVu 24.4, September 2012.

[5] An introduction to CLASP, part 1: C, Matthew Wilson, CVu 23.6, January 2012

[6] Anatomy of a CLI Program written in C++, Matthew Wilson, CVu 27.4, September 2015.

[7] Building C & C++ Programs with the libCLImate Mini-framework , Matthew Wilson, CVu 27.5, November 2015

Matthew Wilson Matthew is a software development consultant and trainer for Synesis Software who helps clients to build high-performance software that does not break, and an author of articles and books that attempt to do the same.

Notes: 

More fields may be available via dynamicdata ..