Adding Some Func to Go’s Flag Package

Friday, December 25, 2020

Go 1.16 is shaping up to be one of the most exciting releases of Go in recent memory. Among its new features is the //go:embed directive and the reorganization/deprecation of io/ioutil. It’s not due out until February 2021 (Edit: it came out Feb. 16, 2021), but now I would like to write about a minor change to the standard library flag package that might otherwise be missed in the sea of changes:

The new Func function allows registering a flag implemented by calling a function, as a lighter-weight alternative to implementing the Value interface.

I proposed and implemented flag.Func. I have made minor contributions to the Go project before, but this is the first time I’ve added a whole top level function to the standard library, so I’d like to explain a little about what it does and the process of getting it accepted into the Go standard library.

As background, the flag package parses command line flags, like myprogram -verbose -input file.txt -output out.txt. (It will actually accept either one dash or two, so myprogram --verbose --input file.txt --output out.txt is fine as well.) The flag package seems pretty basic at first, but actually has a powerful interface for enhancement, flag.Value:

// Value is the interface to the dynamic value stored in a flag.
// (The default value is represented as a string.)
//
// If a Value has an IsBoolFlag() bool method returning true,
// the command-line parser makes -name equivalent to -name=true
// rather than using the next command-line argument.
//
// Set is called once, in command line order, for each flag present.
// The flag package may call the String method with a zero-valued receiver,
// such as a nil pointer.
type Value interface {
  String() string
  Set(string) error
}

Using flag.Value, you can create your own flag handlers. For example, suppose you wanted to construct an command line option that accepts a URL. You could write a custom flag handler like this:

package main

import (
    "flag"
    "fmt"
    "net/url"
)

type URLValue struct {
    URL *url.URL
}

func (v URLValue) String() string {
    if v.URL != nil {
        return v.URL.String()
    }
    return ""
}

func (v URLValue) Set(s string) error {
    if u, err := url.Parse(s); err != nil {
        return err
    } else {
        *v.URL = *u
    }
    return nil
}

var u = &url.URL{}

func main() {
    fs := flag.NewFlagSet("ExampleValue", flag.ExitOnError)
    fs.Var(URLValue{u}, "url", "URL to parse")

    fs.Parse([]string{"-url", "https://golang.org/pkg/flag/"})
    fmt.Printf(`{scheme: %q, host: %q, path: %q}`, u.Scheme, u.Host, u.Path)
    // Output: {scheme: "https", host: "golang.org", path: "/pkg/flag/"}
}

This is very powerful but quite a bit verbose. For Go 1.16, I added flag.Func which simplifies it a bit:

// Func defines a flag with the specified name and usage string.
// Each time the flag is seen, fn is called with the value of the flag.
// If fn returns a non-nil error, it will be treated as a flag value parsing error.
func Func(name, usage string, fn func(string) error)

Behind the scenes, this is implemented as a trivial flag.Value:

type funcValue func(string) error

func (f funcValue) Set(s string) error { return f(s) }

func (f funcValue) String() string { return "" }

Here is the URL example again with flag.Func:

fs := flag.NewFlagSet("ExampleValue", flag.ExitOnError)

var u *url.URL
fs.Func("url", "URL to parse", func(s string) error {
    var err error
    u, err = url.Parse(s)
    return err
})

fs.Parse([]string{"-url", "https://golang.org/pkg/flag/"})
fmt.Printf(`{scheme: %q, host: %q, path: %q}`, u.Scheme, u.Host, u.Path)
// Output: {scheme: "https", host: "golang.org", path: "/pkg/flag/"}

It’s about 15 lines shorter and arguably more clear.

Here’s an example that uses flag.Func to parse an IP address:

fs := flag.NewFlagSet("ExampleFunc", flag.ContinueOnError)

var ip net.IP
fs.Func("ip", "`IP address` to parse", func(s string) error {
  ip = net.ParseIP(s)
  if ip == nil {
    return errors.New("could not parse IP")
  }
  return nil
})
fs.Parse([]string{"-ip", "127.0.0.1"})
fmt.Printf("{ip: %v, loopback: %t}\n\n", ip, ip.IsLoopback())

// 256 is not a valid IPv4 component
fs.Parse([]string{"-ip", "256.0.0.1"})
fmt.Printf("{ip: %v, loopback: %t}\n\n", ip, ip.IsLoopback())
// Output:
// {ip: 127.0.0.1, loopback: true}
//
// invalid value "256.0.0.1" for flag -ip: could not parse IP
// Usage of ExampleFunc:
//   -ip IP address
//      IP address to parse
// {ip: <nil>, loopback: false}

Over the years, I have made a lot of proposals. A lot of them go nowhere. That’s fine with me. In particular, I had previously proposed additions to the flag package: proposal: flag: Add flag.File, flag.FileVar. That proposal was rejected. The reason for the rejection given by Russ Cox was:

Note that you can write your own implementations of the flag.Value interface, so if you have specific semantics you want around this flag, and it’s common enough in your code, then you can implement your own FileFlag in your own library and use it. It doesn’t need to be in the standard library.
It’s too specialized, and there are too many details we’d need to expose user control over, to add to the standard library.

In retrospect, Russ was right. The standard library ought to be a collection of tools that are both generally useful and also straightforward to use. There are too many different nuances to file handling to try to make a convenience function for it.

Here’s what a file handler might look like with flag.Func:

input := os.Stdin
flag.Func("in", "`path` to file input (default STDIN)", func(s string) error {
    if s == "-" {
        return nil
    }
    f, err := os.Open(s)
    if err != nil {
        return err
    }
    input = f
    return nil
})
flag.Parse()

The code could be shorter if there were a flag.File helper, but the helper would have to make decisions to questions like—Is a path required? If not, what’s the default path? Or should it default to STDIN? Should it accept - as a synonym for STDIN? Should it return a file handle, an io.Reader, or a slice of bytes?—and provide an escape hatch to work around whatever the decisions were for users who wanted something else. flag.Func makes the code simple to understand and easy to extend as needed with a tolerable amount of boilerplate. Sometimes the simplest to use API is just plain old code.

Following the failure of my flag.File proposal, I ended up writing my own flagext library to extend the standard flag package with some custom types that I found useful as I needed them. In the course of working on this package, I noticed that it would be convenient to be able to define a flag.Value with just a single function, and wrote flagext.Callback as a convenience wrapper.

The proposal for flag.Func was similar to the failed proposal for flag.File. I was asking to add something to the Go standard library based on my own hunch about what would be useful to others. But the flag.Func proposal was also different in that it wasn’t addressing a specific need so much as a general one, and I already had my own real world experience using the code I was proposing adding. On top of that, I wasn’t the first person to notice that you could turn a function into a flag.Value either. Russ Cox pointed out that the Go tool itself had an internal type that did the same thing, but with a much worse name (objabi.Flagfn1!?). Russ’s suggestions on the issue helped improve the final code (I wanted to call it flag.Callback and include an unnecessary parameter). I was so eager to get the feature added that I ended up submitting the code even before the proposal was formally accepted.

Here are the morals I take away from my experience:

Don’t get turned off by an initial rejection. Most proposals get rejected. If I had gotten turned off by my earlier rejection, I wouldn’t have been able to make this proposal and see it to completion.
Learn from your mistakes. Contributing to a big open source project like Go can seem intimidating, but anyone can do it as long as they are willing to learn from their experiences. In this case, the rejection taught me about what it is that the Go team is looking for in a proposal.
Aim for proposals that are general and simple without being trivial. The Go standard library already covers a wide range of common needs. Think carefully about whether your idea would really benefit from being in the standard library or can stand on its own as a third party, open source library.
Proposals based on real world experience are more likely to be accepted. Releasing my own library for working with flags and using it over time in many personal and work projects gave me the experience to know better what was and wasn’t useful in a convenience method.

Until next time, stay funky, y’all.

The Ethically-Trained Programmer