Reducing the Code to Create a Pipe in the Pipes Library

Published October 1, 2019

After the various refactorings the pipes library went through, to define a pipe such as transform or filter we need to implement two classes: the pipe itself, and the class representing a pipeline starting with this pipe.

It would be nicer if implementing a pipe would solely require one class. That would make the code clearer, and would make it easier to add new pipes to the library.

Let’s refactor the library further to reduce the specific code of a pipe down to one class.

A reason to present this refactoring is because I found it very instructive and it helped me learn about code design. Indeed, this refactoring gives an illustration on how to improve code by defining responsibilities and separating out generic code from specific code.

The two classes needed for a pipe

Here is a simple example of a usage of pipes:

myVector >>= pipes::filter(f)
         >>= pipes::transform(p)
         >>= pipes::push_back(results);

The implementation of the transform pipe had two parts.

The first part represents the pipe itself, that is created with the transform function:

template<typename Function>
class transform_pipe
{
public:
    template<typename Pipeline>
    auto plug_to_pipeline(Pipeline&& pipeline) const
    {
        return transform_pipeline<Function, std::decay_t<Pipeline>>{function_, pipeline};
    }
    
    explicit transform_pipe(Function function) : function_(function){}

private:
    Function function_;
};

template<typename Function>
transform_pipe<std::decay_t<Function>> transform(Function&& function)
{
    return transform_pipe<std::decay_t<Function>>{function};
}

Its role is to store the function associated with transform, and to provide the plug_to_pipeline member function, that is called by operator>>=.

Since operator>>= is right-associative, the transform pipe in our above example is associated with the pipeline consisting of pipes::push_back(results).

This creates a transform_pipeline:

template<typename Function, typename TailPipeline>
class transform_pipeline : public pipeline_base<transform_pipeline<Function, TailPipeline>>
{
public:
    template<typename T>
    void onReceive(T&& input)
    {
        send(function_(std::forward<T>(input)), tailPipeline_);
    }

    explicit transform_pipeline(Function function, TailPipeline tailPipeline) : function_(function), tailPipeline_(tailPipeline) {}
    
private:
    Function function_;
    TailPipeline tailPipeline_;
};

The transform_pipeline in our case stores the function and the rest of the pipeline (here pipes::push_back(results)). When this pipeline receives a value, it applies the function on it and sends the result to the rest of the pipeline.

This is the existing design. Let’s improve it by rationalizing the classes.

Moving the specifics to the pipe class

If we want to reduce our pipe to one class, we need to define its responsibilities. To do that, we need to identify what is specific to the transform pipe in the above code.

There are two things specific to the transform pipe:

storing the function,
sending the result of applying the function to the rest of the pipeline.

The transform_pipe class is already storing the function. But it is the transform_pipeline class that sends the result of applying the function to the rest of the pipeline.

Let’s move this responsibility over to transform_pipe.

Refactoring works better when we do it in small steps. As a first step, let’s add an onReceive member function to the transform_pipe class, and make the onReceive function of the transform_pipeline class call it.

As a step even before that, let’s make the transform_pipeline hold a transform_pipe in order to call it later:

template<typename Function, typename HeadPipe, typename TailPipeline>
class transform_pipeline : public pipeline_base<transform_pipeline<Function, HeadPipe, TailPipeline>>
{
public:
    template<typename T>
    void onReceive(T&& input)
    {
        send(function_(std::forward<T>(input)), tailPipeline_);
    }

    explicit transform_pipeline(Function function, HeadPipe headPipe, TailPipeline tailPipeline) : function_(function), headPipe_(headPipe), tailPipeline_(tailPipeline) {}
    
private:
    Function function_;
    HeadPipe headPipe_;
    TailPipeline tailPipeline_;
};

Now let’s add the onReceive member function to transform_pipe:

template<typename Function>
class transform_pipe
{
public:
    template<typename Pipeline>
    auto plug_to_pipeline(Pipeline&& pipeline) const
    {
        return transform_pipeline<Function, std::decay_t<Pipeline>>{function_, *this, pipeline};
    }

    template<typename Value, typename TailPipeline>
    void onReceive(Value&& input, TailPipeline&& tailPipeline)
    {
        send(function_(std::forward<T>(input)), tailPipeline_);
    }
    
    explicit transform_pipe(Function function) : function_(function){}

private:
    Function function_;
};

Now we can call this function from transform_pipeline. As a result, transform_pipeline no longer need to store the function associated to transform:

template<typename HeadPipe, typename TailPipeline>
class transform_pipeline : public pipeline_base<transform_pipeline<HeadPipe, TailPipeline>>
{
public:
    template<typename T>
    void onReceive(T&& input)
    {
        headPipe_.onReceive(std::forward<T>(input), tailPipeline_);
    }

    explicit transform_pipeline(HeadPipe headPipe, TailPipeline tailPipeline) : headPipe_(headPipe), tailPipeline_(tailPipeline) {}
    
private:
    HeadPipe headPipe_;
    TailPipeline tailPipeline_;
};

Making the non-specific code generic

If we look at transform_pipeline now, we can notice that it doesn’t have anything left that is specific to transform. We can therefore rename it, for example generic_pipeline, and use it in a similar refactoring for all the other pipes, such as filter and the others. Let’s skip this part, for the purpose of brevity in the article.

We’re left with the member function plug_to_pipeline in transform_pipe, that doesn’t belong to the responsibilities of the transform pipe that we listed, which were:

storing the function,
sending the result of applying the function to the rest of the pipeline.

Now that the generic_pipeline doesn’t need anything specific from transform_pipe, plug_to_pipeline no longer has to been a member function. We can move its code to operator>>= itself:

template<typename Pipe, typename Pipeline, detail::IsAPipe<Pipe> = true, detail::IsAPipeline<Pipeline> = true>
auto operator>>=(Pipe&& pipe, Pipeline&& pipeline)
{
    return generic_pipeline<std::decay_t<Pipe>, std::decay_t<Pipeline>>{pipe, pipeline};
}

Redefining a pipe

But plug_to_pipeline was also used to define the pipe concept (emulated with the detection idiom):

struct aPipeline : pipeline_base<aPipeline>{};
template<typename Pipe>
using pipe_expression = decltype(std::declval<Pipe&>().plug_to_pipeline(std::declval<aPipeline&>()));

template<typename Pipe>
constexpr bool pipe_expression_detected = detail::is_detected<pipe_expression, Pipe>;

template<typename Pipe>
using IsAPipe = std::enable_if_t<pipe_expression_detected<Pipe>, bool>;

If we remove plug_to_pipeline, then we need something else to identify a pipe.

One way would be to use onReceive, but I didn’t manage to implement the detection idiom with a template function accepting any type of parameter:

    template<typename Value, typename TailPipeline>
    void onReceive(Value&& input, TailPipeline&& tailPipeline)
    {
        send(function_(std::forward<T>(input)), tailPipeline_);
    }

Do you have any idea how to detect that a class has such a template function? If so, I’d love to read your idea in a comment.

In the meantime, we’ll identify pipes by making them inherit from an empty base class, pipe_base:

template<typename Function>
class transform_pipe : public pipe_base
{
public:
    template<typename Value, typename TailPipeline>
    void onReceive(Value&& value, TailPipeline&& tailPipeline)
    {
        send(tailPipeline, function_(std::forward<Value>(value)));
    }
    
    explicit transform_pipe(Function function) : function_(function){}

private:
    Function function_;
};

We can now update the definition of Pipe accordingly:

template<typename Pipe>
using IsAPipe = std::enable_if_t<std::is_base_of<pipe_base, Pipe>::value, bool>;

Submitting new pipes to the library

After those successive improvements in the pipes library, it is know easier than ever to add a new pipe to the library.

What pipe would you like to see in the library? You can either let me know in a comment, or submit a PR yourself on the Github repository!

About Jonathan Boccara