Composite Pipes, part 2: Implementing Composite Pipes

Published September 20, 2019

After the refactoring of the pipes library we saw in the previous post, we’re in a situation where we have three concepts emulated with C++14 by the detection idiom: Range, Pipe and Pipeline.

This allowed us to write operator>>= with different combinations of parameters:

a Pipe and a Pipeline: add the pipe to the pipeline and return the resulting pipeline,
a Range and a Pipeline: send the elements of the range to the pipeline.

This allowed us in turn to write code like this:

myVector >>= pipes::filter([](int i) { return i % 2 == 0; })
         >>= pipes::transform([](int i){ return i * 2; })
         >>= pipes::push_back(results);

Today we’re going to create an new operator>>= allowing to make composite pipes, that is to say to combine pipes together into complex components, which can be associated to a pipeline later:

auto compositePipe = pipes::filter([](int i) { return i % 2 == 0; })
                 >>= pipes::transform([](int i){ return i * 2;});

myVector >>= compositePipe >>= pipes::push_back(results);

Starting point of the library

Here is a slightly more detailed summary of the previous episodes, that describes our starting point in the implementation of the library:

A Pipeline is a class that inherits from pipeline_base by passing itself:

template<typename Pipeline>
using IsAPipeline = std::enable_if_t<std::is_base_of<pipeline_base<Pipeline>, Pipeline>::value, bool>;

And a Pipe is something that we can tack on a Pipeline with the member function plug_to_pipeline:

struct aPipeline : pipeline_base<aPipeline>{};
template<typename Pipe>
using pipe_expression = decltype(std::declval<Pipe&>().plug_to_pipeline(std::declval<aPipeline&>()));

template<typename Pipe>
constexpr bool pipe_expression_detected = detail::is_detected<pipe_expression, Pipe>;

template<typename Pipe>
using IsAPipe = std::enable_if_t<pipe_expression_detected<Pipe>, bool>;

And for the purposes of the library, a Range is a class that has a begin and an end, as member functions or free functions in the same namespace as the class.:

namespace adl
{
    using std::begin;
    using std::end;
    template<typename T>
    using begin_expression = decltype(begin(std::declval<T&>()));
    template<typename T>
    using end_expression = decltype(end(std::declval<T&>()));
}

template<typename Range>
constexpr bool range_expression_detected = is_detected<adl::begin_expression, Range> && detail::is_detected<adl::end_expression, Range>;

template<typename Range>
using IsARange = std::enable_if_t<range_expression_detected<Range>, bool>;

`operator>>=`

We have so far two overloads of operator>>=. The one that sends the data of a range into a pipeline:

template<typename Range, typename Pipeline, IsARange<Range> = true, IsAPipeline<Pipeline> = true>
void operator>>=(Range&& range, Pipeline&& pipeline)
{
    std::copy(begin(range), end(range), pipeline);
}

And the one that associates a pipe to a pipeline, to create a new pipeline:

template<typename Pipe, typename Pipeline, IsAPipe<Pipe> = true, IsAPipeline<Pipeline> = true>
auto operator>>=(Pipe&& pipe, Pipeline&& pipeline)
{
    return pipe.plug_to_pipeline(pipeline);
}

Now we want to create an operator>>= that associates a pipe with another pipe.

Composite pipes

Since we want to associate it to a pipeline like any other pipe, our composite pipe must be a pipe. In other terms, we’re going to use the Composite design pattern. So by the above definition of a Pipe, a composite pipe must have a plug_to_pipeline function.

One way to implement composite pipes is with a class that stores two pipes and associate them both to an existing pipeline:

template<typename Pipe1, typename Pipe2>
class CompositePipe
{
public:
    template<typename Pipeline>
    auto plug_to_pipeline(Pipeline&& pipeline)
    {
        return pipe1_ >>= pipe2_ >>= pipeline;
    }
    
    template<typename Pipe1_, typename Pipe2_>
    CompositePipe(Pipe1_&& pipe1, Pipe2_&& pipe2) : pipe1_(FWD(pipe1)), pipe2_(FWD(pipe2)){}
private:
    Pipe1 pipe1_;
    Pipe2 pipe2_;
};

Note that to benefit from forwarding references in the constructor, we create artificial template arguments Pipe1_ and Pipe2_. But in practice we expect Pipe1_ and Pipe2_ to be equal to Pipe1 and Pipe2 respectively.

We need to do that because forwarding references require template parameters and, from the point of view of the constructor, Pipe1 and Pipe2 are not template parameters. Indeed, they have been determined when the whole template class was instantiated.

FWD(x) is the macro from Vittorio Romeo that expands to std::forward<decltype(x)>(x) and avoids burdening the code with technical constructs.

We can then use this class to implement the operator>>= between two pipes:

template<typename Pipe1, typename Pipe2, IsAPipe<Pipe1>, IsAPipe<Pipe2>>
CompositePipe<Pipe1, Pipe2> operator>>=(Pipe1&& pipe1, Pipe2&& pipe2)
{
    return CompositePipe<Pipe1, Pipe2>(FWD(pipe1), FWD(pipe2));
}

This works when pipe1 and pipe2 are initialized with rvalues. Indeed, with lvalues, Pipe1 and Pipe2 would be reference types. So the correct code is rather to use std::remove_reference_t to remove the potential references from the Pipe1 and Pipe2 types.

We can now create composite pipes and use them in a pipeline:

auto compositePipe = pipes::filter([](int i) { return i % 2 == 0; })
                 >>= pipes::transform([](int i){ return i * 2;});

myVector >>= compositePipe >>= pipes::push_back(results);

As a passing note, I think it would be more natural to call the composite pipe pipeline than compositePipe in the calling code:

auto pipeline = pipes::filter([](int i) { return i % 2 == 0; })
            >>= pipes::transform([](int i){ return i * 2;});

myVector >>= pipeline >>= pipes::push_back(results);

This is the sort of naming I would expect from calling code. But I’ve used compositePipe in this article to avoid confusion with what the implementation of the library calls a Pipeline, that is an assembly of pipes all the way to the last one (in our examples the last one is pipes::push_back).

This suggests that maybe Pipeline is not such a good name for the implementation of the library. Can you see a better name? If so, please let me know in a comment below.

Composite of composites

We’ve made composite of simple pipes, but CompositePipe can also contain composite pipes via its Pipe1 and Pipe2 parameters.

This is the idea of the Composite design pattern: both the simple elements and the composite ones have the same interface. Here this interface corresponds to being a Pipe, that is to say to have a plug_to_pipeline member functions that adds the object to a pipeline and returns the resulting pipeline.

CompositePipe therefore allows us to write this kind of code:

auto pipeline = pipes::filter([](int i) { return i % 2 == 0; })
            >>= pipes::transform([](int i){ return i * 2;});

auto pipeline2 = pipeline >>= pipeline; // <- composite of composites

myVector >>= pipeline2 >>= pipes::push_back(results);

Adding a feature to refactored code

Introducing composite pipes was a relatively easy thing. But that’s only for one reason: we had prepared the codebase with the refactoring described in the previous post.

It’s only because the concepts of the library were clarified and the components (here, operator>>= and the pipes) were decoupled that we could insert composite pipes. Introducing composite pipes in the library as it was before refactoring would have been laborious and would have likely resulted in complex code.

Here are the best practices we followed and that made this development easier:

paying attention to naming,
decoupling components,
using design patterns (here with the Composite design pattern),
separating the phase of refactoring from the implementation of the new feature.

If you’d like to see the whole implementation of the pipes library, please check out its Github repository. The previous link is the repo as it was after the development we just went through.

The repo evolves after that, to simplify the code even further, as I’ll explain in a future post. Stay tuned!