Smart Output Iterators >>= become(Pipes)

Published August 13, 2019

What DDD calls a refactoring breakthrough is when after making incremental changes to your codebase you suddenly realize that it would make more sense to represent the domain in a different way.

This new point of view allows to make a change on a large scale in the codebase, and that new structure seems to make everything click into place, and to make future tasks easier.

This is what happened with the Smart Output Iterators library. And the refactoring breakthrough is so big that it’s no longer even called Smart Output Iterators. The library is now called C++ Pipes.

Pipes allow to write expressive code when using collections in C++. Let’s see how that works.

Smart output iterators

If you haven’t heard about Smart Output Iterators, they are components that you can put in the output iterators of the STL algorithms. The C++ standard allows to put std::begin, or std::back_inserter at that output position, for example:

std::set_difference(begin(A), end(A),
                    begin(B), end(B),
                    std::back_inserter(C));

std::back_inserter receives data and passes it to the push_back method of its parameter C.

Smart output iterators go further into that direction, by adding logic to the output iterator. For example, applying a function f and passing the result on to another output iterator:

std::set_difference(begin(A), end(A),
                    begin(B), end(B),
                    transform(f) >>= std::back_inserter(C));

Or by filtering data with a predicate p:

std::set_difference(begin(A), end(A),
                    begin(B), end(B),
                    transform(f) >>= filter(p) >>= std::back_inserter(C));

Or by sending data to different directions:

std::set_difference(begin(A), end(A),
                    begin(B), end(B),
                    transform(f)
                    >>= filter(p)
                    >>= demux(std::back_inserter(C),
                              std::back_inserter(D),
                              transform(g) >>= std::back_inserter(E));

(Note: if you know demux from the previous version of the library, forget it, this is one of the evolutions that “clicked into place” with the refactoring breakthrough. Now demux merely sends the data it receives to each of its output branches. We’ll have a detailed post about the story of demux.)

The components evolved in numbers and capabilities, enough that it made sense to use them by themselves, without STL algorithms by using the to_output component:

A >>= to_output
  >>= transform(f)
  >>= filter(p)
  >>= unzip(back_inserter(B),
            demux(back_inserter(C),
                  filter(q) >>= back_inserter(D),
                  filter(r) >>= back_inserter(E));

In the above example, A is a range. That can be an STL container, a range from range-v3, or anything that has a begin and an end.

There is quite a lot more to it, but this is a good sample of the library.

But the same of the library, “Smart output iterators”, isn’t a very catchy one, is it?

If you’re part of my mailing list, you may have taken part of the reflexion around the name of the library (in case you did, thanks a lot!). And we realised that even shortening the name didn’t make it sound great.

This is where the refactoring breakthrough comes in. Smart output iterators are not a library about output iterators. It’s a library about plumbing.

Or at least, until the next refactoring breakthrough.

A library about plumbing

An interesting way to see the library is this: a source of inputs pours its data into the entrance of a pipeline. Each pipe in the pipeline receives pieces of data from the previous pipe, and sends them on to the next one(s), potentially modified.

The source of data can be an STL container, the output of an STL algorithm, or any range.

The pipeline is constituted of an assembly of individual pipe components.

The fact that pipes can be plugged into the output of an STL algorithm is no longer at the center of the library, as it was in the “smart output iterators” version. The pipes work together, and they happen to be pluggable to the output of STL algorithms too.

Example of pipes

For example, here are the pipes of the above example:

The transform pipe, that applies a function to its incoming pieces of data, and sends the results of that function application on to the next pipe:

transform pipe

The filter pipe, that passes on to the next pipe the incoming pieces of data that satisfy its predicate:

filter pipe

The unzip pipe, that breaks down pairs (and tuples) into individual values, and sends each of them to a different pipe:

unzip pipe

The demux pipe, that sends its incoming pieces of data to several pipes:

demux pipe

Sending data to the pipeline

In order to send each element of a range into the assembly of smart output iterators, we used the component called to_output.

Now we can rename this component, to represent that it allows to introduce data into pipes. What’s the word for something that funnels in fluids into a pipe? Well, a funnel.

So to_output is now called funnel:

funnel pipe

An assembly of pipes makes a pipeline

The previous example of code becomes:

A >>= funnel
  >>= transform(f)
  >>= filter(p)
  >>= unzip(back_inserter(B),
            demux(back_inserter(C),
                  filter(q) >>= back_inserter(D),
                  filter(r) >>= back_inserter(E));

And the mental representation we can have of it looks like this:

pipeline

A difference between pipes and ranges

In my very first article on smart output iterators, I compared them to range by opposing their positions relative to the STL algorithm. The ranges are the input of the algorithms, and the smart output iterators work on its output.

This property of smart output iterators remains true with pipes. However, another difference stands out between ranges and pipes: they don’t have the same design.

A range represents an iterable collection of data, potentially with multiple layers of range views on top of each other.

Pipes, on the other hand, are constructs that send data to each other.

I need your feedback

The pipes library is available in its GitHub repository.

Now that the library is more mature, I need more user feedback to make it grow. Would you like to try it out and give me your impressions?

From smart output iterators to pipes, in code

Now that we’ve seen the concept of pipes and the new orientation of the library, we’re going to see in the next post what it means in code to go from smart output iterators to pipes.

Then we’ll see what got unlocked by this refactoring breakthrough, in particular the demux iterator, that changed and led to the creation of a new pipe: the switch_ pipe. And we’ll see some more pipes.

What do you think about this transformation of the library? Do it seems more natural to you now? Do you have ideas for pipes that we could add to the library?

Let me know in the comments section below!