How to Create Your Own C++ Pipe
In this post we’re going to go through a simple example of pipe to add to the C++ pipes library: the tee pipe. This example serves as a tutorial to add a pipe to the library, if you’d like to add one and submit a pull request.
We’re going to see:
- How to write a test for a pipe,
- How to implement a pipe.
If after this tutorial you still have questions to implement your pipe and submit it to the library, don’t hesitate to contact me.
The tee pipe
A tee is an object that has the shape of a T. For example a T-shirt is called a tee, or the little T-shaped thing you put golf balls on before shooting them off with a club is also called a tee.
tee is also a UNIX program that produces the same output as its input, and copies that input to a file too.
In the same spirit, we’re going to design a tee pipe, that receives data from any other pipe, and sends it on both to the next pipe and to another output:
As with everything, we get the best interfaces when we start by writing the calling code and only then write the interface and implementation to make that calling code work.
The desired calling code for our tee is this:
inputs >>= pipes::transform([](int i){ return i * 2; })
>>= pipes::tee(pipes::push_back(intermediaryResults))
>>= pipes::filter([](int i){ return i > 10; })
>>= pipes::push_back(results);
Let’s start by putting that desired code into a test.
Writing a test
The thing to do even before that is to compile the existing tests of the library, to make sure they all compile and pass on your environment.
Synchronise the git repo on your machine:
git clone https://github.com/joboccara/pipes.git
Generate the tests project:
mkdir build cd build cmake ..
Then compile the code:
make
And run the tests:
tests/pipes_test
If all is well, you should see something like this in the console output:
=============================================================================== All tests passed (109 assertions in 58 test cases)
Note that there may be a different number of tests when you run the library, what matters is that they all pass.
Adding a new test
Once the existing tests pass on your environment, we can move on and add new tests for the new pipe.
The unit tests of the pipes library are in the tests/ directory. Let’s create a new file, tee.cpp in this tests/ directory.
The library uses Catch 2 as a testing framework, so you need to add this include in the test file:
#include "catch.hpp"
Important note: you also need to add the new test file to the CMakeLists.txt file of the tests/ directory. To do that, add the file name in the list of files of the add_executable command.
Here is the code to test the above desired syntax for our tee pipe:
#include "catch.hpp"
#include "pipes/filter.hpp"
#include "pipes/tee.hpp"
#include "pipes/transform.hpp"
TEST_CASE("tee outputs to the next pipe as well as the one it takes in argument")
{
auto const inputs = std::vector<int>{1, 2, 3, 4, 5, 6, 7, 8, 9, 10};
auto const expectedIntermediaryResults = std::vector<int>{2, 4, 6, 8, 10, 12, 14, 16, 18, 20};
auto const expectedResults = std::vector<int>{12, 14, 16, 18, 20};
auto intermediaryResults = std::vector<int>{};
auto results = std::vector<int>{};
inputs >>= pipes::transform([](int i){ return i * 2; })
>>= pipes::tee(pipes::push_back(intermediaryResults))
>>= pipes::filter([](int i){ return i > 10; })
>>= pipes::push_back(results);
REQUIRE(results == expectedResults);
REQUIRE(intermediaryResults == expectedIntermediaryResults);
}
We’re doing a pipeline of two steps, a transform and a filter, and we’re inserting a tee in between to capture the intermediary results.
When we run that test, it doesn’t compile… because we haven’t designed the tee pipe yet!
If there are other cases, in particular edge cases, you want to cover with your pipe, you can add more tests.
Implementing the pipe
Now let’s implement the pipe itself, in a tee.hpp file in the include/pipes/ directory.
To implement a pipe we need to implement two classes:
- the pipe itself:
tee_pipe, - the pipeline starting with this pipe:
tee_pipeline.
tee_pipe
tee_pipe is the object that is created when we write pipes::tee(pipes::push_back(intermediaryResults)):. Here is the tee function:
template<typename TeeBranch>
tee_pipe<TeeBranch> tee(TeeBranch const& teeBranch)
{
return tee_pipe<TeeBranch>{teeBranch};
}
This tee_pipe then gets associated with operator>>= to the rest of the pipeline after it or said differently, to the tail of the pipeline. This association produces a tee_pipeline.
The pipe doesn’t directly implement operator>>=, it is done in generic code. Rather, it is required to implement a member function plug_to_pipeline that describes how to associate a tee_pipe with the tail of the pipeline:
template<typename TeeBranch>
class tee_pipe
{
public:
template<typename Pipeline>
auto plug_to_pipeline(Pipeline&& pipeline) const
{
return tee_pipeline<TeeBranch, std::remove_reference_t<Pipeline>>{teeBranch_, pipeline};
}
explicit tee_pipe(TeeBranch teeBranch) : teeBranch_(teeBranch){}
private:
TeeBranch teeBranch_;
};
If you’re wondering about the std::remove_reference_t on line 8, it is necessary because in the forwarding reference Pipeline&&, Pipeline might be a reference type (in the case where it gets an lvalue).
tee_pipeline
A tee_pipeline can receive data, send it both to the tee branch and the rest of tail of the pipeline. tee_pipeline contains both the tee branch and the tail of the pipeline.
Here is the code of tee_pipeline. It might look a little scary at first, but we will analyse it line by line just after. It’s just an assembly of simple things:
template<typename TeeBranch, typename PipelineTail>
class tee_pipeline : public pipeline_base<tee_pipeline<TeeBranch, PipelineTail>>
{
public:
template<typename T>
void onReceive(T&& value)
{
send(teeBranch_, value);
send(pipelineTail_, FWD(value));
}
tee_pipeline(TeeBranch const& teeBranch, PipelineTail const& pipelineTail) : teeBranch_(teeBranch), pipelineTail_(pipelineTail){}
private:
TeeBranch teeBranch_;
PipelineTail pipelineTail_;
};
Let’s analyse this code, so that you can adapt it for your pipe.
Inheriting from pipeline_base
Let’s start with the beginning of the class:
template<typename TeeBranch, typename PipelineTail> class tee_pipeline : public pipeline_base<tee_pipeline<TeeBranch, PipelineTail>>
The pipeline must derive from the CRTP base class pipeline_base. To follow the CRTP pattern, we pass the class itself as template parameter of pipeline_base.
Deriving from pipeline_base allow the generic features of the library to access your pipe. Those features include the various forms of operator>>= and the integration with STL algorithms.
The specific part of your pipe
template<typename T>
void onReceive(T&& value)
{
send(teeBranch_, value);
send(pipelineTail_, FWD(value));
}
This is the main method of your pipe. It gets called when a pipe further up in the pipeline sends data to your pipe. In our case, we want to forward that data both to the tail of the pipeline and to the tee branch. To send data to a pipeline, we use the function pipes::send.
This method has to be called onReceive, because it is called by the CRTP base class.
The library provides the FWD macro, that expands to std::forward<T>(value) here. The FWD macro is available in the pipes/helpers/FWD.hpp header. If you’re not familiar with std::forward and forwarding references (T&&), you can catch up with this refresher.
tee_pipeline(TeeBranch const& teeBranch, PipelineTail const& pipelineTail) : teeBranch_(teeBranch), pipelineTail_(pipelineTail){}
private:
TeeBranch teeBranch_;
PipelineTail pipelineTail_;
This code allows the tee_pipeline to be constructed with its two outputs. This is the code that gets called in the plug_to_pipeline method of the tee_pipe class we saw above.
operator>>=
To make your new pipe compatible with operator>>=, you need to add this header to your file:
#include <pipes/operator.hpp>
This headers contains the definition of operator>>=. By including it in your pipe header file, you will make sure that users of your pipe also benefit from its operator>>=.
Testing operator=
The STL of Visual Studio in the _Recheck function of the debug mode calls operator= on an output iterator on itself, by passing it an lvalue reference.
So we need to write a test for operator=:
TEST_CASE("tee operator=")
{
std::vector<int> results1, results2, results3, results4;
auto tee1 = pipes::tee(pipes::push_back(results1)) >>= pipes::push_back(results2);
auto tee2 = pipes::tee(pipes::push_back(results3)) >>= pipes::push_back(results4);
tee2 = tee1;
pipes::send(tee2, 0);
REQUIRE(results1.size() == 1);
REQUIRE(results2.size() == 1);
REQUIRE(results3.size() == 0);
REQUIRE(results4.size() == 0);
}
Launch the tests. They should be green.
Let’s add new pipes!
Now if you have an idea for a new pipe, you can either let me know or add it yourself by following this tutorial and make a PR on the Github repository.
If there is anything you see that would make this tutorial more convenient, don’t hesitate to let me know.
Together let’s make new pipes, and create pipelines to write expressive code to manipulate data in collections!
You will also like
- Smart Output Iterators >>= become(Pipes)
- Making C++ Pipes Compatible with STL Algorithms
- Smart Output Iterators: A Symmetrical Approach to Range Adaptors
- How Smart Output Iterators Avoid the TPOIASI
Share this post!
