![\"PipesAndFilter\"](\"https:\/\/www.modernescpp.com\/wp-content\/uploads\/2023\/04\/PipesAndFilter.png\")
<\/p>\nThe Pipes-and-Filters architecture pattern describes the structure of systems that process data streams.<\/p>\n
<\/p>\n
<\/p>\n
<\/p>\n
The Pipes-and-Filters pattern is similar to the Layers Pattern. The idea of the Layers Pattern is to structure the system in layers so that higher layers are based on the services of lower layers. The Pipes-and-Filters naturally extend the Layers Pattern, using the layers as filters and the data flow as pipes.<\/p>\n
![\"PipesAndFilterFlow\"](\"https:\/\/www.modernescpp.com\/wp-content\/uploads\/2023\/04\/PipesAndFilterFlow.png\")
<\/p>\n
![\"PipesAndFiltersStructure\"](\"https:\/\/www.modernescpp.com\/wp-content\/uploads\/2023\/04\/PipesAndFiltersStructure.png\")
<\/p>\n
Filter<\/b><\/code><\/pre>\n\n- Gets input data<\/li>\n
- Performs its operation on the input data<\/li>\n
- Produces output data<\/li>\n<\/ul>\n
Pipe<\/b><\/code><\/pre>\n\n- Transmits data<\/li>\n
- Buffers data in a queue<\/li>\n
- Synchronizes neighbors<\/li>\n<\/ul>\n
Data Source<\/b><\/code><\/pre>\n\n- Produces input to the processing pipeline<\/li>\n<\/ul>\n
Data Sink<\/b><\/code><\/pre>\n\n- Consumes data<\/li>\n<\/ul>\n
The most interesting part of the Pipes-and-Filter is the data flow.<\/p>\n
<\/span>Data Flow<\/span><\/h3>\nThere are several ways to control the data flow.<\/p>\n
Push Principle<\/h4>\n\n- The filter is started by passing the data of the previous filter<\/li>\n
- The (n-1)-th filter sends (write operation) data to the n-th filter<\/li>\n
- The data source starts the data flow<\/li>\n<\/ul>\n
Pull Principle<\/h4>\n\n- The filter is started by requesting data from the previous filter<\/li>\n
- The n-th filter requests data from the (n-1)-th filter<\/li>\n
- The data sink starts the data flow<\/li>\n<\/ul>\n
Mixed Push\/Pull Principle<\/h4>\n\n- The n-th filter requests data from the (n-1)-th filter and explicitly passes it to the (n+1)-th filter<\/li>\n
- The n-th filter is the only active filter in the processing chain <\/li>\n
- The n-th filter starts the data flow<\/li>\n<\/ul>\n
Active Filters as Independent Processes<\/h4>\n\n- Each filter is an independent process that reads data from the previous queue or writes data to the following queue<\/li>\n
- The n-th filter can read data only after the (n-1)-th filter has written data to the connecting queue<\/li>\n
- The n-th filter can write its data only after the (n+1)-th filter has read the connecting queue<\/li>\n
- This structure is known as the Producer\/Consumer<\/li>\n
- Each filter can start the data flow<\/li>\n<\/ul>\n
<\/span>Example<\/span><\/h3>\nThe most prominent example of the Pipes-and-Filters Pattern is the UNIX Command Shell.<\/p>\n
Unix Command Shell<\/h4>\n\n- \n
Find the five python files in my python3.6 installation that have the most lines:<\/p>\n<\/li>\n<\/ul>\n
![\"UnixPipe\"](\"https:\/\/www.modernescpp.com\/wp-content\/uploads\/2023\/04\/UnixPipe.png\")
<\/p>\n
Here are the steps of the pipeline:<\/p>\n
\n- Find all files ending with py:
find -name \"*.py\"<\/code><\/li>\n- Get from each file its number of lines:
xargs wc -l<\/code><\/li>\n- Sort numerical:
sort -g<\/code><\/li>\n- Remove the last two lines having irrelevant statistical information:
head -n -2<\/code><\/li>\n- Get the five last lines:
tail -5<\/code><\/li>\n<\/ul>\nFinally, here is the classic of command line processing using pipes from Douglas Mcllroy<\/a>.<\/p>\n<\/p>\n
\ntr -cs A-Za-z '\\n'<\/span> |\r\ntr A-Z a-z |\r\nsort |\r\nuniq -c |\r\nsort -rn |\r\nsed ${<\/span>1<\/span>}<\/span>q\r\n<\/pre>\n<\/div>\n <\/p>\n
If you want to know what this pipeline does, read the full story behind it in the article “More shell, less egg<\/a>“.<\/p>\nThanks to the ranges library in C++20, the Pipes-and-Filters Pattern is directly supported in C++.<\/p>\n
Ranges<\/h4>\n
The following program firstTenPrimes.cpp<\/code> displays the first ten primes starting with 1000.<\/p>\n<\/p>\n
\n\/\/ firstTenPrimes.cpp<\/span>\r\n\r\n#include <iostream><\/span>\r\n#include <ranges><\/span>\r\n#include <vector><\/span>\r\n\r\nbool<\/span> isPrime<\/span>(int<\/span> i) {\r\n for<\/span> (int<\/span> j =<\/span> 2<\/span>; j *<\/span> j <=<\/span> i; ++<\/span>j){\r\n if<\/span> (i %<\/span> j ==<\/span> 0<\/span>) return<\/span> false<\/span>;\r\n }\r\n return<\/span> true<\/span>;\r\n}\r\n\r\nint<\/span> main<\/span>() {\r\n\r\n std::<\/span>cout <<<\/span> '\\n'<\/span>;\r\n \r\n auto<\/span> odd =<\/span> [](int<\/span> i){ return<\/span> i %<\/span> 2<\/span> ==<\/span> 1<\/span>; };\r\n\r\n auto<\/span> vec =<\/span> std::<\/span>views::<\/span>iota(1<\/span>'<\/span>000<\/span>) |<\/span> std::<\/span>views::<\/span>filter(odd) \/\/ (1)<\/span>\r\n |<\/span> std::<\/span>views::<\/span>filter(isPrime) \/\/ (2)<\/span>\r\n |<\/span> std::<\/span>views::<\/span>take(10<\/span>) \/\/ (3)<\/span>\r\n |<\/span> std::<\/span>ranges::<\/span>to<<\/span>std::<\/span>vector><\/span>(); \/\/ (4)<\/span>\r\n\r\n for<\/span> (auto<\/span> v:<\/span> vec) std::<\/span>cout <<<\/span> v <<<\/span> \" \"<\/span>;\r\n\r\n}\r\n<\/pre>\n<\/div>\n <\/p>\n
The data source (std::views::iota(1'000)<\/code>) creates the natural number, starting with 1000. First, the odd numbers are filtered out (line 1), and then the prime numbers (line 2). This pipeline stops after ten values (line 3) and pushes the elements onto the std::vector<\/code> (line 4). The convenient function std::ranges::to<\/code> creates a new range (line 4). This function is new with C++23. Therefore, I can only execute the code with the newest windows compiler on the compiler explorer<\/a>.<\/p>\n![\"firstTenPrimes\"](\"https:\/\/www.modernescpp.com\/wp-content\/uploads\/2023\/04\/firstTenPrimes.png\")
<\/p>\n
<\/span>Pros and Cons<\/span><\/h3>\nI use in my following comparison the term universal interface. This means all filters speak the same language, such as xml or jason.<\/p>\n
<\/span>Pros<\/span><\/h3>\n\n- When one filter pushes or pulls the data directly from its neighbor, no intermediate buffering of data is necessary<\/li>\n
- An n-th filter implements the Layers Pattern and can, therefore, easily be replaced<\/li>\n
- Filters, implementing the universal interface, can be reordered<\/li>\n
- Each filter can work independently of the other and has not had to wait until the neighbored filter is done. This enables the optimal distribution of work between the filters.<\/li>\n
- Filters can run in a distributed architecture. The pipes connect the remote entities. The pipes can also split or synchronize the data flow. Pipes-and-Filters are heavily used in distributed or concurrent architectures and provide excellent performance and scalability opportunities.<\/li>\n<\/ul>\n
<\/span>Cons<\/span><\/h3>\n\n- The parallel processing of data may be inefficient due to communication, serialization, and synchronization overhead<\/li>\n
- A filter such as a sort needs the entire data<\/li>\n
- If the processing power of the filters is not homogenous, you need big queues between them<\/li>\n
- To support the universal interface, that data must be formatted between the filters<\/li>\n
- The most complicated part of this pattern is error handling. When the Pipes-and-Filters architecture crashes during the data processing, you have data that is not partially and fully processed. Now, you have a few options:\n
\n- Start the process once more if you have the original data.<\/li>\n
- Use only the fully processed data.<\/li>\n
- Introduce markers in your input data. You start the process based on the markers when your system crashes.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
<\/span>What’s Next?<\/span><\/h2>\nThe Broker structures distributed software systems that interact with remote service invocations. It is responsible for coordinating the communication, its results, and exceptions. In my next post, I will dive deeper into the architectural pattern Broker.<\/p>\n
<\/p>\n<\/p>\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
The Pipes-and-Filters architecture pattern describes the structure of systems that process data streams.<\/p>\n","protected":false},"author":21,"featured_media":6531,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[379],"tags":[413],"class_list":["post-6536","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-patterns","tag-ranges"],"_links":{"self":[{"href":"https:\/\/www.modernescpp.com\/index.php\/wp-json\/wp\/v2\/posts\/6536","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.modernescpp.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.modernescpp.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.modernescpp.com\/index.php\/wp-json\/wp\/v2\/users\/21"}],"replies":[{"embeddable":true,"href":"https:\/\/www.modernescpp.com\/index.php\/wp-json\/wp\/v2\/comments?post=6536"}],"version-history":[{"count":1,"href":"https:\/\/www.modernescpp.com\/index.php\/wp-json\/wp\/v2\/posts\/6536\/revisions"}],"predecessor-version":[{"id":8138,"href":"https:\/\/www.modernescpp.com\/index.php\/wp-json\/wp\/v2\/posts\/6536\/revisions\/8138"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.modernescpp.com\/index.php\/wp-json\/wp\/v2\/media\/6531"}],"wp:attachment":[{"href":"https:\/\/www.modernescpp.com\/index.php\/wp-json\/wp\/v2\/media?parent=6536"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.modernescpp.com\/index.php\/wp-json\/wp\/v2\/categories?post=6536"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.modernescpp.com\/index.php\/wp-json\/wp\/v2\/tags?post=6536"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}