To analysis log, you will need a log collectors, which is the hard working part function that will runs one server and pull metrics from them, parse log files and post them to the backend system like PostgreSQL and Elastic search. In today Logstash vs Fluentd article, we are going to give you information regarding what these data collectors can offer to you and if there is any how do they differ from each other. If you are interested to use them, go check our article below.
In this article, we are going to give you information about:
– What are Logstash and Fluentd
– What Logstash and Fluentd can offer to you
– Logstash vs Fluentd
About Logstash and Fluentd
Logstash is a part of the RLK, which is short for Elastic search, Logstash, and Kibana stack while the other Fluentd is built by Treasure Data, which is actually a part if Cloud Native Computing Foundation or CNCF portfolio of tools, which is used and continue to grow popularity in many DevOps-oriented communities. Before we are going further, if you prefer using Elastic products then you may want to stay with Logstash even though Fluentd also provide good support for Elastic.
Logstash and Fluentd Features
One of the biggest advantage of Logstash from long ago is because it is written in JRuby, so it will runs on Windows while at the other hand, Fluentd only support windows starting since not too long ago because its dependency on a *NIX platform centric event library. But, now both of them are equally support both Windows and Linux. In support part, unlike Fluentd with enterprises support, until now Logstash still doesn’t offer enterprise grade support.
In plugins, Logstash ecosystem is centralized while the other is not and in GitHub repo itself, there are more than 200 plugins for Logstash. At the other hand, you can find more than 500 Fluentd plugins but only some of them under the official repository, which is not necessarily the most popular among users.
Combined, there are hundreds of them and almost everything you can get data from like application logs, network protocols, container technologies, database, IoT devices, message protocols, orchestration engines, and many more. Another difference you may want to consider is because Logstash is based on JRuby, you will need to install Java while the other uses CRuby.
What set these two apart the most is probably the Transport technology, which means the act and fact of gather data from disparate sources and carrying this date to their correct destinations like a data lake, a database, an API, another application or any other similar place. Logstash is said to be lacking in internal persistent mechanism and only has an on-memory queue to holds a determined number of events while still relying on an external message queue for persistence across restarts and scalability, such as Kafka, ZeroMQ, Redis, and RabbitMQ.
However, this is a previous problem because as this article is written, Logstash persistent-queue feature already allowed persistence to disk for several point versions in its beta form but since it is now out of beta, users can choose to use persist-to-disk or in-memory queue. Read our previous article on Datadog vs Prometheus here.
At the other hand, Fluentd has highly configurable buffering system, which can be either in-memory or on-disk, so it can be said that in comparison with Logstash, this one has built-in reliability and scalability characteristic. However, it is not like without a downside because this ability also makes Fluentd difficult and tricky for beginner.
Another parts you may want to know is the agent-like tool from Elastic called Beats and it sends data to Logstash with minimal filtering capabilities, so it will send almost everything it records while at the other side Bit or Forwarder in Fluentd has a better filtering capabilities to help reducing the amount of data sent over the network. This may not important for some, but for those with data infrastructure deployed on hybrid or public clouds, every little bit send and received will be a cost.
In monitoring and Tuning, both of them produce logs that can be improved and equally offer a heartbeat output which can be used to make sure it works properly. Logstash will give you metrics filter to track report rate of all or specific processing chains and you can sent them to another tools like Graphite and visualize them using Grafana. It also give Restful monitoring API to help you understand resource consumption in acceptable detail level. At the other side, Fluentd offer you a built-in monitoring agent that can be queried for the status of specific monitoring plugins to enable integration with other monitoring stack you are currently using.
In performance, we can’t say which is better since user may have different opinion and experiences. But, looking in most some people opinion on the net, we conclude that Fluentd has a slightly better reputation in term of performance. Logstash will consume around 100-150 MG of RAM while the other only take 30-70 MB of RAM. It may not that meaningful but the difference between applications when you are deploying a whole data center can round up at tens or even thousands of additional RAM gigabytes and you have to pay them.
For IoT devices or small machines, Logstash uses a miniaturized agent to do a set of the product capabilities and Fluentd also have similar thing called Bit and Forwarder to do the same task. Some people who test these two found that they find some slowness on Logstash if compared to the other but it is not that prominent.
Now, let’s compare Logstash with Fluentd. Their biggest difference used to be on the transport technology from these two because Logstash used to be lacking in internal persistent mechanism, but as it has been upgraded now to allow persistence to disk or in the memory queue. Another prominent difference is in Plugins because they are managed differently; Logstash uses centralized repository while the other is decentralized.
Logstash vs Fluentd
|- Centralized plugins||- Decentralized plugins|
|- Doesn’t offer enterprise support||- Offer enterprise support|
|- Need Java||- Doesn’t need Java|
All in all, the decision is all yours to make. In our opinion, people can have difference experience depend on your punctual project needs. However, since Fluentd is written mostly in Ruby and its sensitive performance is written in C as well as with more convenient pre-compiled stable version, we will recommend you Fluentd.