Log monitoring project using Grafana Loki and Promtail

Log monitoring using Grafana Loki and Promtail is a powerful combination that allows you to collect, store, and analyze log data in real-time.

What is Grafana?

Grafana is an open-source data visualization and monitoring tool. It is designed to help users understand and analyze their metrics, logs, and other data through customizable dashboards and intuitive graphical representations.
With Grafana, you can connect to various data sources such as databases, cloud platforms, and monitoring systems, allowing you to pull in real-time and historical data. It supports popular databases like MySQL, PostgreSQL, and Prometheus, as well as cloud platforms like Amazon Web Services (AWS) and Microsoft Azure.
One of Grafana's key features is its ability to create interactive and dynamic dashboards. Users can build visually appealing dashboards by dragging and dropping various panels, which can include graphs, charts, tables, and other visual elements. These panels can be configured to display metrics, logs, or any other data you wish to monitor or analyze.
Grafana also provides extensive customization options, allowing users to fine-tune the appearance and behavior of their dashboards. You can set up alerts and notifications based on specified thresholds, so you are promptly notified when certain metrics exceed or fall below predefined values.
Furthermore, Grafana supports collaboration, allowing users to share their dashboards and collaborate with team members. It also provides role-based access control, enabling administrators to manage user permissions and restrict access to sensitive data.
Overall, Grafana is widely used in various domains, including software development, IT operations, DevOps, and the Internet of Things (IoT), to visualize and analyze data from different sources, making it easier to understand complex systems and make data-driven decisions.

Installation of Grafana.

Grafana can be installed on various operating systems, we are installing it on Aws ec2 t2.micro ubuntu.

To install Grfana, you can follow these general steps:

Download the Grafana GPG key, This GPG key is used to verify the authenticity of Grafana packages during installation.

 sudo apt-get install -y apt-transport-https #This command installs the apt-transport-https package, which allows the system to fetch packages securely over HTTPS.
 sudo apt-get install -y software-properties-common wget #This command installs the software-properties-common package, which provides an easy way to manage software repositories,
 sudo wget -q -O /usr/share/keyrings/grafana.key # to download the Grafana GPG key  
 https://apt.grafana.com/gpg.key #saves key in this directory

Download the Grafana Stable release or Beta release based on the requirements.

 #Stable release
 echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

 #Beta release
 echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com beta main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

Update the list of available packages.
```
 sudo apt-get update
```
Install Grafana.
```
 sudo apt-get install grafana
```

Start Grafana.

 sudo systemctl start  grafana-server # to start grafana service

Enable Grafana.

 sudo systemctl enable  grafana-server # It is Important because whenever our server will closed or restart it will automatically start grafana application

Grafana runs on 3000 ports in order to access Grafana, we have to enable a 3000 port on the instance security group.
Access it by opening a web browser and entering the appropriate URL, such as http://13.232.93.163:3000/. Also for login use the default username password "admin". After that set a new password according to your choice.
Grafana Dashboard.

What is Loki?

Grafana Loki is an open-source log aggregation system that seamlessly integrates with Grafana, the popular visualization and analytics platform. Loki focuses on efficient log storage and retrieval, making it suitable for handling large volumes of log data.
When Loki is integrated with Grafana, it offers the following benefits:
1. Log Visualization: Grafana provides a user-friendly interface for creating dashboards and visualizing log data from Loki. You can leverage Grafana's extensive set of visualization options, including log panels, to explore and analyze log entries effectively.
2. Log Querying: Grafana allows you to query log data stored in Loki using a powerful query language called LogQL. LogQL provides rich filtering and aggregation capabilities, allowing you to search for specific log entries based on labels, time ranges, and other criteria. You can use LogQL queries within Grafana to retrieve the desired log data and display it in the form of tables or log panels.
3. Alerting and Notifications: Grafana supports alerting based on log data in Loki. You can define alert rules that trigger notifications when specific log patterns or conditions are met. For example, you can set up alerts to detect error messages or anomalies in log entries. When an alert is triggered, Grafana can send notifications via various channels such as email, Slack, or PagerDuty.
4. Log Labels and Metadata: Loki allows you to attach labels and metadata to log entries. These labels provide additional context and can be used for filtering and organizing logs. When integrated with Grafana, you can leverage these labels to create dynamic dashboards that adapt to changing log data and filter logs based on specific labels.
5. Log Aggregation and Scalability: Loki is designed to handle high volumes of log data and provides horizontal scalability. It efficiently stores log entries in chunks, making it suitable for distributed systems. When combined with Grafana, you can scale Loki to meet your log storage needs and visualize logs across multiple instances or clusters.

Install Loki using Docker.

Make the grafana_configs directory and Download the Loki config file inside the grafana_configs directory.

Install Docker

 sudo apt-get install docker.io #install Docker
 sudo usermod -aG docker $USER #add user as root user
 sudo restart #restart system

Download Loki Config.

 wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml

Run Loki Docker container.

 docker run -d --name loki -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:2.8.0 --config.file=/mnt/config/loki-config.yaml

Loki runs on 3100 ports in order to access Loki, we have to enable a 3100 port on the instance security group.
Access it by opening a web browser and entering the URL http://13.232.93.163:3100/ready
We can see in the below image Loki is able to fetch the system log. we can check the log using the URL http://13.232.93.163:3100/metrices
Loki serves as the central log aggregation and storage system, while Promtail is responsible for collecting and shipping logs to Loki. For shipping the above log we have to install Promtail

What is Promtail?

Promtail is a powerful open-source log shipper and collector developed by Grafana Labs. It is part of the Loki ecosystem, which is a horizontally scalable, highly available, and multi-tenant log aggregation system. Promtail is specifically designed to work with Prometheus, a popular monitoring and alerting toolkit.
Here are some key details about Promtail:
1. Log Shipping: Promtail is responsible for shipping log data from various sources to a centralized logging system like Loki. It collects logs from different applications, services, and systems and forwards them to Loki for storage and analysis.
2. Log Discovery: Promtail has built-in log discovery capabilities, which means it can automatically identify log files or log streams on a host or container. It supports log discovery for common log formats, including JSON, line-based logs, and structured logs.
3. Labels and Annotations: Promtail allows you to add labels and annotations to log entries before sending them to Loki. These labels and annotations provide additional contextual information and help with filtering and querying logs efficiently.
4. Target Configuration: Promtail supports flexible target configuration. You can define multiple scraping jobs, each with its own set of targets and labels. This allows you to scrape logs from different sources simultaneously and organize them based on their origin or purpose.
5. Relabeling: Promtail provides powerful relabeling capabilities. You can modify log labels and annotations dynamically using relabeling rules. This feature is particularly useful for standardizing log formats, enriching log data, or applying filtering based on specific criteria.
6. Support for Various Inputs: Promtail supports multiple input methods for log collection. It can tail log files, scrape log streams over HTTP, follow log streams from containers, and even receive logs from systemd's journal. This versatility enables Promtail to work effectively with different logging setups.
7. Integration with Prometheus: Promtail integrates seamlessly with Prometheus, a leading monitoring and alerting system. It leverages Prometheus's service discovery and relabeling mechanisms, allowing you to correlate log data with metric data for more comprehensive monitoring and troubleshooting.
8. Scalability and High Availability: Promtail is designed to be highly scalable and fault-tolerant. You can deploy multiple instances of Promtail to handle large log volumes or achieve high availability. Additionally, Promtail supports horizontal scaling by distributing log scraping jobs across multiple instances.
Promtail is a popular choice for log shipping and collection, especially in combination with Loki and Prometheus. It offers a flexible and efficient way to centralize and analyze logs, making it easier to monitor and troubleshoot distributed systems effectively.

Install Promtail using Docker.

Download Promtail Config

 wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/clients/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml

Run Promtail Docker container

 docker run -d --name promtail -v $(pwd):/mnt/config -v /var/log:/var/log --link loki grafana/promtail:2.8.0 --config.file=/mnt/config/promtail-config.yaml

Add a Loki data source in Grafana and Display logs

Log in to your Grafana instance and navigate to the Grafana home page.
Click on "Data Sources"
On the Data Sources page, click on the "Add data source" button.
On the "Add data source" page, you will see a list of available data source types. Locate and click on "Loki" in the list.
On the "Loki" configuration page, you need to provide the following details:
- Name: Give a name for your Loki data source.
- URL: Enter the URL of your Loki instance. For example, http://localhost:3100.
- Access: Choose the access mode for this data source. It determines who can view and interact with the data source.
- HTTP Method: Select the HTTP method to be used for data requests. Usually, the default option "GET" works fine.
- HTTP Header: If you need to provide custom HTTP headers, you can add them here. This is optional.
- HTTP Method and HTTP Header Options: These options control how Grafana handles queries for log labels and values. You can leave them at their default values.

After providing the necessary information, click on the "Save & Test" button to test the connection to your Loki instance.
If the connection is successful, you will see a green "Data source is working" message. Otherwise, you will see an error message indicating the issue.
Finally, the data source is connected it is time to explore.
In Explore UI we can see plugins & injectors added to the data source.

Example job & varlog as seen in the below image. It scrabs data from var/log directory and shows it on the Grafana dashboard

Same we can Find it in the Promtail config file. so If we need other directory log data we have to edit this file.
To see varlog click on Run Query Button.
Finally, We can see logs in Grafana Dashboard