LOG ROTATION USING LINUX COMMAND-LINE UTILITY LOGROTATE

Let us consider a scenario when all the service and custom logs of your ubuntu server gets backed up on S3 on a daily basis after compressing them and the allocated space is freed without using any third party software.

We can achieve this task by the use of logrotate utility provided by the ubuntu server.

Log rotation means the task of archiving any application’s or system’s current log, starting a fresh log, and deleting older logs.Log files left unrotated can lead to the filling of disk space which can be an alarming situation for the server.You can backup the archived logs on to your S3 bucket. Log rotation helps the system to ensure that the useful space is getting utilized in an appropriate manner.

Here is a demo which will help your log files to get rotated in an effective manner and also ensure safe backup of these archived logs on to your S3 bucket.

Step 1 :- Installing logrotate utility

Log rotation is a utility of ubuntu and might be preinstalled in your ubuntu server (/etc/logrotate.d).If it is not installed then you can install it manually by the following commands.

sudo apt-get update

sudo apt-get install logrotate

Step 2 :- Configuring Logrotate

There are two ways to configure logrotate.

1.  By making changes in default global configuration file /etc/logrotate.conf for all services :- The  main configuration file is nicely commented and you can trim this file to your needs.You can also make specified blocks for your application in the file.The attributes of logrotate are described below.By default,, logrotate reads from this configuration file if you haven’t made any specific configuration in the directory /etc/logrotate.d .The configuration in logrotate.conf applies to all the service logs.If you want to set different options on the basis of the applications, then go for option 2.

2.  By making individual configuration file in directory /etc/logrotate.d/ for each service  and application .

This blog will illustrate how you can configure the logrotate for efficient use by going through option 2.This is the optimized way to manage your logs depending upon the type of application and its use.

In the directoy /etc/logrotate.d/, you may find many configuration files already present there.Generally you will find configuration files of those applications in this directory which are installed using package manager.

Lets take an example of a configuration file in /etc/logrotate.d/apache2 . If there is no file in this directory, you can create one.But if you have apache2 installed in your system using package manager (apt-get), you must have this file.Below is my apache configuration file for logrotation :-

[js]
/var/log/apache2/*.log {
weekly
missingok
rotate 4
compress
delaycompress
notifempty
create 640 root adm
sharedscripts
postrotate
aws s3 sync /var/log/apache2/ s3://<Your-bucket-Name>/<Server-Name>/<Apache2-logs>/ –region <your-bucket-region>
endscript
}
[/js]

 Note :- In the above file you can also create specify more than one log directory.For example, /var/log/apache2/*.log  /var/<app-directory>/*.log and then the block can be started for the configuration file.The configuration will be applicable for both the log directories.

 The attributes are described below :-

• 1. Rotate Interval :- This command tells logrotate that how often the logs will be rotated.It basically specifies the time interval after which the logrotation is done.

The options can be :-

1. daily
2. weekly
3. monthly
4. yearly

 If you want to set some other option, for example say hourly then you have to make some adjustments.If you want to rotate logs on per hour basis, then you need to create a directory /etc/cron.hourly and make a new file in it which will contain the following line :-

 /usr/sbin/logrotate /etc/logrotate.d/<name of configuration file for which log rotation is done hourly>

 If the rotation interval is not specified then the logs are rotated meeting to some other condition like size exceeding 100k, 100M, 100G.

•  2. Size :- The size attribute tells logrotate to perform the operation when the size of the file becomes greater than the specified size.
  1. size 100k :- The logs are rotated when the size becomes 100 kilobytes
  2. size 100M :- The logs are rotated when the size becomes 100 Megabytes
  3. size 100G :- The logs are rotated when the size becomes 100 Gigabytes
• 3. Rotate Count :- The rotate count specifies how many archived log files will be kept around before the logrotate starts deleting the older files.
  • Rotate 4

This will specify that, at a time  4 archived files will be kept by logrotate and when there are four archived files already present then the oldest one named “<log-file-name>.4 “ will be deleted to make space for the newly created archive.

• 4. Missingok :- This attribute tells that if the log file is missing then logrotate will not throw any error and will jump to next one.

• 5. Compress :-  This command will archive log files in compressed ( gzip format).It is generally a good idea to compress the files to save the useful space since log files are only text files so a level of compression can be really helpful in  proper utilization of the storage.If you don’t want to compress the log files, you can simply write nocompress instead of compress  in the configuration file.

• 6. Delaycompress :- With delaycompress option active, the compression of the previous log files is postponed for the next rotation cycle.This option will work in combination with compress option and is  really helpful when the application keeps writing logs to the previous log files.

• 7. Notifempty :-  This option tells logrotate that do not rotate the logs if empty.

• 8. create 640 root adm :- This option specifies the permission of newly created log files.It specifies that the newly created log files will be having permission of read/write for owner root and only read for group adm.

• 9. postrotate :- The postrotate script is run by the logrotate every time it rotates log in specified configuration block.The postrotate command specifies that the next line will be the starting line of script and the script will be ended when the endscript command is encountered.Here you can specifiy something like restarting apache server to switch to new log file.I am writing a command to sync the directory of my logs to have a backup on S3 bucket.You can also include your custom logic like making a date folder on S3 bucket to create dated backups of the logs.

• 10. sharedscripts :-  This option tells that the postrotate script will be run only once for multiple configuration files having same log directory.For example, apache2 has both error logs as well as access logs but the directory for both is same (/var/log/apache2/*.log) . In this case, if the sharedscripts tag is on then it will run postrotate script only once otherwise postrotate script will be run everytime the log rotation is done.

 Similarly, configuration files for other services and applications can be made to manage their logs.In each file you can specify a new folder of your S3 bucket to manage logs according to the application name.

To start the logrotate service manually or to check the configuration for proper functioning, you can use the command :-

logrotate -v /etc/logrotate.d/apache2  

where option -v means verbose so that you can view all the progress done by logrotate utility.

Logrotate utility provides an easy and efficient way to manage service logs as well as application logs.