Analyzing access logs of WSO2 Identity Server with Elastic Stack

In my previous post here we saw how to analyze the access logs of WSO2 Identity Server with GoAccess. Today lets see how to configure Elastic Stack to analyze access logs of WSO2 Identity Server.

First of all, you need to have Elasticsearch and Kibana up and running. You can follow the documentation of Elastic Stack to do so. Next, you can download Logstash and configure it to parse the access logs in the way we want. Below is the configuration file I have used.

Below are the important parts of the above config (replace the IP address of the Elasticsearch in the above config properly).

• First, we are parsing the log lines with grok. Values will be added to the semantics defined in the pattern (clientip, timestamp, etc)
• Note that the messagesize field has handled in a different way since it can have a number as well as “-” when there is a redirect response. Next, we will replace “-” with “0” and then convert it to an integer.
• We are using the timestamp in the logline as the timestamp of the event published to Elasticsearch.
• We are parsing the “useragent” in the logline. We are creating new fields named “browser” and “operating_system” from the parsed parts of the user agent and remove some unnecessary fields.
• Next, we are publishing the formatted logline to Elasticsearch.

Now start Logstash pointing to the config file we created.

./logstash -f logstash.conf

The next thing we need to do is configure and start Filebeat. Use the below config file.

The above configs are self-explanatory. Change the path to the Identity Server distribution, hostnames of Logstash, and Kibana. Place the above config file inside the Filebeat directory and start the filebeat.

If you are using Identity Server 5.11.0 you do not need to configure anything on the Identity Server side. If you are using 5.10.0 or above, you have to configure the access log pattern like below in the deployment.toml file.

Now everything is set. Go to the Kibana dashboard and navigate to Management -> Kibana -> Index Patterns and create a new index pattern with “access-log” which we have configured previously in Logstash. Now navigate to the Discover section and you will be able to see the access logs published from Logstash.

You can expand log entries and observe the fields we have published from Logstash.

You can try different filters and analyze the access patterns to the Identity Server and enjoy it!