We were using wso2 das as our analytics tool and faced a lot of performance issues which made us to find an alternative and we stopped at elastic stack which was free to use.
We were using wso2 api manager and wso2 esb to handle traffic between two external parties.
Our initial das implementation had only response capturing and our customers were looking for requess as well. we had a hell of lot of work to overcome that problem if we were to stay with wso2 das.
Elastic stack mainly consists
Then we found another tool in the elk family called packetbeat which can sniff network traffic and extract requests and responses. This seems to be a very easy solution for the moment. the only drawback we found was it doesn't support https. as our few customers work with http traffic internally we desided to go with it. This also had another import feature whre it can correlate requests and reponses automatically.
We picked the latest elk products
Setting-Up Elk
To be honest this was the easiest software installation I have done apart from something like microsoft word installation. If you are setting up in a single machine just just extract the downloaded files, all set to go.
Put in packetbeat-* as index pattern and click next. select @timestamp and click create index. now go back to discovery tab and select creted index pattern. you should be see a lot of network data captured by packetbeat.
Wow now you have a working elastic setup with almost zero changes/configurations done.
Going Production
As the setup was very simple and everything was working as smoothly we desided to go to production without any hesitation. Now I will cover how did I managed to fix issues found on that environment.
High disk usage
The first problem we faced was very high disk usage by elastic search server. Initially we had to deside all the index data daily as it started creating like 12GB per hour. our setup is like
A sends a request to B. Then B sends a requests to C. C sends reponse to B and B sends a response to A. So basically two transactions were there between A-B and B-C. Packebeat can correalte these 4 records and convert in to two events. So it will show two transactions for the 4 calls. Directin for A to B as in and for B to C as out is displayed in kibana. This helped us gathering another informaiton to track network latency berween servers using response time. Response time AB can consider as latency of server B that is our ESB server while response time in BC can categorise as latency of C which is in our is an operator.
Packetbeat Duplicates
When we analysing data for looking reasons for the space issue we saw data duplicaiton. Reason was we had a packetbeat on both apim and esb. Where when apim emits a event as out, the same event arrived to esb as in. Which made the event duplication. So eimply we desided to go with a one packetbeat running on esb only. This helpled redusing the size in to half.
Reducing fields
Changing index patterns to make hourly indices.
Remove unwanted events
Kibana Timeout
Download Requests & Responses
<Rest after having a coffee ...>
We were using wso2 api manager and wso2 esb to handle traffic between two external parties.
Our initial das implementation had only response capturing and our customers were looking for requess as well. we had a hell of lot of work to overcome that problem if we were to stay with wso2 das.
Elastic stack mainly consists
- Filebeat for reading data from files/logs
- Logstash for processing logs/data
- Elasticsearch for storing data
- Kibana for presenting data
Then we found another tool in the elk family called packetbeat which can sniff network traffic and extract requests and responses. This seems to be a very easy solution for the moment. the only drawback we found was it doesn't support https. as our few customers work with http traffic internally we desided to go with it. This also had another import feature whre it can correlate requests and reponses automatically.
We picked the latest elk products
- Packetbeat 6.2.4
- Elasticsearch 6.2.4
- Kibana 6.2.4
Setting-Up Elk
To be honest this was the easiest software installation I have done apart from something like microsoft word installation. If you are setting up in a single machine just just extract the downloaded files, all set to go.
- Start elasticsearch first
- Then packetbeat
- And then kibana
Put in packetbeat-* as index pattern and click next. select @timestamp and click create index. now go back to discovery tab and select creted index pattern. you should be see a lot of network data captured by packetbeat.
Wow now you have a working elastic setup with almost zero changes/configurations done.
Going Production
As the setup was very simple and everything was working as smoothly we desided to go to production without any hesitation. Now I will cover how did I managed to fix issues found on that environment.
High disk usage
The first problem we faced was very high disk usage by elastic search server. Initially we had to deside all the index data daily as it started creating like 12GB per hour. our setup is like
A -> B -> CCapturing network latency
A sends a request to B. Then B sends a requests to C. C sends reponse to B and B sends a response to A. So basically two transactions were there between A-B and B-C. Packebeat can correalte these 4 records and convert in to two events. So it will show two transactions for the 4 calls. Directin for A to B as in and for B to C as out is displayed in kibana. This helped us gathering another informaiton to track network latency berween servers using response time. Response time AB can consider as latency of server B that is our ESB server while response time in BC can categorise as latency of C which is in our is an operator.
Packetbeat Duplicates
When we analysing data for looking reasons for the space issue we saw data duplicaiton. Reason was we had a packetbeat on both apim and esb. Where when apim emits a event as out, the same event arrived to esb as in. Which made the event duplication. So eimply we desided to go with a one packetbeat running on esb only. This helpled redusing the size in to half.
Reducing fields
Changing index patterns to make hourly indices.
Remove unwanted events
Kibana Timeout
Download Requests & Responses
<Rest after having a coffee ...>
Comments
Post a Comment