You may have seen some of my previous blog posts regarding monitoring Azure VM’s using Azure Log Analytics. If not you can see them at https://pixelrobots.co.uk/tag/azure-log-analytics/ One question that keeps coming up is “How long does it take for the information to be visible in Log Analytics” and its a good question. So I thought, let’s write a blog post to answer the question.
According to a Microsoft docs article typical latency to ingest log data is between 2 and 5 minutes, but this can vary based on a few factors.
So what are the factors
There are 3 high-level factors.
Agent Time:- This is the time it takes the agent to discover an event, collect it, and then send it up to Azure for ingestion.
Pipeline Time:- This is the time it takes to process the log record. The pipeline is where parsing the properties of the event happens.
Indexing Time:- This is the time it takes to ingest the log record into the big data store within Azure Monitor.
You can read more about each factor at https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-ingestion-time
Prove it!
So, its all well and good reading about something, but nothing beats seeing it. Luckily there is a hidden DateTime Column available in each Log Analytics table. This column is called IngestionTime. Nice name hey?
In this column, you will find the time of ingestion for each record. You will also notice that every table also has a TimeGenerated column. This column is the timestamp from the source of the log entry. Now we have the ingested time and the time the event actually happened on the source we can use a Kusto query (KQL) to calculate the estimated latency. I say estimated as time slips happen on servers etc.
The below query will look at the Perf table and find the latency in Minutes.
| Perf | |
| | extend LatencyInMinutes = datetime_diff('minute', ingestion_time(), TimeGenerated) | |
| | project TimeGenerated, ingestion_time(), LatencyInMinutes | |
| | order by LatencyInMinutes |
You could even go one better and get the average latency over a period of say 5 hours, set it as a chart and then pin it to a dashboard. The following query will create just that. You will have to click the Pin icon to actually pin it to a dashboard, but the rest is done.
| let startDate = ago(5h); | |
| Perf | |
| | where TimeGenerated > startDate | |
| | extend LatencyInMinutes = datetime_diff('minute', ingestion_time(), TimeGenerated) | |
| | project TimeGenerated, ingestion_time(), LatencyInMinutes | |
| | summarize avg(LatencyInMinutes) by bin(TimeGenerated, 1h) | |
| | render timechart |
If you decided to pin it to a dashboard, you will then have something like this:
I hope this answers the question “How long does it take for the information to be visible in Log Analytics” gives you a little bit more. If you do have any further questions or comments please reach out in the comments below or via social media.
Thanks for reading.
0 Comments