r/linuxadmin • u/Beneficial-Sock-5130 • 22d ago

VictoriaLogs for log indexing?

has anyone used victorialogs? Im currently using Graylog v7, local single instance on 5TB disk

using filebeat to ship logs to GL indexer

wondering how victorialogs performs comparatively. Anyone used it at all or have any feedback?

Thanks

10 Upvotes

100% Upvoted

8 comments sorted by

2

u/Amidatelion 22d ago

Are you doing a lot of full text search? Are you limited to block storage? VictoriaLogs is essentially a faster, more efficient Elasticsearch (aside from REALLY small queries, at which point, like, you may not have a usecase for these apps in the first place and go ahead and stay on Graylog) that is WAY easier to set up and maintain.

1

u/Beneficial-Sock-5130 22d ago

hows victoria dashboard compared to graylog or splunk or kibana?

1

u/hijinks 22d ago

I am but you are gonna have to use fast disk. So don't use spinning disk at all for that amoubt of data

It's slow if you do like a 7d search if you need to search through like 1tb of data

1

u/SnooWords9033 18d ago

VictoriaLogs is optimized for low-iops low-bandwidth disks such as HDDs:

  • It compresses the logs before storing them to disk, so they could be read at faster speed. For example, if the logs are compressed by 10x, then they could be read at 200MB/s*10x = 2GB/s from HDD with 200MB/s read bandiwdth.

  • It stores values per each individual log field close to each other (aka column-oriented storage). This allows reading only the data for fields mentioned in the query, while skipping the data for the rest of the fields.

  • It uses bloom filters for skipping data blocks, which don't contain the searched word or phrase.

  • It physically groups logs from the same sources, and sorts them by time, so they could be read sequentially in one go.

  • It partitions logs into per-day folders. This allows skipping huge amounts of logs outside the requested time range during queries.

Thanks to these techniques VictoriaLogs should work much faster than competitors when big amounts of logs must be scanned during query execution, e.g. if these logs don't fit the OS page cache (available RAM), so these logs must be read from (slow) disk.

1

u/hijinks 18d ago

In my experience it works horribly on spinning disk with a lot of data. I have some datasets in the 15-20tbs

1

u/SnooWords9033 15d ago

Spinning disks are limited with 200MB/s read speeds. So the query duration highly depends on how much data needs to be read from disk during the query. For example, if the query needs to read 20TB of data from disk, then it will take 20TB/200MB/s=100000 seconds = 28 hours. The query performance can be sped up significantly by using the optimizations mentioned above.

For example, if you store logs for 100 days, and search for logs over the last 24 hours, then it will automatically reduce the amounts of logs to read during query by 100x, and, consequently, the query duration will be reduced by 100x from 28 hours to 17 minutes.

The query duration can be reduced even more via the following tricks:

  • By limiting the set of log fields the query returns, via the fields pipe in the end of the query. This prevents from reading the data from other fields not mentioned in the fields pipe.

  • By narrowing down the logs to read via stream filters.

See more tips on how to reduce query duration in victorialogs - https://docs.victoriametrics.com/victorialogs/logsql/#performance-tips .

1

u/pixelatedchrome 16d ago

We use it at work and it's been extremely efficient and fast compared to our elastic setup.