Child pages
  • [system log] System Log Aggregator

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

We've added System Log in 

JIRA
serverIn-Portal Issue Tracker
keyINP-811
 and it's been a huge success so far, since it gave a developer invaluable information from production websites, that we weren't able to gather otherwise before it. However soon enough we've faced a following problem, that lead to exponential system log record count increase making it completely unmanageable.

Consider following scenario:

  1. developer missed 1 notice message on a page, while he was developing it (or maybe that 1 notice only happens on production server)
  2. page has 1000 visits per day

Since each notice is logged and 1000 new records are added to system log every day. System log records about errors, warnings and notices have backtrace information included to ease up debugging and it adds extra to overall database table size.

On one of a projects we've faced system log table grown to 370MB and 12490 records in it. After simplified duplicate removal (matching by "error message", "page url" and "error backtrace") this table had only 2480 records left. This can be even smaller number if we'll be processing url more intellectually (e.g. first remove irrelevant parameters from url, like "page number") and compare resulting urls.

I'm proposing to:

  1. add "LogLastRecurrenceOn" (date/time) and "LogRecurrenceCount" (int) columns
  2. from time to time (e.g. once a day) do system log record grouping (aggregation)

This way we still have all the info that was in each error, but also save space (on disk) and searching time needed to find right error along many similar ones.