Hi @hamedbrd

#4334 (comment)

What do you think?

This is really needed.

@ilyam8 could explain it a bit more
do you mean instead of

regex.search(line)

It's better to use

regex.match(line)

Am i right?

It depends on the method/value.

Example

foo:
  bar1: 'string=GET'
  bar2: 'string=^GOT'
  bar3: 'regex=G[QWERTY]T'

You should parse value and use appropriate matcher for that value.

We need a matcher. It has only one method match(self, row) which returns True/False.

We can use any logic for matchig - value in row, row.startswith(value), value.match(row) - all of them return bool.

Matchers:

class BaseStringMatcher:
    def __init__(self, value):
        self.value = value

    def match(self, row):
        raise NotImplementedError


class BaseRegexMatcher:
    def __init__(self, value):
        self.value = re.compile(value)

    def match(self, row):
        raise NotImplementedError


class StringMatcher(BaseStringMatcher):

    def match(self, row):
        return self.value in row


class StringPrefixMatcher(BaseStringMatcher):

    def match(self, row):
        return row.startswith(self.value)


class StringSuffixMatcher(BaseStringMatcher):

    def match(self, row):
        return row.endswith(self.value)


class RegexMatchMatcher(BaseRegexMatcher):

    def match(self, row):
        return self.value.match(row)


class RegexSearchMatcher(BaseRegexMatcher):

    def match(self, row):
        return self.value.search(row)

METHOD_REGEX = "regex"
METHOD_STRING = "string"


def regex_matcher_factory(value):
    if value.startswith('^'):
        return RegexMatchMatcher(value)
    return RegexSearchMatcher(value)


def string_matcher_factory(value):
    if value.startswith("^"):
        return StringPrefixMatcher(value)
    elif value.endswith('$'):
        return StringSuffixMatcher(value)
    return StringMatcher(value)


def matcher_factory(raw_value):
    method, value = raw_value.split("=")

    if method == METHOD_REGEX:
        return regex_matcher_factory(value)

    if method == METHOD_STRING:
        return string_matcher_factory(value)

    raise ValueError('unknown search method')

@hamedbrd

if you need any help with the module feel free to ask

@hamedbrd if you don't have time to finish it i can take over, np

@ilyam8 as you guessed ,unfortunately i do not have time for these changes and also i have some other changes in my mind for this plugin to being able to do something more but no free time.
I will update it in next couple of weeks

So i would appreciate if you help on this

@hamedbrd so what help do you need? do you want me to add this directly to your PR?

@ilyam8 could you please take look at new changes and if you think there is not problem then i will update the readme?

This pull request introduces 1 alert when merging 13f5c48 into f1bb78a - view on LGTM.com

new alerts:

• 1 for Unused import

Comment posted by LGTM.com

@hamedbrd logic should be

1. create [(dim_name, matcher), ...] list in check (use namedtuple, so it will be list of namedtuples), populate self.data with dimension names zero values, create charts, etc.

2. iterate over dim_name, mather list in get_data

for row in raw:
    for m in self.matchers:
        if m.match(row):
            self.data[m.name] += 1
            break

return self.data

example^^

@ilyam8
I can not get your last feedback.
What do you mean?Could you please tell me more?

@hamedbrd have you tested lastest version? please confirm that it works.

Yes, I've test it many times because it's used on different services

i mean the latest, becase we made few changes in the last hour.

@cakrit please have a look, i see you had some questions

code looks ok to me

i mean the latest, becase we made few changes in the last hour.
Yes, each time i ran the latest changes to assure it worked well

@hamedbrd

One more - please add default info to the readme

after that - lgtm

Ok, so what I still don't see here is a time element. What I got from the code is that we just have an incremental counter for each pattern that starts at zero every time netdata is restarted and just keeps increasing with every matching line. However, the metric unit states jobs/s, so perhaps I missed something here?

jobs/s

imo should be matches/s or something

What I got from the code is that we just have an incremental counter for each pattern that starts at zero every time netdata is restarted and just keeps increasing with every matching line.

Yes
#3729 (comment)

So what I'm saying is that we don't actually have a way here to show e.g. the occurences in a day, as the issue and the README say we want to.
I guess it's the best we can do though.

The issue has or

and then create a chart based on occurences of a log line (like amount of occurences in a day, or a line graph of when the occurences happen).

I think the PR implements the part after or 😄 So the issue isfixed after this PR will be merged.

@ilyam8 @cakrit
What more does it need to be merged?

@hamedbrd please remove the [wip] from the title, to signify it's ready. @ilyam8 please provide the final review after the automated checks are completed again

@ilyam8 any update?

@hamedbrd sorry for delay. The module is finished, good job 👍

But we will rewrite it in go (netdata has go.d.plugin) (you can do it yourself btw if you willing)

We need to wait untis this PR is merged.

And we use new syntax for all that string/regex stuff:
https://github.com/netdata/go.d.plugin/tree/master/pkg/matcher#supported-format

@ilyam8 For rewriting to go, I will take over this part too but you won't merge this MR, will you?
At the moment I don't have time for this rewrite but definitely I will do it asap.

@hamedbrd

you won't merge this MR, will you?

No.

At the moment I don't have time for this rewrite but definitely I will do it asap.

Nice! As i said lets wait for netdata/go.d.plugin#141.

I will ping you when it will be merged.

@hamedbrd you can follow netdata/go.d.plugin#141. I'm closing this one.

@ilyam8 I think the python code for logparser never got merged because rewrite was desired in Go and #7420 is still open. Do we have a reliable way to do basic application log monitoring using regular expressions in Netdata?