On Thu, 2016-06-30 at 20:01 -0600, Rich Megginson wrote:

On 06/30/2016 07:52 PM, William Brown wrote:

...

Hi,

I've been thinking about this for a while, so I decided to dump my thoughts to a document. I think I won't get to implementing this for a while, but it would really help our server performance.

http://www.port389.org/docs/389ds/design/logging-performance-improvement.htm...

Looks good. Can we quantify the current log overhead?

Sure, I could probably sit down and work out a way to bench mark this .....

But without the alternative being written, hard to say. I could always patch out logging and drop the lock in a hacked build so we can show what "without logging contention" looks like?

...

-- 389-devel mailing list 389-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.or...

-- 389-devel mailing list 389-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.or...

On Thu, 2016-06-30 at 20:53 -0600, Rich Megginson wrote:

On 06/30/2016 08:14 PM, William Brown wrote:

...

On Thu, 2016-06-30 at 20:01 -0600, Rich Megginson wrote:

...

On 06/30/2016 07:52 PM, William Brown wrote:

...

Hi,

I've been thinking about this for a while, so I decided to dump my thoughts to a document. I think I won't get to implementing this for a while, but it would really help our server performance.

http://www.port389.org/docs/389ds/design/logging-performance-improvement.htm...

Looks good. Can we quantify the current log overhead?

Sure, I could probably sit down and work out a way to bench mark this .....

But without the alternative being written, hard to say. I could always patch out logging and drop the lock in a hacked build so we can show what "without logging contention" looks like?

That's only one part of it - you'd have to figure out some way to get rid of the overhead of the formatting and flushing in the operation threads too.

This change would move the formatting into the thread that does the write + flush.

Currently, we format the message in the operation thread, and send it to the log buffer, and every now and again, an "unlucky" operation thread has to stall to write the buffer ....

However, I'm proposing we send unformatted time struct + unformatted message to the queue, then the actual thread which is separate does the dequeue and format.

I suppose you could just write it and see what happens.

Well, a hacked up test that has slapi_log_access return 0 immediately (IE does not take the lock) showed a 6% performance improvement on my laptop. On a higher cpu count system, this would exceed that number easily, as more cpus == more cost to acquire the lock.

I've put the numbers on the design doc page, but I think with this, the proof would be in the implementation. We also likely wouldn't see the benefit of this as much until we remove other serialisation points in the code IE ldbm vs lmdb, plugins that take locks, etc. So it's only a 6% possible gain now, but it may be 10% or more in the future in combination with other efforts such as nunc-stans.

...

...

...

-- 389-devel mailing list 389-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.or...

-- 389-devel mailing list 389-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.or...

-- 389-devel mailing list 389-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.or...

-- 389-devel mailing list 389-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.or...

William,

Yes this looks great, but I think we should apply this to the audit and errors log as well. We always have the problem of enabling verbose error logging because of the severe performance impact it has (especially since the errors log is not buffered). Having all the logs using this model would be very useful for DS users and the support team.

Thanks,

Mark

On 07/07/2016 06:03 AM, thierry bordaz wrote:

Hi William,

This is looking a great idea. You are proposing to use liblfds to communicate with the log write. Do you think dbus would be an other option ? Would it help external mechanism to collect the DS logs ?

thanks thierry

On 07/01/2016 03:52 AM, William Brown wrote:

...

Hi,

I've been thinking about this for a while, so I decided to dump my thoughts to a document. I think I won't get to implementing this for a while, but it would really help our server performance.

http://www.port389.org/docs/389ds/design/logging-performance-improvement.htm...

-- 389-devel mailing list 389-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.or...

-- 389-devel mailing list 389-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.or...