Changes to 'refs/tags/389-ds-console-1.2.3'
by Richard Allen Megginson
Changes since the dawn of time:
Endi S. Dewata (3):
Bug 496863 - 'Construct' button creates incorrect referral urls
Resolves: bug 574098
Bug 563513 - New replication -> No changelong configured-msg
Nathan Kinder (19):
171941 - Adjusted the version number and branding. I had to modify the checkVersion method to deal with running in Console 1.0. I also cleaned up references to consolesdk in the Ant build files. They should be console instead since we dropped the consolesdk naming. The external and internal imports files were adjusted to pick up the new 1.0 Console component.
171941 - checkVersion was comparing versions with greater than when it should have been using greater than or equal to
177696 - Changed usage of enum keyword as a variable name for Java 1.5 compatibility
Bug(s) fixed: 181570
Resolves: 246513
Summary: Initial fedora-ds-console specfile.
Resolves: ?
Resolves: 250137
Resolves: 250145
Resolves: 204510
Resolves: 250636
Resolves: 252036
Add default ldapjdk path as well as a settable parameter
Fixed typo from last checkin
Use less restrictive version of Open Publication License for online help docs.
Resolves: 308221
Resolves: 333171
Resolves: 178247
Resolves: 379191
Noriko Hosoi (13):
Changed the ldapconsole package name to <brand>-ds-<version>.jar
Modified ant move syntax to support ant 1.6.2
[186105] Admin Server Makefile updates for Internal build
Resolves: #247215
Resolves: #248073
Resolves: #379191
Resolves: #386041
Resolves: #379191
Resolves: #379191
Resolves: #379191
Resolves: #379191
Resolves: #379191
Resolves: 178947
Rich Megginson (34):
Bug(s) fixed: 167761
allow the definition of the console location on the command line with different directory layouts than the default
Bug(s) fixed: 178478
The console now builds jar files in the format
bump version to 1.0.3
updated spec for Fedora DS 1.1 release
Resolves: bug 428357
Bump version to 1.1.2
this is for the 1.1.2 release
Resolves: bug 469261
Resolves: bug 469261
Resolves: bug 234948
Resolves: bug 452596
Resolves: bug 178947
Resolves: bug 177334
Resolves: bug 249120
Resolves: bug 238762
Resolves: bug 179193
Resolves: bug 179184
change version to 1.1.3
Resolves: bug 487831
Resolves: bug 481213
Resolves: bug 483660
Resolves: bug 483660
updated for 1.1.3
version must correspond to ds base version - 1.2.0
rename to 389
change mode of spec file to 644
added separate doc subpackage
update to version 1.2.1
Bug 586571 - DS Console shows escaped DNs
bump version to 1.2.2
Bug 586571 - DS Console shows escaped DNs
Bug 591989 - [console] mis-matched trademark and text
foxworth (2):
Import initial source drop of Fedora DirectoryConsole
Import initial source drop of Fedora DirectoryConsole
13 years, 10 months
build.properties src/com
by Richard Allen Megginson
build.properties | 2 +-
src/com/netscape/admin/dirserv/about.properties | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
New commits:
commit bf23aaef8b2b456468c437df295d0c01565c57f1
Author: Rich Megginson <rmeggins(a)redhat.com>
Date: Thu May 13 14:59:16 2010 -0600
Bug 591989 - [console] mis-matched trademark and text
https://bugzilla.redhat.com/show_bug.cgi?id=591989
Resolves: bug 591989
Bug Description: [console] mis-matched trademark and text
Reviewed by: ???
Branch: HEAD
Fix Description: Branded graphics have been moved into the theme sub-package
Platforms tested: RHEL5 x86_64
Flag Day: no
Doc impact: no
diff --git a/build.properties b/build.properties
index 5961274..4640f55 100755
--- a/build.properties
+++ b/build.properties
@@ -21,7 +21,7 @@
lang=en
ldapconsole.root=..
-ldapconsole.version=1.2.2
+ldapconsole.version=1.2.3
ldapconsole.gen.version=1.2
brand=389
ldapconsole.name=${brand}-ds-${ldapconsole.version}
diff --git a/src/com/netscape/admin/dirserv/about.properties b/src/com/netscape/admin/dirserv/about.properties
index 4d1d26a..a8505a7 100644
--- a/src/com/netscape/admin/dirserv/about.properties
+++ b/src/com/netscape/admin/dirserv/about.properties
@@ -20,7 +20,7 @@
# Strings and logos used by the DSAboutDialog
aboutDialog-dialogTitle=389 Directory Server 1.2.0
-aboutDialog-productLogo=com/netscape/management/client/images/logo32.gif
+aboutDialog-productLogo=com/netscape/management/client/theme/images/logo32.gif
aboutDialog-productCopyright=Copyright (C) 2001 Sun Microsystems, Inc. Used by permission.\nCopyright (C) 2005 Red Hat, Inc.\nAll rights reserved.
# not needed for 389
#aboutDialog-productLicense=Fedora is a trademark of Red Hat, Inc. in the United States and other countries and is used by permission.
13 years, 10 months
wrappers/initscript.in
by Richard Allen Megginson
wrappers/initscript.in | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
New commits:
commit 6d581d5b8199b0641e3305acb669498604af4cba
Author: Rich Megginson <rmeggins(a)redhat.com>
Date: Thu May 13 13:42:27 2010 -0600
Bug 591685 - Server instances Fail to Start on Solaris due to Library Path and pcre
https://bugzilla.redhat.com/show_bug.cgi?id=591685
Resolves: bug 591685
Bug Description: Server instances Fail to Start on Solaris due to Library Path and pcre
Reviewed by: self
Branch: master
Fix Description: Added pcre_libdir to the initscript LD_LIBRARY_PATH
Platforms tested: Solaris 9
Flag Day: no
Doc impact: no
(cherry picked from commit 21629abd33d98a000ae89c73cadd6782b885f793)
diff --git a/wrappers/initscript.in b/wrappers/initscript.in
index f166371..e84ff9d 100644
--- a/wrappers/initscript.in
+++ b/wrappers/initscript.in
@@ -120,7 +120,7 @@ fi
start() {
if [ -n "$INSTANCES" ]; then
- LD_LIBRARY_PATH=@libdir@/@package_name@:@nss_libdir@
+ LD_LIBRARY_PATH=@libdir@/@package_name@:@nss_libdir@:@pcre_libdir@
export LD_LIBRARY_PATH
echo "Starting $prog: "
# Start every slapd instance that isn't already running
13 years, 10 months
Branch 'Directory_Server_8_2_Branch' - wrappers/initscript.in
by Richard Allen Megginson
wrappers/initscript.in | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
New commits:
commit 21629abd33d98a000ae89c73cadd6782b885f793
Author: Rich Megginson <rmeggins(a)redhat.com>
Date: Thu May 13 13:42:27 2010 -0600
Bug 591685 - Server instances Fail to Start on Solaris due to Library Path and pcre
https://bugzilla.redhat.com/show_bug.cgi?id=591685
Resolves: bug 591685
Bug Description: Server instances Fail to Start on Solaris due to Library Path and pcre
Reviewed by: self
Branch: Directory_Server_8_2_Branch
Fix Description: Added pcre_libdir to the initscript LD_LIBRARY_PATH
Platforms tested: Solaris 9
Flag Day: no
Doc impact: no
diff --git a/wrappers/initscript.in b/wrappers/initscript.in
index f166371..e84ff9d 100644
--- a/wrappers/initscript.in
+++ b/wrappers/initscript.in
@@ -120,7 +120,7 @@ fi
start() {
if [ -n "$INSTANCES" ]; then
- LD_LIBRARY_PATH=@libdir@/@package_name@:@nss_libdir@
+ LD_LIBRARY_PATH=@libdir@/@package_name@:@nss_libdir@:@pcre_libdir@
export LD_LIBRARY_PATH
echo "Starting $prog: "
# Start every slapd instance that isn't already running
13 years, 10 months
Branch 'Directory_Server_8_2_Branch' - ldap/servers
by Noriko Hosoi
ldap/servers/slapd/back-ldbm/import-threads.c | 18 +++++++++---------
1 file changed, 9 insertions(+), 9 deletions(-)
New commits:
commit c41920065398819fed0f1ec31dc89746fe7113f0
Author: Noriko Hosoi <nhosoi(a)redhat.com>
Date: Thu May 13 12:18:43 2010 -0700
Another bug fix made in commit 142d7ada626173de4937330be6776fabbebe9f60
591336 - Implementing upgrade DN format tool
Mistakenly removed a password encoding code.
Meant to remove it from index_producer, not from import_producer.
diff --git a/ldap/servers/slapd/back-ldbm/import-threads.c b/ldap/servers/slapd/back-ldbm/import-threads.c
index 336c3d6..20dce13 100644
--- a/ldap/servers/slapd/back-ldbm/import-threads.c
+++ b/ldap/servers/slapd/back-ldbm/import-threads.c
@@ -627,6 +627,15 @@ void import_producer(void *param)
continue;
}
+ /* not sure what this does, but it looked like it could be
+ * simplified. if it's broken, it's my fault. -robey
+ */
+ if (slapi_entry_attr_find(ep->ep_entry, "userpassword", &attr) == 0) {
+ Slapi_Value **va = attr_get_present_values(attr);
+
+ pw_encodevals( (Slapi_Value **)va ); /* jcm - cast away const */
+ }
+
if (job->flags & FLAG_ABORT) {
backentry_free(&ep);
goto error;
@@ -855,15 +864,6 @@ void index_producer(void *param)
if (!ep)
goto error;
- /* not sure what this does, but it looked like it could be
- * simplified. if it's broken, it's my fault. -robey
- */
- if (slapi_entry_attr_find(ep->ep_entry, "userpassword", &attr) == 0) {
- Slapi_Value **va = attr_get_present_values(attr);
-
- pw_encodevals( (Slapi_Value **)va ); /* jcm - cast away const */
- }
-
if (job->flags & FLAG_ABORT)
goto error;
13 years, 10 months
Branch 'Directory_Server_8_2_Branch' - ldap/servers
by Noriko Hosoi
ldap/servers/slapd/back-ldbm/import-threads.c | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
New commits:
commit 1d2ee6dc6296f9c2efeb468edf9d4fddd17b2138
Author: Noriko Hosoi <nhosoi(a)redhat.com>
Date: Thu May 13 12:11:20 2010 -0700
A bug fix made in commit 142d7ada626173de4937330be6776fabbebe9f60
591336 - Implementing upgrade DN format tool
Mistakenly removed a function call "import_generate_uniqueid(job, e);"
Meant to remove it from index_producer, not from import_producer.
diff --git a/ldap/servers/slapd/back-ldbm/import-threads.c b/ldap/servers/slapd/back-ldbm/import-threads.c
index 09a156f..336c3d6 100644
--- a/ldap/servers/slapd/back-ldbm/import-threads.c
+++ b/ldap/servers/slapd/back-ldbm/import-threads.c
@@ -609,6 +609,8 @@ void import_producer(void *param)
continue;
}
+ /* generate uniqueid if necessary */
+ import_generate_uniqueid(job, e);
if (g_get_global_lastmod()) {
import_add_created_attrs(e);
}
@@ -849,9 +851,6 @@ void index_producer(void *param)
}
slapi_ch_free(&(data.data));
- /* generate uniqueid if necessary */
- import_generate_uniqueid(job, e);
-
ep = import_make_backentry(e, temp_id);
if (!ep)
goto error;
13 years, 10 months
mod_nss TODO, 1.3, 1.4 mod_nss.c, 1.18, 1.19 mod_nss.h, 1.21, 1.22 nss_engine_config.c, 1.16, 1.17
by Rob Crittenden
Author: rcritten
Update of /cvs/dirsec/mod_nss
In directory cvs01.phx2.fedoraproject.org:/tmp/cvs-serv30758
Modified Files:
TODO mod_nss.c mod_nss.h nss_engine_config.c
Log Message:
Compare CN value of remote host with requested host in reverse proxy.
Add configuration option to disable this, defaulting to on.
591224
Index: TODO
===================================================================
RCS file: /cvs/dirsec/mod_nss/TODO,v
retrieving revision 1.3
retrieving revision 1.4
diff -u -r1.3 -r1.4
--- TODO 21 Jun 2006 14:42:06 -0000 1.3
+++ TODO 13 May 2010 15:21:24 -0000 1.4
@@ -1,5 +1,2 @@
-- Offer to automatically generate a self-signed cert using gencert during
- install?
-- Should gencert create a database with an empty password or continue
- to create a protected on?
- Once NSS fully supports the SNI TLS extension, add that.
+- Add support for OCSP stapling
Index: mod_nss.c
===================================================================
RCS file: /cvs/dirsec/mod_nss/mod_nss.c,v
retrieving revision 1.18
retrieving revision 1.19
diff -u -r1.18 -r1.19
--- mod_nss.c 2 Mar 2010 20:12:04 -0000 1.18
+++ mod_nss.c 13 May 2010 15:21:25 -0000 1.19
@@ -142,6 +142,8 @@
SSL_CMD_SRV(ProxyNickname, TAKE1,
"SSL Proxy: client certificate Nickname to be for proxy connections "
"(`nickname')")
+ SSL_CMD_SRV(ProxyCheckPeerCN, FLAG,
+ "SSL Proxy: check the peers certificate CN")
#ifdef IGNORE
/* Deprecated directives. */
@@ -238,23 +240,30 @@
SECStatus NSSBadCertHandler(void *arg, PRFileDesc * socket)
{
conn_rec *c = (conn_rec *)arg;
+ SSLSrvConfigRec *sc = mySrvConfig(c->base_server);
PRErrorCode err = PR_GetError();
SECStatus rv = SECFailure;
CERTCertificate *peerCert = SSL_PeerCertificate(socket);
+ const char *hostname_note;
switch (err) {
case SSL_ERROR_BAD_CERT_DOMAIN:
- if (c->remote_host != NULL) {
- rv = CERT_VerifyCertName(peerCert, c->remote_host);
- if (rv != SECSuccess) {
- char *remote = CERT_GetCommonName(&peerCert->subject);
+ if (sc->proxy_ssl_check_peer_cn == TRUE) {
+ if ((hostname_note = apr_table_get(c->notes, "proxy-request-hostname")) != NULL) {
+ apr_table_unset(c->notes, "proxy-request-hostname");
+ rv = CERT_VerifyCertName(peerCert, hostname_note);
+ if (rv != SECSuccess) {
+ char *remote = CERT_GetCommonName(&peerCert->subject);
+ ap_log_error(APLOG_MARK, APLOG_ERR, 0, NULL,
+ "SSL Proxy: Possible man-in-the-middle attack. The remove server is %s, we expected %s", remote, hostname_note);
+ PORT_Free(remote);
+ }
+ } else {
ap_log_error(APLOG_MARK, APLOG_ERR, 0, NULL,
- "SSL Proxy: Possible man-in-the-middle attack. The remove server is %s, we expected %s", remote, c->remote_host);
- PORT_Free(remote);
+ "SSL Proxy: I don't have the name of the host we're supposed to connect to so I can't verify that we are connecting to who we think we should be. Giving up.");
}
} else {
- ap_log_error(APLOG_MARK, APLOG_ERR, 0, NULL,
- "SSL Proxy: I don't have the name of the host we're supposed to connect to so I can't verify that we are connecting to who we think we should be. Giving up. Hint: See Apache bug 36468.");
+ rv = SECSuccess;
}
break;
default:
Index: mod_nss.h
===================================================================
RCS file: /cvs/dirsec/mod_nss/mod_nss.h,v
retrieving revision 1.21
retrieving revision 1.22
diff -u -r1.21 -r1.22
--- mod_nss.h 2 Mar 2010 20:12:04 -0000 1.21
+++ mod_nss.h 13 May 2010 15:21:25 -0000 1.22
@@ -306,6 +306,7 @@
int vhost_id_len;
modnss_ctx_t *server;
modnss_ctx_t *proxy;
+ BOOL proxy_ssl_check_peer_cn;
};
/*
@@ -410,6 +411,7 @@
const char *nss_cmd_NSSProxyProtocol(cmd_parms *, void *, const char *);
const char *nss_cmd_NSSProxyCipherSuite(cmd_parms *, void *, const char *);
const char *nss_cmd_NSSProxyNickname(cmd_parms *cmd, void *dcfg, const char *arg);
+const char *nss_cmd_NSSProxyCheckPeerCN(cmd_parms *cmd, void *dcfg, int flag);
/* module initialization */
int nss_init_Module(apr_pool_t *, apr_pool_t *, apr_pool_t *, server_rec *);
Index: nss_engine_config.c
===================================================================
RCS file: /cvs/dirsec/mod_nss/nss_engine_config.c,v
retrieving revision 1.16
retrieving revision 1.17
diff -u -r1.16 -r1.17
--- nss_engine_config.c 2 Mar 2010 20:12:05 -0000 1.16
+++ nss_engine_config.c 13 May 2010 15:21:25 -0000 1.17
@@ -140,6 +140,7 @@
sc->vhost_id_len = 0; /* set during module init */
sc->proxy = NULL;
sc->server = NULL;
+ sc->proxy_ssl_check_peer_cn = TRUE;
modnss_ctx_init_proxy(sc, p);
@@ -214,6 +215,7 @@
cfgMergeBool(fips);
cfgMergeBool(enabled);
cfgMergeBool(proxy_enabled);
+ cfgMergeBool(proxy_ssl_check_peer_cn);
modnss_ctx_cfg_merge_proxy(base->proxy, add->proxy, mrg->proxy);
@@ -544,6 +546,15 @@
return NULL;
}
+const char *nss_cmd_NSSProxyCheckPeerCN(cmd_parms *cmd, void *dcfg, int flag)
+{
+ SSLSrvConfigRec *sc = mySrvConfig(cmd->server);
+
+ sc->proxy_ssl_check_peer_cn = flag ? TRUE : FALSE;
+
+ return NULL;
+}
+
const char *nss_cmd_NSSEnforceValidCerts(cmd_parms *cmd,
void *dcfg,
int flag)
13 years, 10 months
mod_nss/docs mod_nss.html,1.11,1.12
by Rob Crittenden
Author: rcritten
Update of /cvs/dirsec/mod_nss/docs
In directory cvs01.phx2.fedoraproject.org:/tmp/cvs-serv30758/docs
Modified Files:
mod_nss.html
Log Message:
Compare CN value of remote host with requested host in reverse proxy.
Add configuration option to disable this, defaulting to on.
591224
Index: mod_nss.html
===================================================================
RCS file: /cvs/dirsec/mod_nss/docs/mod_nss.html,v
retrieving revision 1.11
retrieving revision 1.12
diff -u -r1.11 -r1.12
--- mod_nss.html 5 Sep 2006 14:58:56 -0000 1.11
+++ mod_nss.html 13 May 2010 15:21:25 -0000 1.12
@@ -1028,7 +1028,21 @@
<br>
<span style="font-weight: bold;">Example</span><br>
<br>
-<code>NSSProxyNickname beta</code><br>
+<code>NSSProxyNickname beta<br>
+<br>
+</code><big><big>NSSProxyCheckPeerCN</big></big><br>
+<br>
+Compare the CN value of the peer certificate with the hostname being
+requested. If this is set to on, the default, then the request will
+fail if they do not match. If this is set to off then this comparison
+is not done. Note that this test is your only protection against a
+man-in-the-middle attack so leaving this as on is strongly recommended.<br>
+<br>
+<span style="font-weight: bold;">Example</span><br>
+<br>
+<span style="font-family: monospace;">NSSProcyCheckPeerCN</span><code>
+on<br>
+</code><br>
<h1><a name="Environment"></a>Environment Variables</h1>
Quite a few environment variables (for CGI and SSI) may be set
depending on the NSSOptions configuration. It can be expensive to set
@@ -1435,42 +1449,9 @@
<h1><a name="FAQ"></a>Frequently Asked Questions</h1>
Q. Does mod_nss support mod_proxy?<br>
<br>
-A. In order to use the mod_nss proxy support you will need to build
-your own mod_proxy by applying a patch found in bug <a
- href="http://issues.apache.org/bugzilla/show_bug.cgi?id=36468">36468</a>.
-The patch is needed so we can compare the hostname contained in the
-remote certificate with the hostname you meant to visit. This prevents
-man-in-the-middle attacks.<br>
-<br>
-You also have to change the SSL functions that mod_proxy looks to use.
-You'll need to apply this patch:<br>
-<br>
-<code>1038,1039c1038,1039<br>
-< APR_DECLARE_OPTIONAL_FN(int, ssl_proxy_enable, (conn_rec *));<br>
-< APR_DECLARE_OPTIONAL_FN(int, ssl_engine_disable, (conn_rec *));<br>
----<br>
-> APR_DECLARE_OPTIONAL_FN(int, nss_proxy_enable, (conn_rec *));<br>
-> APR_DECLARE_OPTIONAL_FN(int, nss_engine_disable, (conn_rec *));<br>
-1041,1042c1041,1042<br>
-< static APR_OPTIONAL_FN_TYPE(ssl_proxy_enable) *proxy_ssl_enable =
-NULL;<br>
-< static APR_OPTIONAL_FN_TYPE(ssl_engine_disable) *proxy_ssl_disable
-= NULL;<br>
----<br>
-> static APR_OPTIONAL_FN_TYPE(nss_proxy_enable) *proxy_ssl_enable =
-NULL;<br>
-> static APR_OPTIONAL_FN_TYPE(nss_engine_disable) *proxy_ssl_disable
-= NULL;<br>
-1069,1070c1069,1070<br>
-< proxy_ssl_enable =
-APR_RETRIEVE_OPTIONAL_FN(ssl_proxy_enable);<br>
-< proxy_ssl_disable =
-APR_RETRIEVE_OPTIONAL_FN(ssl_engine_disable);<br>
----<br>
-> proxy_ssl_enable =
-APR_RETRIEVE_OPTIONAL_FN(nss_proxy_enable);<br>
-> proxy_ssl_disable =
-APR_RETRIEVE_OPTIONAL_FN(nss_engine_disable);<br>
-</code><br>
+A. Yes but you need to make sure that mod_ssl is not loaded. mod_proxy
+provides a single interface for SSL providers and mod_nss defers to
+mod_ssl
+if it is loaded.
</body>
</html>
13 years, 10 months
Branch 'Directory_Server_8_2_Branch' - ldap/servers
by Noriko Hosoi
ldap/servers/slapd/back-ldbm/dblayer.c | 140 ++++++++++++++++---------
ldap/servers/slapd/back-ldbm/dblayer.h | 8 +
ldap/servers/slapd/back-ldbm/ldif2ldbm.c | 5
ldap/servers/slapd/back-ldbm/proto-back-ldbm.h | 2
4 files changed, 104 insertions(+), 51 deletions(-)
New commits:
commit a5a9949fd16c6d268f0f2676f8bdbb926dc43539
Author: Noriko Hosoi <nhosoi(a)redhat.com>
Date: Wed May 12 15:48:42 2010 -0700
590931 - rhds81 import - hardcoded pages_limit for nsslapd-import-cache-autosize
Fix Description:
1. Got rid of the old hardcoded limit 200MB.
2. Introduced the memory hard limit and soft limit.
Standalone command line import ldif2db behaves as follows:
If import cache autosize is enabled:
nsslapd-import-cache-autosize: -1 or 1 ~ 99
(if the value is greater than or equal to 100,
it's reset to 50 with a warning.)
the import cache size is calculated as
nsslapd-import-cache-autosize * pages / 125
(./125 instead of ./100 is for adjusting the BDB overhead.)
If import cache is disabled:
nsslapd-import-cache-autosize: 0
get the nsslapd-import-cachesize.
Calculate the memory size left after allocating the import cache size.
If the size is less than the hard limit, it issues an error and quit.
If the size is greater than the hard limit and less than the soft limit,
it issues a warning, but continues the import task.
Note: this function is called only if the import is executed as a stand
alone command line (ldif2db).
diff --git a/ldap/servers/slapd/back-ldbm/dblayer.c b/ldap/servers/slapd/back-ldbm/dblayer.c
index c60451f..08df608 100644
--- a/ldap/servers/slapd/back-ldbm/dblayer.c
+++ b/ldap/servers/slapd/back-ldbm/dblayer.c
@@ -869,7 +869,7 @@ void dblayer_sys_pages(size_t *pagesize, size_t *pages, size_t *procpages, size_
#ifdef OS_solaris
*pagesize = (int)sysconf(_SC_PAGESIZE);
*pages = (int)sysconf(_SC_PHYS_PAGES);
- *availpages = dblayer_getvirtualmemsize() / *pagesize;
+ *availpages = dblayer_getvirtualmemsize() / *pagesize;
/* solaris has THE most annoying way to get this info */
if (procpages) {
struct prpsinfo psi;
@@ -1757,59 +1757,105 @@ int dblayer_start(struct ldbminfo *li, int dbmode)
return 0;
}
-void
-autosize_import_cache(struct ldbminfo *li)
+/*
+ * If import cache autosize is enabled:
+ * nsslapd-import-cache-autosize: -1 or 1 ~ 99
+ * calculate the import cache size.
+ * If import cache is disabled:
+ * nsslapd-import-cache-autosize: 0
+ * get the nsslapd-import-cachesize.
+ * Calculate the memory size left after allocating the import cache size.
+ * If the size is less than the hard limit, it issues an error and quit.
+ * If the size is greater than the hard limit and less than the soft limit,
+ * it issues a warning, but continues the import task.
+ *
+ * Note: this function is called only if the import is executed as a stand
+ * alone command line (ldif2db).
+ */
+int
+check_and_set_import_cache(struct ldbminfo *li)
{
+ size_t import_pages = 0;
+ size_t pagesize, pages, procpages, availpages;
+ size_t soft_limit = 0;
+ size_t hard_limit = 0;
+ size_t page_delta = 0;
+ char s[64]; /* big enough to hold %ld */
+
+ dblayer_sys_pages(&pagesize, &pages, &procpages, &availpages);
+ if (0 == pagesize || 0 == pages) {
+ LDAPDebug2Args(LDAP_DEBUG_ANY, "check_and_set_import_cache: "
+ "Failed to get pagesize: %ld or pages: %ld\n",
+ pagesize, pages);
+ return ENOENT;
+ }
+ LDAPDebug(LDAP_DEBUG_ANY, "check_and_set_import_cache: "
+ "pagesize: %ld, pages: %ld, procpages: %ld\n",
+ pagesize, pages, procpages);
+
+ /* Soft limit: pages equivalent to 1GB (defined in dblayer.h) */
+ soft_limit = (DBLAYER_IMPORTCACHESIZE_SL*1024) / (pagesize/1024);
+ /* Hard limit: pages equivalent to 100MB (defined in dblayer.h) */
+ hard_limit = (DBLAYER_IMPORTCACHESIZE_HL*1024) / (pagesize/1024);
/*
* default behavior for ldif2db import cache,
* nsslapd-import-cache-autosize==-1,
* autosize 50% mem to import cache
*/
- if (li->li_import_cache_autosize == -1) {
+ if (li->li_import_cache_autosize < 0) {
li->li_import_cache_autosize = 50;
}
/* sanity check */
- if (li->li_import_cache_autosize > 100) {
- LDAPDebug(LDAP_DEBUG_ANY,
- "cache autosizing: bad setting, "
- "import cache autosizing value should not be larger than 100(%).\n"
- "set: 100(%).\n", NULL, NULL, NULL);
- li->li_import_cache_autosize = 100;
+ if (li->li_import_cache_autosize >= 100) {
+ LDAPDebug0Args(LDAP_DEBUG_ANY,
+ "check_and_set_import_cache: "
+ "import cache autosizing value "
+ "(nsslapd-import-cache-autosize) should not be "
+ "greater than or equal to 100(%). Reset to 50(%).\n");
+ li->li_import_cache_autosize = 50;
}
- /* autosizing importCache */
- if (li->li_import_cache_autosize > 0) {
- size_t pagesize, pages, procpages, availpages;
+ if (li->li_import_cache_autosize == 0) {
+ /* user specified importCache */
+ import_pages = li->li_import_cachesize / pagesize;
- dblayer_sys_pages(&pagesize, &pages, &procpages, &availpages);
- LDAPDebug(LDAP_DEBUG_ANY, "autosize_import_cache: "
- "pagesize: %d, pages: %d, procpages: %d\n",
- pagesize, pages, procpages);
- if (pagesize) {
- char s[32]; /* big enough to hold %ld */
- int import_pages;
- int pages_limit = (200 * 1024) / (pagesize/1024);
- import_pages = (li->li_import_cache_autosize * pages) / 125;
- /* We don't want to go wild with memory when auto-sizing, cap the
- * cache size at 200 Megs to try to avoid situations where we
- * attempt to allocate more memory than there is free page pool for, or
- * where there's some system limit on the size of process memory
- */
- if (import_pages > pages_limit) {
- import_pages = pages_limit;
- }
- LDAPDebug(LDAP_DEBUG_ANY, "cache autosizing: import cache: %dk \n",
- import_pages*(pagesize/1024), NULL, NULL);
- LDAPDebug(LDAP_DEBUG_ANY,
- "li_import_cache_autosize: %d, import_pages: %d, pagesize: %d\n",
- li->li_import_cache_autosize, import_pages,
- pagesize);
+ } else {
+ /* autosizing importCache */
+ /* ./125 instead of ./100 is for adjusting the BDB overhead. */
+ import_pages = (li->li_import_cache_autosize * pages) / 125;
+ }
- sprintf(s, "%lu", (unsigned long)(import_pages * pagesize));
- ldbm_config_internal_set(li, CONFIG_IMPORT_CACHESIZE, s);
- }
+ page_delta = pages - import_pages;
+ if (page_delta < hard_limit) {
+ LDAPDebug(LDAP_DEBUG_ANY,
+ "After allocating import cache %ldKB, "
+ "the available memory is %ldKB, "
+ "which is less than the hard limit %ldKB. "
+ "Please decrease the import cache size and rerun import.\n",
+ import_pages*(pagesize/1024), page_delta*(pagesize/1024),
+ hard_limit*(pagesize/1024));
+ return ENOMEM;
+ }
+ if (page_delta < soft_limit) {
+ LDAPDebug(LDAP_DEBUG_ANY,
+ "WARNING: After allocating import cache %ldKB, "
+ "the available memory is %ldKB, "
+ "which is less than the soft limit %ldKB. "
+ "You may want to decrease the import cache size and "
+ "rerun import.\n",
+ import_pages*(pagesize/1024), page_delta*(pagesize/1024),
+ soft_limit*(pagesize/1024));
+ }
+
+ LDAPDebug1Arg(LDAP_DEBUG_ANY, "Import allocates %ldKB import cache.\n",
+ import_pages*(pagesize/1024));
+ if (li->li_import_cache_autosize > 0) { /* import cache autosizing */
+ /* set the calculated import cache size to the config */
+ sprintf(s, "%lu", (unsigned long)(import_pages * pagesize));
+ ldbm_config_internal_set(li, CONFIG_IMPORT_CACHESIZE, s);
}
+ return 0;
}
/* mode is one of
@@ -1987,15 +2033,13 @@ int dblayer_instance_start(backend *be, int mode)
oflags |= DB_PRIVATE;
}
PR_Lock(li->li_config_mutex);
- if ((li->li_flags & SLAPI_TASK_RUNNING_FROM_COMMANDLINE) &&
- (li->li_import_cache_autosize)) /* Autosizing importCache
- * Need to re-eval every time
- * to guarantee the memory is
- * really available
- * (just for command line I/F)
- */
- {
- autosize_import_cache(li);
+ /* import cache checking and autosizing is available only
+ * for the command line */
+ if (li->li_flags & SLAPI_TASK_RUNNING_FROM_COMMANDLINE) {
+ return_value = check_and_set_import_cache(li);
+ if (return_value) {
+ goto out;
+ }
}
cachesize = li->li_import_cachesize;
PR_Unlock(li->li_config_mutex);
diff --git a/ldap/servers/slapd/back-ldbm/dblayer.h b/ldap/servers/slapd/back-ldbm/dblayer.h
index c30fe43..d0d05b4 100644
--- a/ldap/servers/slapd/back-ldbm/dblayer.h
+++ b/ldap/servers/slapd/back-ldbm/dblayer.h
@@ -97,6 +97,14 @@
#define DB_REGION_NAME 25 /* DB: named regions, no backing file. */
#endif
+/* Used in check_and_set_import_cache */
+/* After allocating the import cache, free memory must be left more than
+ * the hard limit to run import. */
+/* If the free memory size left is greater than hard limit and less than
+ * soft limit, the import utility issues a warning, but it runs */
+#define DBLAYER_IMPORTCACHESIZE_HL 100 /* import cache hard limit 100MB */
+#define DBLAYER_IMPORTCACHESIZE_SL 1024 /* import cache soft limit 1GB */
+
struct dblayer_private_env {
DB_ENV *dblayer_DB_ENV;
PRRWLock * dblayer_env_lock;
diff --git a/ldap/servers/slapd/back-ldbm/ldif2ldbm.c b/ldap/servers/slapd/back-ldbm/ldif2ldbm.c
index fc07443..79f7225 100644
--- a/ldap/servers/slapd/back-ldbm/ldif2ldbm.c
+++ b/ldap/servers/slapd/back-ldbm/ldif2ldbm.c
@@ -584,7 +584,6 @@ int ldbm_back_ldif2ldbm( Slapi_PBlock *pb )
li->li_flags |= SLAPI_TASK_RUNNING_FROM_COMMANDLINE;
ldbm_config_load_dse_info(li);
- autosize_import_cache(li);
}
/* Find the instance that the ldif2db will be done on. */
@@ -2058,7 +2057,9 @@ int ldbm_back_upgradedb(Slapi_PBlock *pb)
{
ldbm_config_load_dse_info(li);
}
- autosize_import_cache(li);
+ if (check_and_set_import_cache(li) < 0) {
+ return -1;
+ }
}
else
{
diff --git a/ldap/servers/slapd/back-ldbm/proto-back-ldbm.h b/ldap/servers/slapd/back-ldbm/proto-back-ldbm.h
index c8bf3e2..96b4320 100644
--- a/ldap/servers/slapd/back-ldbm/proto-back-ldbm.h
+++ b/ldap/servers/slapd/back-ldbm/proto-back-ldbm.h
@@ -164,7 +164,7 @@ void dblayer_set_recovery_required(struct ldbminfo *li);
char *dblayer_get_home_dir(struct ldbminfo *li, int *dbhome);
char *dblayer_get_full_inst_dir(struct ldbminfo *li, ldbm_instance *inst,
char *buf, int buflen);
-void autosize_import_cache(struct ldbminfo *li);
+int check_and_set_import_cache(struct ldbminfo *li);
int dblayer_db_uses_locking(DB_ENV *db_env);
int dblayer_db_uses_transactions(DB_ENV *db_env);
13 years, 10 months
ldap/servers
by Noriko Hosoi
ldap/servers/slapd/back-ldbm/dblayer.c | 140 ++++++++++++++++---------
ldap/servers/slapd/back-ldbm/dblayer.h | 8 +
ldap/servers/slapd/back-ldbm/ldif2ldbm.c | 5
ldap/servers/slapd/back-ldbm/proto-back-ldbm.h | 2
4 files changed, 104 insertions(+), 51 deletions(-)
New commits:
commit d78de3617b6d6aa3928e3a88b2cba83fec4eaaab
Author: Noriko Hosoi <nhosoi(a)redhat.com>
Date: Wed May 12 15:48:42 2010 -0700
590931 - rhds81 import - hardcoded pages_limit for nsslapd-import-cache-autosize
Fix Description:
1. Got rid of the old hardcoded limit 200MB.
2. Introduced the memory hard limit and soft limit.
Standalone command line import ldif2db behaves as follows:
If import cache autosize is enabled:
nsslapd-import-cache-autosize: -1 or 1 ~ 99
(if the value is greater than or equal to 100,
it's reset to 50 with a warning.)
the import cache size is calculated as
nsslapd-import-cache-autosize * pages / 125
(./125 instead of ./100 is for adjusting the BDB overhead.)
If import cache is disabled:
nsslapd-import-cache-autosize: 0
get the nsslapd-import-cachesize.
Calculate the memory size left after allocating the import cache size.
If the size is less than the hard limit, it issues an error and quit.
If the size is greater than the hard limit and less than the soft limit,
it issues a warning, but continues the import task.
Note: this function is called only if the import is executed as a stand
alone command line (ldif2db).
diff --git a/ldap/servers/slapd/back-ldbm/dblayer.c b/ldap/servers/slapd/back-ldbm/dblayer.c
index b3d61db..e744321 100644
--- a/ldap/servers/slapd/back-ldbm/dblayer.c
+++ b/ldap/servers/slapd/back-ldbm/dblayer.c
@@ -869,7 +869,7 @@ void dblayer_sys_pages(size_t *pagesize, size_t *pages, size_t *procpages, size_
#ifdef OS_solaris
*pagesize = (int)sysconf(_SC_PAGESIZE);
*pages = (int)sysconf(_SC_PHYS_PAGES);
- *availpages = dblayer_getvirtualmemsize() / *pagesize;
+ *availpages = dblayer_getvirtualmemsize() / *pagesize;
/* solaris has THE most annoying way to get this info */
if (procpages) {
struct prpsinfo psi;
@@ -1757,59 +1757,105 @@ int dblayer_start(struct ldbminfo *li, int dbmode)
return 0;
}
-void
-autosize_import_cache(struct ldbminfo *li)
+/*
+ * If import cache autosize is enabled:
+ * nsslapd-import-cache-autosize: -1 or 1 ~ 99
+ * calculate the import cache size.
+ * If import cache is disabled:
+ * nsslapd-import-cache-autosize: 0
+ * get the nsslapd-import-cachesize.
+ * Calculate the memory size left after allocating the import cache size.
+ * If the size is less than the hard limit, it issues an error and quit.
+ * If the size is greater than the hard limit and less than the soft limit,
+ * it issues a warning, but continues the import task.
+ *
+ * Note: this function is called only if the import is executed as a stand
+ * alone command line (ldif2db).
+ */
+int
+check_and_set_import_cache(struct ldbminfo *li)
{
+ size_t import_pages = 0;
+ size_t pagesize, pages, procpages, availpages;
+ size_t soft_limit = 0;
+ size_t hard_limit = 0;
+ size_t page_delta = 0;
+ char s[64]; /* big enough to hold %ld */
+
+ dblayer_sys_pages(&pagesize, &pages, &procpages, &availpages);
+ if (0 == pagesize || 0 == pages) {
+ LDAPDebug2Args(LDAP_DEBUG_ANY, "check_and_set_import_cache: "
+ "Failed to get pagesize: %ld or pages: %ld\n",
+ pagesize, pages);
+ return ENOENT;
+ }
+ LDAPDebug(LDAP_DEBUG_ANY, "check_and_set_import_cache: "
+ "pagesize: %ld, pages: %ld, procpages: %ld\n",
+ pagesize, pages, procpages);
+
+ /* Soft limit: pages equivalent to 1GB (defined in dblayer.h) */
+ soft_limit = (DBLAYER_IMPORTCACHESIZE_SL*1024) / (pagesize/1024);
+ /* Hard limit: pages equivalent to 100MB (defined in dblayer.h) */
+ hard_limit = (DBLAYER_IMPORTCACHESIZE_HL*1024) / (pagesize/1024);
/*
* default behavior for ldif2db import cache,
* nsslapd-import-cache-autosize==-1,
* autosize 50% mem to import cache
*/
- if (li->li_import_cache_autosize == -1) {
+ if (li->li_import_cache_autosize < 0) {
li->li_import_cache_autosize = 50;
}
/* sanity check */
- if (li->li_import_cache_autosize > 100) {
- LDAPDebug(LDAP_DEBUG_ANY,
- "cache autosizing: bad setting, "
- "import cache autosizing value should not be larger than 100(%).\n"
- "set: 100(%).\n", NULL, NULL, NULL);
- li->li_import_cache_autosize = 100;
+ if (li->li_import_cache_autosize >= 100) {
+ LDAPDebug0Args(LDAP_DEBUG_ANY,
+ "check_and_set_import_cache: "
+ "import cache autosizing value "
+ "(nsslapd-import-cache-autosize) should not be "
+ "greater than or equal to 100(%). Reset to 50(%).\n");
+ li->li_import_cache_autosize = 50;
}
- /* autosizing importCache */
- if (li->li_import_cache_autosize > 0) {
- size_t pagesize, pages, procpages, availpages;
+ if (li->li_import_cache_autosize == 0) {
+ /* user specified importCache */
+ import_pages = li->li_import_cachesize / pagesize;
- dblayer_sys_pages(&pagesize, &pages, &procpages, &availpages);
- LDAPDebug(LDAP_DEBUG_ANY, "autosize_import_cache: "
- "pagesize: %d, pages: %d, procpages: %d\n",
- pagesize, pages, procpages);
- if (pagesize) {
- char s[32]; /* big enough to hold %ld */
- int import_pages;
- int pages_limit = (200 * 1024) / (pagesize/1024);
- import_pages = (li->li_import_cache_autosize * pages) / 125;
- /* We don't want to go wild with memory when auto-sizing, cap the
- * cache size at 200 Megs to try to avoid situations where we
- * attempt to allocate more memory than there is free page pool for, or
- * where there's some system limit on the size of process memory
- */
- if (import_pages > pages_limit) {
- import_pages = pages_limit;
- }
- LDAPDebug(LDAP_DEBUG_ANY, "cache autosizing: import cache: %dk \n",
- import_pages*(pagesize/1024), NULL, NULL);
- LDAPDebug(LDAP_DEBUG_ANY,
- "li_import_cache_autosize: %d, import_pages: %d, pagesize: %d\n",
- li->li_import_cache_autosize, import_pages,
- pagesize);
+ } else {
+ /* autosizing importCache */
+ /* ./125 instead of ./100 is for adjusting the BDB overhead. */
+ import_pages = (li->li_import_cache_autosize * pages) / 125;
+ }
- sprintf(s, "%lu", (unsigned long)(import_pages * pagesize));
- ldbm_config_internal_set(li, CONFIG_IMPORT_CACHESIZE, s);
- }
+ page_delta = pages - import_pages;
+ if (page_delta < hard_limit) {
+ LDAPDebug(LDAP_DEBUG_ANY,
+ "After allocating import cache %ldKB, "
+ "the available memory is %ldKB, "
+ "which is less than the hard limit %ldKB. "
+ "Please decrease the import cache size and rerun import.\n",
+ import_pages*(pagesize/1024), page_delta*(pagesize/1024),
+ hard_limit*(pagesize/1024));
+ return ENOMEM;
+ }
+ if (page_delta < soft_limit) {
+ LDAPDebug(LDAP_DEBUG_ANY,
+ "WARNING: After allocating import cache %ldKB, "
+ "the available memory is %ldKB, "
+ "which is less than the soft limit %ldKB. "
+ "You may want to decrease the import cache size and "
+ "rerun import.\n",
+ import_pages*(pagesize/1024), page_delta*(pagesize/1024),
+ soft_limit*(pagesize/1024));
+ }
+
+ LDAPDebug1Arg(LDAP_DEBUG_ANY, "Import allocates %ldKB import cache.\n",
+ import_pages*(pagesize/1024));
+ if (li->li_import_cache_autosize > 0) { /* import cache autosizing */
+ /* set the calculated import cache size to the config */
+ sprintf(s, "%lu", (unsigned long)(import_pages * pagesize));
+ ldbm_config_internal_set(li, CONFIG_IMPORT_CACHESIZE, s);
}
+ return 0;
}
/* mode is one of
@@ -2011,15 +2057,13 @@ int dblayer_instance_start(backend *be, int mode)
oflags |= DB_PRIVATE;
}
PR_Lock(li->li_config_mutex);
- if ((li->li_flags & SLAPI_TASK_RUNNING_FROM_COMMANDLINE) &&
- (li->li_import_cache_autosize)) /* Autosizing importCache
- * Need to re-eval every time
- * to guarantee the memory is
- * really available
- * (just for command line I/F)
- */
- {
- autosize_import_cache(li);
+ /* import cache checking and autosizing is available only
+ * for the command line */
+ if (li->li_flags & SLAPI_TASK_RUNNING_FROM_COMMANDLINE) {
+ return_value = check_and_set_import_cache(li);
+ if (return_value) {
+ goto out;
+ }
}
cachesize = li->li_import_cachesize;
PR_Unlock(li->li_config_mutex);
diff --git a/ldap/servers/slapd/back-ldbm/dblayer.h b/ldap/servers/slapd/back-ldbm/dblayer.h
index c30fe43..d0d05b4 100644
--- a/ldap/servers/slapd/back-ldbm/dblayer.h
+++ b/ldap/servers/slapd/back-ldbm/dblayer.h
@@ -97,6 +97,14 @@
#define DB_REGION_NAME 25 /* DB: named regions, no backing file. */
#endif
+/* Used in check_and_set_import_cache */
+/* After allocating the import cache, free memory must be left more than
+ * the hard limit to run import. */
+/* If the free memory size left is greater than hard limit and less than
+ * soft limit, the import utility issues a warning, but it runs */
+#define DBLAYER_IMPORTCACHESIZE_HL 100 /* import cache hard limit 100MB */
+#define DBLAYER_IMPORTCACHESIZE_SL 1024 /* import cache soft limit 1GB */
+
struct dblayer_private_env {
DB_ENV *dblayer_DB_ENV;
PRRWLock * dblayer_env_lock;
diff --git a/ldap/servers/slapd/back-ldbm/ldif2ldbm.c b/ldap/servers/slapd/back-ldbm/ldif2ldbm.c
index 60b72db..76404d5 100644
--- a/ldap/servers/slapd/back-ldbm/ldif2ldbm.c
+++ b/ldap/servers/slapd/back-ldbm/ldif2ldbm.c
@@ -632,7 +632,6 @@ int ldbm_back_ldif2ldbm( Slapi_PBlock *pb )
li->li_flags |= SLAPI_TASK_RUNNING_FROM_COMMANDLINE;
ldbm_config_load_dse_info(li);
- autosize_import_cache(li);
}
/* Find the instance that the ldif2db will be done on. */
@@ -2484,7 +2483,9 @@ int ldbm_back_upgradedb(Slapi_PBlock *pb)
{
ldbm_config_load_dse_info(li);
}
- autosize_import_cache(li);
+ if (check_and_set_import_cache(li) < 0) {
+ return -1;
+ }
}
else
{
diff --git a/ldap/servers/slapd/back-ldbm/proto-back-ldbm.h b/ldap/servers/slapd/back-ldbm/proto-back-ldbm.h
index f12d41d..41fd54c 100644
--- a/ldap/servers/slapd/back-ldbm/proto-back-ldbm.h
+++ b/ldap/servers/slapd/back-ldbm/proto-back-ldbm.h
@@ -164,7 +164,7 @@ void dblayer_set_recovery_required(struct ldbminfo *li);
char *dblayer_get_home_dir(struct ldbminfo *li, int *dbhome);
char *dblayer_get_full_inst_dir(struct ldbminfo *li, ldbm_instance *inst,
char *buf, int buflen);
-void autosize_import_cache(struct ldbminfo *li);
+int check_and_set_import_cache(struct ldbminfo *li);
int dblayer_db_uses_locking(DB_ENV *db_env);
int dblayer_db_uses_transactions(DB_ENV *db_env);
13 years, 10 months