for reference [Fwd: regarding spotting unowned directories in fedora]
by Rakesh Pandit
---------- Forwarded message ----------
From: Michael Schwendt <mschwendt(a)gmail.com>
Date: 2009/11/29
Subject: Re: regarding spotting unowned directories in fedora
To: Rakesh Pandit <rakesh.pandit(a)gmail.com>
On Fri, 27 Nov 2009 14:18:35 +0530, Rakesh wrote:
> Hello Michael,
>
> I believe you have a script which can figure out unowned directories
> in a package. May you post it to me. I want to have a look at it and
> develop a check which can go into gach[1]. I also request you to fill
> up some tickets for all guideline checks which you consider can be
> automated into out trac instance[2], in case you feel like doing it :)
>
> I haven't been able to invest time in this project for long now, but
> these days I am fairly active and am planning release near new year
> eve. It would be great if you can help in figuring out what all checks
> can be automated.
>
> As you might already know I plan to try implementing checks in rpmlint
> .. but in case they don't go there and are hybrid checks as in require
> inspection of srpms, logs, source in mix, or interaction with koji, or
> other build systems will do that.
>
> Thanks,
>
> [1] https://fedorahosted.org/gach
> [2] https://fedorahosted.org/gach/newticket
Hi!
The tool is called "dircheck-remote" and has not been developed
further since Jan/Feb. As I've pointed to it in bugzilla, it's
still found in a public place, though:
http://mschwendt.fedorapeople.org/dircheck-remote.py
Yum API has changed a few months ago once more and required a small
update.
By default, it evaluates yum.conf and enabled repos, but it can
also be told to look at specific repos or specific packages only,
such as:
$ dircheck-remote.py -n ^ypserv
[...]
=> ypserv-2.19-13.fc12.src.rpm
=> ypserv-2.19-13.fc12.i686 (fedora)
/usr/include/rpcsvc
provided by: 1:quota-devel-3.17-8.fc12.i686
provided by: glibc-headers-2.11-2.i686
Any path it prints, is "unowned". The "provided by" lines are hints
about what other packages own a directory.
Note that one big problem caused by unowned directories is gone
as of RPM 4.4.2.3:
https://fedoraproject.org/wiki/Packaging:UnownedDirectories#Inaccessible_...
With that item no longer being an issue, the problem of unowned
directories is less interesting. What's left is that -- regularly --
discovering unowned directories leads to discovering major mistakes made
in a package. e.g. misplaced files, bad subpackage dependencies.
The most recent RPM (announced on fedora-devel-list some days ago) will do
ordered package erasures. To include a directory in the _right_ package
instead of in an arbitrary package of a dependency-chain is something that
hasn't been checked before. Basically, with a weird dependency-chain, it
is possible to own a directory in a package that would be installed _last_
and erased _first_. That would lead to unremoved directories _after_
erasing packages. A related problem, but would require a check that
simulates in which order RPM will erase a set of packages.
--
Michael Schwendt <mschwendt(a)fedoraproject.org>
Fedora release 12 (Constantine) - Linux 2.6.31.6-145.fc12.i686.PAE
loadavg: 1.59 1.75 1.83
--
Rakesh Pandit
https://fedoraproject.org/
freedom, friends, features, first
14 years, 5 months
some adjustment to Log check
by Rakesh Pandit
some adjustment to Log check
--
Rakesh Pandit
https://fedoraproject.org/
freedom, friends, features, first
From: Rakesh Pandit <rakesh(a)fedoraproject.org>
Date: Thu, 26 Nov 2009 06:18:01 +0000 (+0530)
Subject: BuildLog.py: Fixed log file exception.
X-Git-Url: http://git.fedorahosted.org/git/gach.git/?p=gach.git;a=commitdiff_plain;h...
BuildLog.py: Fixed log file exception.
LogWarningCheck.py: Adjusted report for no log.
---
diff --git a/BuildLog.py b/BuildLog.py
index 866215d..e95a79d 100644
--- a/BuildLog.py
+++ b/BuildLog.py
@@ -27,7 +27,14 @@ class BuildLog(object):
self.file = file
self.filename = os.path.basename(file)
self.path = os.path.dirname(file)
- fobj = open(self.file, "r")
+ self.log_lines = []
+
+ fobj = None
+ try:
+ fobj = open(self.file, "r")
+ except IOError:
+ # Return in case no log found
+ return
try:
self.log_lines = fobj.readlines()
finally:
diff --git a/LogWarningCheck.py b/LogWarningCheck.py
index c257ab3..cbe4713 100644
--- a/LogWarningCheck.py
+++ b/LogWarningCheck.py
@@ -41,11 +41,14 @@ class LogWarningCheck(Check.Check):
"""
log_file = details["log"]
log = BuildLog(log_file)
- reporter.printDividerInfo("Log info %s" %(log_file))
-
# Grep all lines
lines = log.searchstring(["warning", "Warning", "WARNING"])
+ # Return if no warnings found
+ if not lines:
+ return
+
+ reporter.printDividerInfo("Log info %s" %(log_file))
for line in lines:
reporter.printLogWarnings(line)
14 years, 5 months
Check for warning .. whether a doc subpackage is required
by Rakesh Pandit
ExcessDocFileCheck: New check
--
Rakesh Pandit
https://fedoraproject.org/
freedom, friends, features, first
From: Rakesh Pandit <rakesh(a)fedoraproject.org>
Date: Tue, 24 Nov 2009 10:52:07 +0000 (+0530)
Subject: ExcessDocFileCheck.py: New file.
X-Git-Url: http://git.fedorahosted.org/git/gach.git/?p=gach.git;a=commitdiff_plain;h...
ExcessDocFileCheck.py: New file.
GachApplication.py, GachRPMReviewer.py, GachReviewer.py,
GachUtils.py: Adjusted accordingly.
---
diff --git a/ExcessDocFileCheck.py b/ExcessDocFileCheck.py
new file mode 100644
index 0000000..2b1146d
--- /dev/null
+++ b/ExcessDocFileCheck.py
@@ -0,0 +1,70 @@
+#!/usr/bin/env python
+#
+# Copyright 2009 Rakesh Pandit <rakesh(a)fedoraproject.org>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with translate; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+
+#
+# Check for warning about files in /usr/share/doc more then 1024KB or
+# 100 files and would be nice to go for a separate subpackage.
+#
+
+import os
+
+import RPMCheck
+from GachUtils import extract_rpm, rm_rf, get_folder_size
+from GachReporter import GachReporter
+
+reporter = GachReporter()
+
+class ExcessDocFileCheck(RPMCheck.RPMCheck):
+ """Check whether Doc files are in excess
+ """
+
+ def __init__(self):
+ """
+ """
+ RPMCheck.RPMCheck.__init__(self, "ExcessDocFileCheck")
+
+ def check(self, **details):
+ """
+ """
+ rpm_file = details["rpm"]
+ extract_outdir = extract_rpm(rpm_file)
+ docdir = extract_outdir + "/usr/share/doc/"
+ if os.path.exists(docdir):
+ folder_size = get_folder_size(docdir)
+ # If size greater then 1024K and 100 files
+ if int(folder_size) >= 1024:
+ reporter.printWarning("excess-doc-files-in-docdir",
Size=folder_size, File=rpm_file)
+
+ files = []
+ for dir, subdir, fl in os.walk(docdir):
+ files.append(fl)
+ # Check whether number of files >= 100
+ lg = len(files)
+ if lg >= 100:
+ reporter.printWarning("excess-doc-files-in-docdir",
Numberoffiles=lg, File=rpm_file)
+
+ rm_rf(extract_outdir)
+ return
+
+check = ExcessDocFileCheck()
+
+reporter.addDetails(
+'excess-doc-files-in-docdir',
+'''Doc files in /usr/share/doc more then 1024KB or file count >= 100.'''
+)
diff --git a/GachApplication.py b/GachApplication.py
index fd5459d..a9ea5aa 100644
--- a/GachApplication.py
+++ b/GachApplication.py
@@ -131,10 +131,11 @@ class GachApplication:
# Need to check later why this is needed
print reporter.getReport()
- # Load all src rpm checks checks
reviewer = GachReviewer()
srpmreviewer = GachSrcRPMReviewer()
+ rpmreviewer = GachRPMReviewer()
+ # Load all src rpm checks checks
for file in downloaded_files:
if file.endswith("src.rpm"):
srpmreviewer.review(file)
@@ -143,6 +144,11 @@ class GachApplication:
for file in downloaded_files:
if file.endswith("build.log"):
reviewer.review(log=file)
+ reporter.printDividerInfo("Log check ends")
+
+ # Load all rpm checks
+ for file in downloaded_files:
+ if file.endswith(".rpm") and not file.endswith("src.rpm"):
print reporter.getReport()
rm_rf(srpm_spec_downdir)
diff --git a/GachRPMReviewer.py b/GachRPMReviewer.py
index 7a820cb..b41ead4 100644
--- a/GachRPMReviewer.py
+++ b/GachRPMReviewer.py
@@ -28,14 +28,16 @@ class GachRPMReviewer(GachReviewer.GachReviewer):
def __init__(self):
"""
"""
- pass
+ # All Checks are loaded here.
+ self.default_guidelines = ("ExcessDocFileCheck",)
+ GachReviewer.GachReviewer.__init__(self,
guidelines=self.default_guidelines)
def review(self, rpm):
"""
Implements reviewer for rpm.
"""
- for guide in RPMCheck.RPMCheck.checks:
- guide.check(rpm)
+ for guide in RPMCheck.RPMCheck.rpm_checks:
+ guide.check(rpm=rpm)
return
diff --git a/GachReviewer.py b/GachReviewer.py
index f6beaca..3c7fd98 100644
--- a/GachReviewer.py
+++ b/GachReviewer.py
@@ -64,9 +64,9 @@ class GachReviewer(object):
"""
Implements the loading module.
"""
- print "Guidelines:", self.default_guidelines
+ # print "Guidelines:", self.default_guidelines
for name in self.default_guidelines:
- print "Loaded: ", name
+ # print "Loaded: ", name
file = None
path = None
description = None
diff --git a/GachUtils.py b/GachUtils.py
index b93dad8..b876081 100644
--- a/GachUtils.py
+++ b/GachUtils.py
@@ -21,6 +21,7 @@
#
import os
+import stat
import fnmatch
import glob
import sys
@@ -98,3 +99,20 @@ def istext(s):
t = s.translate(_null_trans, text_characters)
return float(len(t))/len(s) >= 0.30
+
+def get_folder_size(folder):
+ """Return folder size
+ """
+ folder_size = commands.getoutput("du -s " + folder)
+# FIXME: Current method does not ok, but below one even does not
+# work fine. Need to get it right.
+
+# folder_size = 0
+# for (path, dirs, files) in os.walk(folder):
+# for file in files:
+# filename = os.path.join(path, file)
+# if os.path.islink(filename) and not
os.path.exists(os.readlink(filename)):
+# continue
+# folder_size += os.stat(filename)[stat.ST_SIZE]
+ return folder_size.split("\t")[0]
+
14 years, 5 months
fix to Koji build URL
by Rakesh Pandit
$subect
--
Rakesh Pandit
https://fedoraproject.org/
freedom, friends, features, first
commit aca8c665e40a2ca3e5cbf7f02f3ad6810fca5ce5
Author: Rakesh Pandit <rakesh(a)fedoraproject.org>
Date: Mon Nov 23 14:33:10 2009 +0530
GachKojiBuilder.py: Add Koji URL
diff --git a/GachKojiBuilder.py b/GachKojiBuilder.py
index 4f63806..720f265 100644
--- a/GachKojiBuilder.py
+++ b/GachKojiBuilder.py
@@ -85,10 +85,10 @@ class GachKojiBuilder(GachBuilder.GachBuilder):
print "done."
server_source = server_dir + "/" + os.path.basename(source)
build_opts = {"scratch": True, "skip_tag": None}
- print "Started koji build with",
+ print "Started koji build"
self.taskId = self._session.build(server_source, target, build_opts)
self.__logout_session()
- print "taskid", self.taskId, ".. waiting ..."
+ print "Koji Build URL:
http://koji.fedoraproject.org/koji/taskinfo?taskID=" +
str(self.taskId), ".. waiting ..."
if wait:
while True:
state = self.getState()
14 years, 5 months
Just pushed new Log warning Check
by Rakesh Pandit
Just pushed Log Warning Check
--
Rakesh Pandit
https://fedoraproject.org/
freedom, friends, features, first
From: Rakesh Pandit <rakesh(a)fedoraproject.org>
Date: Mon, 23 Nov 2009 11:00:47 +0000 (+0530)
Subject: BuildLog.py: New file
X-Git-Url: http://git.fedorahosted.org/git/gach.git/?p=gach.git;a=commitdiff_plain;h...
BuildLog.py: New file
LogWarningCheck.py: New file
Check.py, GachApplication.py, GachReporter.py, GachReviewer.py,
SampleCheck.py: Minor adjustments to work with new check.
---
diff --git a/BuildLog.py b/BuildLog.py
new file mode 100644
index 0000000..866215d
--- /dev/null
+++ b/BuildLog.py
@@ -0,0 +1,48 @@
+#!/usr/bin/env python
+#
+# Copyright 2009 Rakesh Pandit <rakesh(a)fedoraproject.org>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with translate; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+import os
+
+
+class BuildLog(object):
+ """Log file class"""
+
+ def __init__(self, file):
+ """ """
+ self.file = file
+ self.filename = os.path.basename(file)
+ self.path = os.path.dirname(file)
+ fobj = open(self.file, "r")
+ try:
+ self.log_lines = fobj.readlines()
+ finally:
+ fobj.close()
+
+ def searchstring(self, strglist):
+ """Search string in log"""
+ lines = []
+ for line in self.log_lines:
+ for strg in strglist:
+ flag = 0
+ if strg in line:
+ lines.append(line)
+ flag = 1
+ # Prevents line to be included twice
+ if flag:
+ break
+ return lines
diff --git a/Check.py b/Check.py
index b97006e..722c977 100644
--- a/Check.py
+++ b/Check.py
@@ -32,9 +32,9 @@ class Check(object):
def __init__(self, name):
"""
"""
- if name not in SpecCheck.check_names_seen:
- SpecCheck.checks.append(self)
- SpecCheck.check_names_seen.append(name)
+ if name not in Check.check_names_seen:
+ Check.checks.append(self)
+ Check.check_names_seen.append(name)
self.name = name
def check(self, **details):
diff --git a/GachApplication.py b/GachApplication.py
index fbde2d2..fd5459d 100644
--- a/GachApplication.py
+++ b/GachApplication.py
@@ -126,7 +126,11 @@ class GachApplication:
if file.endswith(".rpm"):
reporter.printDividerInfo("Koji build rpm %s output" %(file))
runRpmlint(reporter, Pkg(file))
-
+
+ # Prints rpmlint report
+ # Need to check later why this is needed
+ print reporter.getReport()
+
# Load all src rpm checks checks
reviewer = GachReviewer()
srpmreviewer = GachSrcRPMReviewer()
@@ -135,6 +139,11 @@ class GachApplication:
if file.endswith("src.rpm"):
srpmreviewer.review(file)
+ # Load all build log checks
+ for file in downloaded_files:
+ if file.endswith("build.log"):
+ reviewer.review(log=file)
+
print reporter.getReport()
rm_rf(srpm_spec_downdir)
diff --git a/GachReporter.py b/GachReporter.py
index 0a5f162..51c06c8 100644
--- a/GachReporter.py
+++ b/GachReporter.py
@@ -144,5 +144,10 @@ class GachReporter(object):
"""
self.report += "\n" + "Rpmlint Messages:" + "\n" + messages + "\n"
+ def printLogWarnings(self, messages):
+ """
+ Adds log messages
+ """
+ self.report += "Log warnings:" + messages
# Abstract reporter interface ends here
diff --git a/GachReviewer.py b/GachReviewer.py
index cbc9451..f6beaca 100644
--- a/GachReviewer.py
+++ b/GachReviewer.py
@@ -55,7 +55,7 @@ class GachReviewer(object):
if guidelines:
self.default_guidelines = guidelines
else:
- self.default_guidelines = ("SampleCheck",)
+ self.default_guidelines = ("SampleCheck", "LogWarningCheck")
# Load all checks.
self.loadChecks()
@@ -66,6 +66,7 @@ class GachReviewer(object):
"""
print "Guidelines:", self.default_guidelines
for name in self.default_guidelines:
+ print "Loaded: ", name
file = None
path = None
description = None
@@ -86,7 +87,10 @@ class GachReviewer(object):
obects.
"""
for guide in Check.Check.checks:
- guide.check(details)
+ if "log" in details:
+ guide.check(log=details["log"])
+ else:
+ guide.check(details)
return
diff --git a/LogWarningCheck.py b/LogWarningCheck.py
new file mode 100644
index 0000000..e49435e
--- /dev/null
+++ b/LogWarningCheck.py
@@ -0,0 +1,56 @@
+#!/usr/bin/env python
+#
+# Copyright 2009 Rakesh Pandit <rakesh(a)fedoraproject.org>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with translate; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+
+#
+# Check: Print warnings from log
+#
+
+import Check
+from BuildLog import BuildLog
+from GachReporter import GachReporter
+
+reporter = GachReporter()
+
+class LogWarningCheck(Check.Check):
+ """Print log warnings
+ """
+
+ def __init__(self):
+ """
+ """
+ Check.Check.__init__(self, "LogWarningCheck")
+
+ def check(self, **details):
+ """
+ """
+ log_file = details["log"]
+ log = BuildLog(log_file)
+ reporter.printDividerInfo("Log info %s:" %(log_file))
+
+ # Grep all lines
+ lines = log.searchstring(["warning", "Warning", "WARNING"])
+
+ for line in lines:
+ reporter.printLogWarnings(line)
+
+ return
+
+check = LogWarningCheck()
+# Ends here
+
diff --git a/SampleCheck.py b/SampleCheck.py
index ddf0f61..36fbb2b 100644
--- a/SampleCheck.py
+++ b/SampleCheck.py
@@ -30,9 +30,9 @@ class SampleCheck(SpecCheck.SpecCheck):
"""Sample constructor"""
SpecCheck.SpecCheck.__init__(self, "SampleCheck")
- def check(self, spec):
+ def check(self, **details):
"""Sample Check"""
- reporter.printInfo(spec, "sample-testing")
+ reporter.printInfo("sample-testing")
# An object to autoregister the check
check=SampleCheck()
14 years, 5 months
few fixes along with binary file check
by Rakesh Pandit
Few fixes which I missed committing with Binary file check
--
Rakesh Pandit
https://fedoraproject.org/
freedom, friends, features, first
commit 7f9b708e1e7e3de11b29ac048cfdef40dd1d666f
Author: Rakesh Pandit <rakesh(a)fedoraproject.org>
Date: Tue Nov 17 14:20:41 2009 +0530
GachReporter.py (locate, istextfile, istext): New functions.
GachUtils.py (_print): fixed details key value printing.
diff --git a/GachReporter.py b/GachReporter.py
index 090dd6c..0a5f162 100644
--- a/GachReporter.py
+++ b/GachReporter.py
@@ -86,7 +86,7 @@ class GachReporter(object):
self.report += "%s: %s" % (type, reason)
for d in details:
- self.report += " %s" %d
+ self.report += " %s: %s" %(d, details[d])
self.report +="\n"
diff --git a/GachUtils.py b/GachUtils.py
index 768b675..b93dad8 100644
--- a/GachUtils.py
+++ b/GachUtils.py
@@ -21,11 +21,18 @@
#
import os
+import fnmatch
+import glob
import sys
+import string
import commands
import tempfile
+text_characters = "".join(map(chr, range(32, 127)) + list("\n\r\t\b"))
+_null_trans = string.maketrans("", "")
+
+
def rm_rf(d):
"""Remove directory recursively
"""
@@ -64,3 +71,30 @@ def get_source_from_srpm(srpm_file, output_dir):
print "rpmbuild failed to build source from srpm"
sys.exit(1)
return
+
+
+def locate(pattern, root=os.curdir):
+ """Locate all files matching supplied filename pattern in and below
+ supplied root directory.
+ """
+ matches = []
+ for dir, subdir, files in os.walk(root):
+ for file in files:
+ if glob.fnmatch.fnmatch(file, pattern):
+ matches.append(file)
+
+ return matches
+
+
+def istextfile(filename, blocksize = 512):
+ return istext(open(filename).read(blocksize))
+
+
+def istext(s):
+ if "\0" in s:
+ return 0
+ if not s:
+ return 1
+ t = s.translate(_null_trans, text_characters)
+ return float(len(t))/len(s) >= 0.30
+
14 years, 5 months
Check: Whether source contains binary files
by Rakesh Pandit
Pushed initial prototype check for following:
+# Check: Whether source contains binary files.
+# URL: https://fedoraproject.org/wiki/Packaging/Guidelines#No_inclusion_of_pre-b...
--
Rakesh Pandit
https://fedoraproject.org/
freedom, friends, features, first
commit 37aafbe670cd4fec768b7982ce216bfa37e35873
Author: Rakesh Pandit <rakesh(a)fedoraproject.org>
Date: Tue Nov 17 14:16:47 2009 +0530
BinaryFilesCheck.py: New Check
diff --git a/BinaryFilesCheck.py b/BinaryFilesCheck.py
new file mode 100644
index 0000000..892b01d
--- /dev/null
+++ b/BinaryFilesCheck.py
@@ -0,0 +1,100 @@
+#!/usr/bin/env python
+#
+# Copyright 2009 Rakesh Pandit <rakesh(a)fedoraproject.org>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with translate; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+
+#
+# Check: Whether source contains binary files.
+# URL: https://fedoraproject.org/wiki/Packaging/Guidelines#No_inclusion_of_pre-b...
+#
+
+import commands
+import tempfile
+import os
+import glob
+import stat
+
+import SrcRPMCheck
+from GachUtils import rm_rf, get_source_from_srpm, locate
+from GachReporter import GachReporter
+from GachUtils import istextfile
+
+reporter = GachReporter()
+
+# *.class, *.dll, *.DS_Store, *.exe, *.jar, *.o, *.pyc, *.pyo, *.so
+list_of_binary_extensions = ['*.class', '*.dll', '*.DS_Store',
'*.exe', '*.jar', '*.o', '*.pyc', '*.pyo', '*.so']
+
+class BinaryFilesCheck(SrcRPMCheck.SrcRPMCheck):
+ """Binary files present in source files
+ """
+
+ def __init__(self):
+ """
+ """
+ SrcRPMCheck.SrcRPMCheck.__init__(self, "BinaryFilesCheck")
+
+ def check(self, **details):
+ """
+ """
+ srpm_file = details["srpm"]
+ srpm_extract_dir = tempfile.mkdtemp(prefix="gach")
+ build_dir = srpm_extract_dir + "/BUILD/"
+ get_source_from_srpm(srpm_file, srpm_extract_dir)
+
+ binary_files_with_extension = {}
+ for extension in list_of_binary_extensions:
+ matches = locate(extension, root=build_dir)
+ if matches:
+ binary_files_with_extension[extension] = matches
+
+ for extension in binary_files_with_extension:
+ for file in binary_files_with_extension[extension]:
+
reporter.printWarning("binary-extension-file-in-source", File=file)
+
+ binary_files_nontext_chars = []
+ empty_files = []
+ for dir, subdir, files in os.walk(build_dir):
+ for file in files:
+ fullpath = os.path.join(dir, file)
+ # Knock out empty files
+ if os.stat(fullpath)[stat.ST_SIZE] == 0:
+ empty_files.append(fullpath)
+ continue
+ if istextfile(fullpath):
+ binary_files_nontext_chars.append(fullpath)
+
+ if binary_files_nontext_chars:
+ for binary in binary_files_nontext_chars:
+
reporter.printWarning("binary-file-with-30%-plus-non-text-chars",
File=binary)
+
+ if empty_files:
+ for empty_file in empty_files:
+ reporter.printWarning("empty-file", File=empty_file)
+
+ rm_rf(srpm_extract_dir)
+ return
+
+check = BinaryFilesCheck()
+
+reporter.addDetails(
+'binary-extension-file-in-source',
+'''Files with *.class, *.dll, *.DS_Store, *.exe, *.jar, *.o, *.pyc,
*.pyo, *.so extensions are most probably binary files.''',
+'binary-file-with-30%-plus-non-text-chars',
+'''Binary file with 30% plus no text characters has been considered
as binary file''',
+'empty-file',
+'''Empty file'''
+)
14 years, 5 months
fixing structure to make src checks work
by Rakesh Pandit
These fixes I made while writing binary file checks over packages.
--
Rakesh Pandit
https://fedoraproject.org/
freedom, friends, features, first
>From c26b5a7ce9c12c6ed4f9785f5dfb8b7389b891e2 Mon Sep 17 00:00:00 2001
From: Rakesh Pandit <rakesh(a)fedoraproject.org>
Date: Tue, 17 Nov 2009 02:11:20 +0530
Subject: [PATCH] GachApplication, GachReviewer.py,
GachSrcRPMReviewer.py: Fixed for writing src
checks.
GachUtils.py: New file.
---
GachApplication.py | 41 +++++++++++++++++++-----------
GachReviewer.py | 18 +++++++++---
GachSrcRPMReviewer.py | 9 +++---
GachUtils.py | 66 +++++++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 110 insertions(+), 24 deletions(-)
create mode 100644 GachUtils.py
diff --git a/GachApplication.py b/GachApplication.py
index 5e6fcc3..fbde2d2 100644
--- a/GachApplication.py
+++ b/GachApplication.py
@@ -25,6 +25,9 @@ import tempfile
from Spec import Spec
from Pkg import Pkg
+from RPMCheck import RPMCheck
+from SrcRPMCheck import SrcRPMCheck
+
from GachBugzillaFeeder import GachBugzillaFeeder
from GachRPMReviewer import GachRPMReviewer
from GachSpecReviewer import GachSpecReviewer
@@ -32,16 +35,9 @@ from GachSrcRPMReviewer import GachSrcRPMReviewer
from GachBugzillaReporter import GachBugzillaReporter
from GachKojiBuilder import GachKojiBuilder
from GachReporter import GachReporter
-from GachReviewer import runRpmlint
-
-
-def rm_rf(d):
- for path in (os.path.join(d,f) for f in os.listdir(d)):
- if os.path.isdir(path):
- rm_rf(path)
- else:
- os.unlink(path)
- os.rmdir(d)
+from GachReviewer import runRpmlint, GachReviewer
+from GachSrcRPMReviewer import GachSrcRPMReviewer
+from GachUtils import rm_rf
class NoBugId(Exception):
@@ -107,24 +103,39 @@ class GachApplication:
print "Downloaded files are at: "
for file in file_path:
print file
-
+
reporter = GachReporter()
spec = Spec(file_path[0])
pkg = Pkg(file_path[1])
# TODO Test expecution and building via builders
- reporter.printDividerInfo("Spec file rpmlint output")
+ reporter.printDividerInfo("Spec file rpmlint output (one from bug) ")
runRpmlint(reporter, spec)
- reporter.printDividerInfo("Pkg rpmlint output")
+ reporter.printDividerInfo("Pkg rpmlint output (one from bug) ")
runRpmlint(reporter, pkg)
builder = GachKojiBuilder()
- builder.build(file_path[1], 'dist-f12', wait=True)
+
+ builder.build(file_path[1], 'dist-f13', wait=True)
downloaded_files = builder.retrieve(koji_downdir)
- print reporter.getReport()
+ reporter.printDividerInfo("Lint output on koji build rpms")
+
+ for file in downloaded_files:
+ if file.endswith(".rpm"):
+ reporter.printDividerInfo("Koji build rpm %s output" %(file))
+ runRpmlint(reporter, Pkg(file))
+
+ # Load all src rpm checks checks
+ reviewer = GachReviewer()
+ srpmreviewer = GachSrcRPMReviewer()
+
+ for file in downloaded_files:
+ if file.endswith("src.rpm"):
+ srpmreviewer.review(file)
+ print reporter.getReport()
rm_rf(srpm_spec_downdir)
return
diff --git a/GachReviewer.py b/GachReviewer.py
index f79ca81..cbc9451 100644
--- a/GachReviewer.py
+++ b/GachReviewer.py
@@ -47,24 +47,32 @@ class GachReviewer(object):
derived class.
"""
- def __init__(self):
+ def __init__(self, guidelines=None):
"""
"""
# Need to update it for each guideline.
- self.default_guidelines = ("SampleCheck")
+ self.default_guidelines = None
+ if guidelines:
+ self.default_guidelines = guidelines
+ else:
+ self.default_guidelines = ("SampleCheck",)
# Load all checks.
- self.__loadChecks()
+ self.loadChecks()
- def __loadChecks(self):
+ def loadChecks(self):
"""
Implements the loading module.
"""
+ print "Guidelines:", self.default_guidelines
for name in self.default_guidelines:
+ file = None
+ path = None
+ description = None
try:
# Later on we need to generalize the second parameter
# depending on where guideline files go after installation.
- (file, path, description) = imp.find_module(name, ".")
+ (file, path, description) = imp.find_module(name, ["."])
except ImportError:
print "Guideline import error: ", name
sys.exit(1)
diff --git a/GachSrcRPMReviewer.py b/GachSrcRPMReviewer.py
index 9e91ad1..3f00ae8 100644
--- a/GachSrcRPMReviewer.py
+++ b/GachSrcRPMReviewer.py
@@ -28,15 +28,16 @@ class GachSrcRPMReviewer(GachReviewer.GachReviewer):
def __init__(self):
"""
"""
- pass
+ # All Checks are loaded here.
+ self.default_guidelines = ("BinaryFilesCheck",)
+ GachReviewer.GachReviewer.__init__(self,
guidelines=self.default_guidelines)
def review(self, srpm):
"""
Implements reviewer for srpm.
"""
- for guide in SrcRPMCheck.SrcRPMCheck.checks:
- guide.check(srpm)
+ for guide in SrcRPMCheck.SrcRPMCheck.srpm_checks:
+ guide.check(srpm=srpm)
return
-
# GachSrcRPMReviewer ends here
diff --git a/GachUtils.py b/GachUtils.py
new file mode 100644
index 0000000..768b675
--- /dev/null
+++ b/GachUtils.py
@@ -0,0 +1,66 @@
+#!/usr/bin/env python
+#
+# Copyright 2009 Rakesh Pandit <rakesh(a)fedoraproject.org>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with translate; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+#
+# Misc functions go in here.
+#
+
+import os
+import sys
+import commands
+import tempfile
+
+
+def rm_rf(d):
+ """Remove directory recursively
+ """
+ for path in (os.path.join(d,f) for f in os.listdir(d)):
+ if os.path.isdir(path):
+ rm_rf(path)
+ else:
+ os.unlink(path)
+ os.rmdir(d)
+
+
+def extract_rpm(file):
+ """Extract rpm into output_dir
+ """
+ curdir = os.curdir
+ srpm_extract_dir = tempfile.mkdtemp(prefix="gach")
+ os.chdir(srpm_extract_dir)
+ status = commands.getstatusoutput("rpm2cpio " + file + "| cpio -idmv")
+ os.chdir(curdir)
+ return srpm_extract_dir
+
+
+def get_source_from_srpm(srpm_file, output_dir):
+ """build source from srpm file and return path to build directory
+ """
+ if commands.getstatusoutput("which rpmbuild")[0] != 0:
+ print "rpmbuild command not present!"
+ sys.exit(1)
+
+ name = commands.getoutput("rpm -qp --qf \"%{NAME}\" " + srpm_file)
+ if commands.getstatusoutput("rpm -i --define \"_topdir " +
output_dir + "\" " + srpm_file)[0] != 0:
+ print "Installation of srpm file failed."
+ sys.exit(1)
+
+ if commands.getstatusoutput("rpmbuild -bp --quiet --define
\"_topdir " + output_dir + "\" " + output_dir + "/SPECS/" + name +
".spec")[0] != 0:
+ print "rpmbuild failed to build source from srpm"
+ sys.exit(1)
+ return
--
1.6.2.5
14 years, 5 months