IU> This URL format is not recommended by CRAN, but more importantly, IU> the Source0 format does not work anymore, as [1] noted, when a new IU> version is released. However, there is an immutable format IU> available, as [2] pointed out. So my proposal is to use always the IU> following lines instead: IU> URL: https://cran.r-project.org/package=%{packname} IU> Source0: %{url}&version=%{version} That's good to know. We should consider an %r_source macro similar to the existing %pypi_source macro which is used for python packages. See: https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_source... I would be happy to write it, but I don't enough about R packaging to know how uniformly the tags you mention above (%packname, specifically) are adhered to in the current packaging set. IU> which are both shorter and immutable, and I propose to add this to IU> the R packaging guidelines too. Well you should certainly open a ticket with the packaging committee if you wish to propose changes to the packaging guidelines. https://pagure.io/packaging-committee/ You can even send a pull request. IU> If we agree on this, is there any easy way to request a system-wide IU> change like that to all existing packages? https://fedoraproject.org/wiki/Mass_package_changes - J<

Kind reminder. Any update on this? Iñaki On Wed, 24 Oct 2018 at 17:01, Iñaki Ucar <iucar(a)fedoraproject.org&gt; wrote: -- Iñaki Ucar

Hi Dominik, On Wed, 31 Oct 2018 at 00:43, Dominik 'Rathann' Mierzejewski <dominik(a)greysector.net&gt; wrote: I thought that the existence of the new macro was a precondition to open an issue or a PR against the guidelines recommending it. That's why I wrote the following: On Wed, 24 Oct 2018 at 17:01, Iñaki Ucar <iucar(a)fedoraproject.org&gt; wrote: But if you confirm that this is not the case, I certainly misunderstood Jason's email, so I'll do it right away. -- Iñaki Ucar

Spent a couple of seconds looking at this and have a couple of questions. Once I have answers it should take me only a couple of minutes to give you something to test. Does the source URL for CRAN not have any kind of file extension? Could you provide a couple of sample packages for me to look at and use in testing? I don't know much of anything about R. - J<

On Wed, 31 Oct 2018 at 18:25, Jason L Tibbitts III <tibbs(a)math.uh.edu&gt; wrote: Correct. But that URL, for instance: https://cran.r-project.org/package=simmer&version=3.0.0 returns a redirection (303) to the complete URL, with file extension. CRAN maintainers are pretty strict with this kind of stuff: if it works now, it's guaranteed to continue to work. There are no other formats: every package is tar.gz. But, as I pointed out above, the immutable URL is a redirection to the complete URL, so you can still extract the extension. -- Iñaki Ucar

IU> https://cran.r-project.org/package=simmer&version=3.0.0 IU> returns a redirection (303) to the complete URL, with file IU> extension. 303 is actually "See Other". Which is odd as that's usually sent in response to a PUT or POST, not a GET. Maybe you can get the files via POST as well; I'm not sure. In any case, none of that has any effect on the filename that spectool (really curl) will use. It can't use data supplied by the remote host for that, for obvious reasons. IU> CRAN maintainers are pretty strict with this kind of stuff: if it IU> works now, it's guaranteed to continue to work. Well, that's good, but this is a hack so it's not a terrible idea to inform them that we have to use this kind of thing so that they can either bless this method or provide a cleaner one. IU> There are no other formats: every package is tar.gz. But, as I IU> pointed out above, the immutable URL is a redirection to the IU> complete URL, so you can still extract the extension. No, you can't. Not in the context and under the limitations where we're running. Certainly we can't know anything about that in an RPM macro as we have to provide an extension in a complete vacuum. But given what you say, certainly defaulting to tar.gz will work for everything now. Try dropping the below into /usr/lib/rpm/macros.d/macros.test (temporarily, make sure to deleted it when you're done experimenting). See if it gives the results you expect when you use %cran_url and %cran_source. Do fedpkg prep or some local builds. Try spectool -g and rpmspec -P. Note that I've snuck a bit of magic in there which I'm not sure should be kept: If you don't have %packname defined and you call either %cran_url or %cran_source, then it will be automatically defined for you by stripping the leading "R-" from the name. With this, you can just have: Name: R-webp Version: 0.4 Release: 3%{?dist} Summary: A New Format for Lossless and Lossy Image Compression License: MIT URL: %cran_url Source0: %cran_source And use %packname in %prep as usual without explicitly defining it. But I'm not sure that much magic is a good idea. (And while we're doing R macros, that package suggests that we should also have a macro defined to %_libdir/R/library....) - J< # Macros to replace overly complicated references to CRAN URLs and source files. # %cran_source - # Expands to the CRAN URL for a package # Accepts zero to three arguments: # 1: The CRAN project name, defaulting to %packname if it is defined. # If not, R- will be stripped from %name and %packname defined to that. # 2: The CRAN version, defaulting to %version. # 3: The file extension, defaulting to %__cran_default_extension (tar.gz). # Requires %__cran_package_url_template and %__cran_default_extension to be defined. # %__cran_package_url_template will undergo substitution (case-sensitive): # * "PACKNAME" will be replaced with the above CRAN project name. # * "PACKVERSION" will be replaced with the above CRAN version. # * "EXTENSION" will be replaced with the above extension. # # %cran_url - # Expands to the CRAN URL for a package # Accepts zero or one arguments: # 1: The CRAN project name, defaulting to %packname if it is defined. # If not, R- will be stripped from %name and %packname defined to that. # Requires %__cran_project_url_template to be defined. # %__cran_project_url_template will undergo substitution (case-sensitive): # * "PACKNAME" will be replaced with the above CRAN project name. %__cran_project_url_template https://cran.r-project.org/package=PACKNAME %__cran_package_url_template %{__cran_project_url_template}&version=PACKVERSION#/PACKNAME_PACKVERSION.EXTENSION %__cran_default_extension tar.gz %cran_source() %{lua: local src = rpm.expand('%1') local ver = rpm.expand('%2') local ext = rpm.expand('%3') local url = rpm.expand('%__cran_package_url_template') \ -- If no first argument, try %packname, then %name with 'R-' stripped. -- Note that rpm leaves macros unchanged if they are not defined. if src == '%1' then src = rpm.expand('%packname') end if src == '%packname' then src = string.gsub(rpm.expand('%name'), "^R%-", "") -- Since packname wasn't defined, define it for convenience. rpm.define("packname " .. src) end \ -- If no second argument, use %version if ver == '%2' then ver = rpm.expand('%version') end \ -- If no third argument, use the preset default extension if ext == '%3' then ext = rpm.expand('%__cran_default_extension') end \ -- Now substitute in all the values url = string.gsub(url, "PACKNAME", src) url = string.gsub(url, "PACKVERSION", ver) url = string.gsub(url, "EXTENSION", ext) \ print(url) } %cran_url() %{lua: local src = rpm.expand('%1') local url = rpm.expand('%__cran_project_url_template') \ -- If no first argument, try %packname, then %name with 'R-' stripped. -- Note that rpm leaves macros unchanged if they are not defined. if src == '%1' then src = rpm.expand('%packname') end if src == '%packname' then src = string.gsub(rpm.expand('%name'), "^R%-", "") -- Since packname wasn't defined, define it for convenience. rpm.define("packname " .. src) end \ -- Substitute in the URL value url = string.gsub(url, "PACKNAME", src) \ print(url) }

On Wed, 31 Oct 2018 at 21:22, Iñaki Ucar <iucar(a)fedoraproject.org&gt; wrote: Well... the guidelines for Python state: 'Note that when a module that has a dot in its name, the usual rule about changing "." to "-" applies.' But the guidelines for R say nothing about that, and as a result, most packages with a dot don't change it. Maybe R-TH-data is the only one that is compliant after all? :) -- Iñaki Ucar

IU> This is great. However, in theory, given the naming guidelines, by IU> stripping the leading "R-" you should get the package name. In IU> practice, at least one package doesn't adhere to this: R-TH-data, IU> while the R package name is TH.data, not TH-data. I see that the SPEC IU> says "# Cannot use . in name", but this is clearly not true (maybe it IU> was true long ago?). Why is that a problem? You would just define %packname in that case and nothing changes. Look for the 50% case. Does it simplify at least half of the packages while not making things harder for the rest? I don't know the answer but I would be surprised if it wasn't 'yes' even if you change 50% to 90%. IU> That would require a good ton of magic. You have seen something like IU> this: IU> %global rlibdir %{_libdir}/R/library IU> The thing is, this is the path for R packages *with* compiled code, IU> while R packages *without* compiled code must go to IU> %_datadir/R/library. That's why every R package has this global on IU> top of the SPEC. Well, what you'd generally do is simply use a different macro for the noarch location and the arch-specific location. So you'd one defined macro for things under libdir, another macro for things under datadir. Like Perl and Python and such do. If you really wanted to get down into it, it's tough to magically define a macro depending on BuildArch (though I could be missing a trick) but ou could conceivably have macros like %r_archful_package and %r_noarch_package. They could some macro like your %rlibdir and also take care of adding the build dependencies and (where needed) the BuildArch: line and the R-core dependency. There's really a whole lot you could do, down to having macros used in %install, adding that annoying empty %build section, even generating a file list so you don't have to manually list so much in %files. It depends on how far you want to go, and how specific you can be before you're not actually simplifying a majority of the R packages we have. - J<

On Wed, Oct 31, 2018, 5:01 PM Iñaki Ucar <iucar(a)fedoraproject.org wrote: All my packages were created using R2spec, so they would all use %{packname}. I would guess that most packages used the generator as well. This can be determined from the NeedsCompilation key in the DESCRIPTION file, which is what (my fork of) R2spec does. Basically anything under %{rlibdir}%{packname} is needed and nothing appears elsewhere. If we didn't need to mark doc and license files, we could have just specified that top-level directory. The hard part about that is the naming is pretty inconsistent. Because CRAN disallows providing the full license text in LICENSE, upstreams that want to follow the terms of the license will put it in a separate file and the name is never the same. In the same way Go has moved to mostly generated, I think R can mostly get away with this as well. We can get most of the information from the DESCRIPTION file or re-write R2spec to output what we need. I don't really know how macros work or I might have attempted this earlier.

JLT> It depends on how far you want to go, and how specific you can be JLT> before you're not actually simplifying a majority of the R packages JLT> we have. Just for fun I took the R-webp package and constructed a macro that generates most of the package. Obviously it's somewhat bound to this specific package, but it appears that most R packages have rather similar boilerplate so it might be more generally useful. It's just a hack but it seems like it might be interesting to explore this a bit. The technique should still apply to package sets which have significant uniformity. The spec looks like: ----- Name: R-webp Version: 0.4 Release: 3%{?dist} Summary: A New Format for Lossless and Lossy Image Compression License: MIT URL: %cran_url Source0: %cran_source Suggests: R-jpeg R-png BuildRequires: libwebp-devel BuildRequires: R-jpeg R-png %description Lossless webp images are 26% smaller in size compared to PNG. Lossy webp images are 25-34% smaller in size compared to JPEG. This package reads and writes webp images into a 3 (rgb) or 4 (rgba) channel bitmap array using conventions from the 'jpeg' and 'png' packages. %r_simple_archful_package %files -f %packname.files %changelog ----- This builds fine and produces the same output as the current spec in git. I could go even further and not have the %files section at all, really. I don't really know how useful this would be. I know (from another package I glanced at) that applying patches seems to be a bit weird because of the directory structure, so maybe using %autosetup isn't the right way to go. But I could still easily duplicate the patch application part of %autosetup. And the auto file-list generation stuff could easily grow smarts to handle other cases. The macros, which you can drop temporarily in /usr/lib/rpm/macros.d for testing, follow. These aren't "complete" and you still need the other macros I posted earlier. You can cram them together in one file if you like. - J< # The basic sections %r_prep %{lua:\ local packname = rpm.expand('%packname') local autosetup = rpm.expand("%{autosetup -c -n " .. packname .. "}") \ print(autosetup .. "\\\n") } %r_install %{lua:\ local rlibdir = rpm.expand('%rlibdir') local bindir = rpm.expand('%_bindir') local packname = rpm.expand('%packname') local buildroot = rpm.expand('%buildroot') \ -- A function to simplify adding to the file list local function add_file(file, type) print("echo '") if (type ~= nil) then print("%" .. type .. " ") end print(rlibdir .. "/" .. packname) if (file ~= nil) then print("/" .. file) end print("' >> " .. packname .. ".files\\\n") end \ print("mkdir -p " .. buildroot .. rlibdir .. "\\\n") print(bindir .. "/R CMD INSTALL -l " .. buildroot .. rlibdir .. " " .. packname .. "\\\n") print("test -d " .. packname .. "/src && (cd " .. packname .. "/src; rm -f *.o *.so)\\\n") print("rm -f " .. buildroot .. rlibdir .. "/R.css\\\n") \ -- R packages have a somewhat regularlized file structure -- This could be pushed out to a shell script and called like %find_lang is -- called. But for now we'll just inline the code. print("\\\n# Generate a default file list\\\n") add_file(nil, "dir") add_file("html/", "doc") add_file("libs", "dir") add_file("libs/" .. packname .. ".so") add_file("DESCRIPTION") add_file("LICENSE", "license") add_file("NEWS", "doc") add_file("INDEX") add_file("NAMESPACE") add_file("Meta/") add_file("R/") add_file("help/") \ print("ls -lR " .. buildroot .. "\\\n") } %r_check %{lua:\ local bindir = rpm.expand('%_bindir') local packname = rpm.expand('%packname') \ print(bindir .. "/R CMD check " .. packname .. "\\\n") } # For noarch packages %r_noarch_package %{lua:\ print("BuildArch: noarch\\\n") print("Requires: R-core\\\n") print("BuildRequires: R-devel tex(latex)\\\n") rpm.define("rlibdir %{_datadir}/R/library") print("%build\\\n\\\n") } %r_archful_package %{lua:\ print("BuildRequires: R-devel tex(latex)\\\n") rpm.define("rlibdir %{_libdir}/R/library") print("%build\\\n\\\n") } %r_simple_archful_package %{lua:\ print(rpm.expand("%r_archful_package")) \ local prep = rpm.expand("%r_prep") print("%prep\\\n") print(prep .. "\\\n\\\n") \ -- Hack around bizarre redefinition of %install. local pre_install = rpm.expand('%{?_enable_debug_packages:%{debug_package}}') print(pre_install .. "\\\n") \ local install = rpm.expand('%r_install') print("%install\\\n") print(install .. "\\\n\\\n") \ local check = rpm.expand('%r_check') print("%check\\\n") print(check .. "\\\n\\\n") \ -- We could even add the files section here, but this needs magic to -- determine packname since it hasn't yet been defined. -- print("%files -f " .. packname .. ".files\\n") }

All this would be very useful. CRAN guidelines are pretty strict, and they are enforced by R packaging and checking tools, and additionally upon submission to CRAN, with many automatic checks. As a result, and as you say, most SPEC sections in most packages can be fully automated, and I think we should tend to that, with proper mechanisms to treat special cases. Let me add to your findings from my experience as both R and Fedora package maintainer: - Version: As you already found out, there are many version formats allowed. The only special cases are those using a hyphen, such as R-uuid, that needs to be substituted with a dot. Other formats may contain two or three components, year and month... but they don't require special treatment if there are no hyphens. - Arch, Requires, BuildRequires, libdir/datadir: As Elliott pointed out already, archful packages can be detected just by looking into the DESCRIPTION file, by checking whether the NeedsCompilation flag exists. If it does, the package must go to libdir and R-devel must be added as BuildRequires. Furthermore, the LinkingTo field in the DESCRIPTION file lists packages that must be added as BuildRequires, such as R-Rcpp-devel. If NeedsCompilation does not exist, the package must go to datadir, noarch needs to be specified and R-core must be added as Requires. Further Requires usually are a little bit trickier to automate and should be managed by us. - Package devel: This could be automated also. LinkingTo other R packages is possible if headers are listed under inst/include (in the sources; the installation path is %{rlibdir}/%{packname}/include) in those packages. So the R-%packname-devel package must be defined if such directory exists. There are two flavors for this: packages that NeedsCompilation, so they are archful, and packages that don't. The latter are devel-only (see, e.g., R-BH). - %prep, %build, %install, %check, %files: These can be automated too. %build (void) and %install are always the same. %prep and %check may require some tuning (patching, unbundling; defining some exports and setting some flags in the "R CMD check" command). %files are pretty standard: there is a basic set, and then other may or may not be there, such as LICENSE/LICENCE, NEWS/NEWS.md, README/README.md, Changelog, and CITATION; and directories such as libs and include. I'm not familiar at all with RPM macros, but I can help with all these details. On Thu, 1 Nov 2018 at 03:22, Jason L Tibbitts III <tibbs(a)math.uh.edu&gt; wrote: -- Iñaki Ucar

IU> - Version: As you already found out, there are many version formats IU> allowed. But it's easy to work from Version: in the normal case and simply allow the CRAN version to be specified separately, as I showed in the R-uuid sample. If it's rather more common that updating the version in two places is a burden, we can certainly look at other ways to do it. IU> - Arch, Requires, BuildRequires, libdir/datadir: As Elliott pointed IU> out already, archful packages can be detected just by looking into IU> the DESCRIPTION file, by checking whether the NeedsCompilation flag IU> exists. As nim pointed out in a previous message, it's "not possible" to make use of this information for many things as long as DESCRIPTION is within the tarball, because unpacking does not happen until well after specfile parsing (and thus macro expansion) has happened. (Technically all things are possible but there isn't too much you can do with a source file before it's even been downloaded, and we certainly have to be able to parse the spec before that point.) There are of course some things you can do. The %prep and %install scriptlets, for example, are just shell code which can be inserted by macros but which actually runs much later. So you can use that information to figure out how to build, and what files to install. But you can't look in there to add BuildArch: noarch or add build dependencies because those have to be done much earlier. So that's why I just provided separate macros for the archful and noarch cases. IU> - %prep, %build, %install, %check, %files: These can be automated IU> too. Well, look at the two examples I provided. The %files list isn't completely consistent (one package has COPYING and the other doesn't) but the test macro file I provided conditionalizes this properly. IU> I'm not familiar at all with RPM macros, but I can help with all IU> these details. So then perhaps have a look at those examples I provided and see how they would actually work in real life on more than the two specs I converted. If I get time today I'll fill things in for the noarch package case. - J<

IU> I see. Anyway, I suppose that it's healthy to preserve some manual IU> intervention in these sections. Well, it would be super great if we didn't have to do that and one day RPM might give us some reasonable way to generate more of the specfile based on the contents of the source files. See https://github.com/rpm-software-management/rpm/issues/104 But even if we get that, you won't be able to use it on anything but rawhide for some time. IU> You can count on that. I'll try to identify corner cases to see how IU> your macros behave in such packages and report back as time permits. I'm certainly happy to develop them in tandem with your investigations. I have no expectation that what I banged out yesterday is remotely complete, but it at least shows that for at least a couple of packages you can hide rather a lot of the details behind macros. I think the next step is for me to stuff those macros in a temporary pagure repo. We can track what's missing with tickets and then eventually get the macros packaged up so that they can be used in the buildsystem. - J<

Hello, On Thu, 1 Nov 2018 at 20:45, Jason L Tibbitts III <tibbs(a)math.uh.edu&gt; wrote: This is a rather old email thread that I'm resurrecting, but I'm wondering what the state of those macros is now? We now have a place in upstream rpm where we could store these macros: https://github.com/rpm-software-management/R-rpm-macros It is currently used for automated dependency generators only. -- Elliott