On Tue, 2010-02-09 at 08:06 +0000, Richard Hughes wrote:

On 8 February 2010 22:46, Kevin Kofler kevin.kofler@chello.at wrote:

...

As a result, you'll be causing dozens of FTBFS bugs just before the feature freeze. I think this is entirely the wrong time in the release cycle to do such a change, if it is done at all.

I've been fixing upstream projects for weeks to build with --no-as-needed. The list of projects that fail to build should be much smaller now, especially for GNOME and Freedesktop stuff.

Just as a reminder, this change is --no-add-needed, not --no-as-needed. They have infuriatingly similar names; one of the changes is also to change the name of --{no-,}add-needed to be more obvious.

--no-as-needed is already the default behaviour, and means "libraries specified with -lfoo will be emitted into the link output in a DT_NEEDED entry, regardless of whether any symbols from libfoo are used in the link output object itself". This may mean your binary or library ends up with more dependencies than it needs, but is generally harmless.

--no-add-needed is quite different. Your binary a.out uses symbols from libfoo and libbar. libfoo is linked against libbar. But your link line only says -lfoo. --add-needed behaviour, the old default, would implicitly add a "-lbar" as well. --no-add-needed, the new default, will not, and therefore your link will probably fail.

- ajax

On Thu, 2010-02-11 at 11:06 +0100, yersinia wrote:

On Tue, Feb 9, 2010 at 3:38 PM, Adam Jackson ajax@redhat.com wrote:

    --no-add-needed is quite different.  Your binary a.out uses
    symbols from
    libfoo and libbar.  libfoo is linked against libbar.  But your
    link line
    only says -lfoo.  --add-needed behaviour, the old default,
    would
    implicitly add a "-lbar" as well.  --no-add-needed, the new
    default,
    will not, and therefore your link will probably fail.

This is the case described in the libtool manual

http://www.gnu.org/software/libtool/manual/html_node/Inter_002dlibrary-depen...

Nice to know that Fedora will have the same behavior of the AIX libraries in this area, if I understood correctly the topic under discussion.

This change does not imply AIX linker semantics.

Note that _libraries_ generally do not have a problem building in a --no-add-needed world. ELF does not require that all references in a DSO be resolvable at ld time, and this linking change does not change that. If your library libfoo uses symbols from libbar but does not itself link against libbar, that's still legal (although probably impolite).

Executables, however, must be fully resolved at link time. You can ask for this to be true for libraries as well with the --no-allow-shlib-undefined option to ld. This plus --no-add-needed is much closer to the AIX linker behaviour,

Also note that the runtime linker will still do recursive lookups. If you have a binary that did not link against some needed library, but one of its dependencies did link against it, the binary will still work.

- ajax

Once upon a time, Kevin Kofler kevin.kofler@chello.at said:

And this makes this ld (mis)feature particularly silly, ld now gratuitously errors on "undefined" symbols which would be found just fine at runtime.

No, it errors on undefined symbols that may or may not be found at runtime. Why do you want binaries that may or may not work, depending on the configuration of shared libraries from other projects?

For example, let's say you have a program foo that uses routines from libm but doesn't link with -lm. Program foo does link with an image processing library libbar that uses libm internally. Now libbar releases an update that drops the dependency on libm (maybe somebody found a more efficient way to do the processing with integer math). Since libbar still exports the same ABI (-lm vs. integer math is an internal change), they don't change the so version.

The libbar update goes into Fedora, and suddenly program foo breaks (when it hits certain code paths), for no obvious reason (maybe program foo hasn't been updated in months).

You could even end up with an update respin where the necessary shared library isn't even installed because neither foo nor libbar depend on it (not the case for libm, but could be for other dependencies).

I like deterministic linking, not linking that says "well, maybe it'll work, we'll let the user tell us if it breaks later".

Note that _libraries_ generally do not have a problem building in a --no-add-needed world. ELF does not require that all references in a DSO be resolvable at ld time, and this linking change does not change that. If your library libfoo uses symbols from libbar but does not itself link against libbar, that's still legal (although probably impolite).

It is ill-advised. It's recommended to use -shared -Wl,-z,defs so that you will get a link-time failure for being sloppy in this way. (You can't do this if the DSO intentionally has free undefined symbols as part of its ABI, but that is not a style we would recommend for any new libraries.) The reason this really matters is that you want to get symbol version bindings for the references from your DSO to another DSO. It's also the most reliable way (the implicit way) to make sure you have rpm dependencies for the libraries you require.

Thanks, Roland

Richard Hughes wrote:

I've been fixing upstream projects for weeks to build with --no-[add]-needed. The list of projects that fail to build should be much smaller now, especially for GNOME and Freedesktop stuff.

1. But have those fixes been applied in the Fedora packages? At least NetworkManager is now failing to build due to this "feature" having been rushed in just before the freeze, so that's at least one package which didn't get fixed in Fedora yet. 2. GNOME and freedesktop.org form only a small part of the Fedora package collection.

I really don't see why this change can't at least wait for F14! Breaking the build of half of the distro the day of the feature freeze is completely unacceptable. The change should get reverted for F13 and put into Rawhide after F13 gets branched.

(I still oppose the feature altogether, but postponing it to F14 would at least give packagers time to fix their packages.)

Kevin Kofler

Hi,

On Tue, Feb 9, 2010 at 9:08 PM, Kevin Kofler kevin.kofler@chello.at wrote:

Richard Hughes wrote:

...

I've been fixing upstream projects for weeks to build with --no-[add]-needed. The list of projects that fail to build should be much smaller now, especially for GNOME and Freedesktop stuff.

1. But have those fixes been applied in the Fedora packages? At least

NetworkManager is now failing to build due to this "feature" having been rushed in just before the freeze, so that's at least one package which didn't get fixed in Fedora yet. 2. GNOME and freedesktop.org form only a small part of the Fedora package collection.

I really don't see why this change can't at least wait for F14! Breaking the build of half of the distro the day of the feature freeze is completely unacceptable. The change should get reverted for F13 and put into Rawhide after F13 gets branched.

(I still oppose the feature altogether, but postponing it to F14 would at least give packagers time to fix their packages.)

+1. Please revert the changes.

Parag.

I wrote:

I've filed this proposal for FESCo, it should get considered at the meeting tonight (20:00 UTC): https://fedorahosted.org/fesco/ticket/338 "Proposal: postpone ChangeInImplicitDSOLinking feature to F14 and revert its implementation from pre-branch Rawhide"

Sadly, this proposal got rejected. :-( (My own vote was the only one in favor of my proposal.)

So any complaints about the breakage should be directed at FESCo.

Kevin Kofler

On Tue, 2010-02-16 at 15:57 +0100, Gerd Hoffmann wrote:

On 02/09/10 09:06, Richard Hughes wrote:

...

On 8 February 2010 22:46, Kevin Koflerkevin.kofler@chello.at wrote:

...

As a result, you'll be causing dozens of FTBFS bugs just before the feature freeze. I think this is entirely the wrong time in the release cycle to do such a change, if it is done at all.

I've been fixing upstream projects for weeks to build with --no-as-needed. The list of projects that fail to build should be much smaller now, especially for GNOME and Freedesktop stuff.

Well. Even pretty fundamental GNOME stuff like gtk2-devel is still broken. Look here:

[root@localhost ~]# pkg-config --libs gtk+-2.0 -pthread -lgtk-x11-2.0 -lgdk-x11-2.0 -latk-1.0 -lgio-2.0 -lpangoft2-1.0 -lgdk_pixbuf-2.0 -lpangocairo-1.0 -lcairo -lpango-1.0 -lfreetype -lfontconfig -lgobject-2.0 -lgmodule-2.0 -lgthread-2.0 -lrt -lglib-2.0

-lX11 is missing (at least, maybe more). Any gtk app using pkg-config to figure which libraries are required fails to build now.

I've seen a simliar issue with libpng (-lz missing) and I suspect tons of other *.pc files are broken too.

No. Why should we put libX11 in the .pc file ? If you do any direct Xlib calls in your program, you get to put it in the link line yourself.

On 02/16/10 16:06, Jakub Jelinek wrote:

On Tue, Feb 16, 2010 at 03:57:52PM +0100, Gerd Hoffmann wrote:

...

Well. Even pretty fundamental GNOME stuff like gtk2-devel is still broken. Look here:

[root@localhost ~]# pkg-config --libs gtk+-2.0 -pthread -lgtk-x11-2.0 -lgdk-x11-2.0 -latk-1.0 -lgio-2.0 -lpangoft2-1.0 -lgdk_pixbuf-2.0 -lpangocairo-1.0 -lcairo -lpango-1.0 -lfreetype -lfontconfig -lgobject-2.0 -lgmodule-2.0 -lgthread-2.0 -lrt -lglib-2.0

-lX11 is missing (at least, maybe more). Any gtk app using pkg-config to figure which libraries are required fails to build now.

Only if the library you are compiling actually uses any libX11 APIs. If it only uses APIs from the named libraries, it will link fine.

Sure? For the libpng case I have a build failure which looks like this isn't the case (bug 565047):

<quote> /usr/bin/ld: /usr/lib/gcc/i686-redhat-linux/4.4.3/../../../libpng12.so: undefined reference to symbol 'crc32' /usr/bin/ld: note: 'crc32' is defined in DSO /lib/libz.so.1 so try adding it to the linker command line </quote>

Which made me think -lz is needed on the command line even if zlib is used only indirectly via libpng (and likewise that -lX11 should be there for gtk2 because gtk2 needs it).

cheers, Gerd

Gerd Hoffmann kraxel@redhat.com writes:

Well. Even pretty fundamental GNOME stuff like gtk2-devel is still broken. Look here:

[root@localhost ~]# pkg-config --libs gtk+-2.0 -pthread -lgtk-x11-2.0 -lgdk-x11-2.0 -latk-1.0 -lgio-2.0 -lpangoft2-1.0 -lgdk_pixbuf-2.0 -lpangocairo-1.0 -lcairo -lpango-1.0 -lfreetype -lfontconfig -lgobject-2.0 -lgmodule-2.0 -lgthread-2.0 -lrt -lglib-2.0

Are all these libraries really required? Putting them into a linker line causes a huge overlinking adding lots of unneeded direct dependencies to rpm packages.

-lX11 is missing (at least, maybe more).

That's the problem with the current DSO change :( It causes packagers or upstream developers to define typical use cases (e.g. when somebody calls 'mylib_foo(otherlib_struct *)' he calls usually 'otherlib_init(otherlib_struct *)', so that '-lotherlib' is usually needed).

Such soft decisions are far away from being reliable and to work for everybody...

I ended up for 'xmlrpc-c' to replace .so symlinks by linker scripts with

| INPUT(<main-lib> AS_NEEDED(<dep-libs>+))

commands. The <dep-libs> list is created by extracting recursively the NEEDED libs from <main-lib>.

Not very nice but probably to only way to cope with '--no-add-needed' :(

Enrico

On Tue, 2010-02-16 at 17:03 +0100, Enrico Scholz wrote:

Gerd Hoffmann kraxel@redhat.com writes:

...

Well. Even pretty fundamental GNOME stuff like gtk2-devel is still broken. Look here:

[root@localhost ~]# pkg-config --libs gtk+-2.0 -pthread -lgtk-x11-2.0 -lgdk-x11-2.0 -latk-1.0 -lgio-2.0 -lpangoft2-1.0 -lgdk_pixbuf-2.0 -lpangocairo-1.0 -lcairo -lpango-1.0 -lfreetype -lfontconfig -lgobject-2.0 -lgmodule-2.0 -lgthread-2.0 -lrt -lglib-2.0

Are all these libraries really required? Putting them into a linker line causes a huge overlinking adding lots of unneeded direct dependencies to rpm packages.

gtk+-2.0.pc has:

Requires: gdk-${target}-2.0 atk cairo gio-2.0 pangoft2

It seems likely that some, if not all, of the latter four belong in Requires.private.

Milos Jakubicek wrote:

On 8.2.2010 23:37, Roland Grunberg wrote:

...

https://fedoraproject.org/wiki/DSOLinkBugs

281 packages? Wov! (That's after the most often occurring problems have been already resolved, am I right?)

• let's say 300 "regular" FTBFS bugs -- the F13 mass rebuild will be

really great...

To me, this shows that this "feature" is completely broken and should not make it into Fedora, ever.

You sure it wouldn't be better to wait for F14 with this? With "no frozen rawhide" you can submit the changes into rawhide quite soon and everybody has time enough to work on it.

+1. Why does this get rushed into F13 the day of the feature freeze?

Kevin Kofler

Adam Williamson wrote:

I disagree with that. Previous changes to the build environment - even upstream GCC changes - have broken way more packages (every GCC .x release tends to break a lot of things temporarily).

And that's something which really irks me about GCC upstream, they don't seem to understand what "backwards compatibility" means. That doesn't mean it's a good idea to break code like this.

Kevin Kofler

Jakub Jelinek wrote:

You are probably looking for bug compatibility, and that isn't something GCC guarantees, definitely not between major versions.

And that's one half of what I'm complaining about.

The C/C++ standards (and standards governing the extensions to the languages) is what matters, if you write standard conforming code, GCC won't (intentionally) start rejecting it. But if you have code that happens to compile because of some GCC bug, eventhough it was invalid, or some C/C++ defect report clarifies some corner case, you need to be prepared to fix up your code.

What about those documented extensions that got deprecated and later removed? That's the second half of what I'm complaining about: even things which are NOT bugs but documented extensions get deprecated and soon later removed.

IMHO a compiler should accept code whenever there's a sane interpretation of it, no matter whether it conforms to some standard or not (in fact, this used to be a GCC design principle, but sadly no longer is these days), and code which has been compiling for years definitely has a sane interpretation.

(If you ever look at TIGCC, you'll also notice that TIGCC's GCC is patched to re-add things like multi-line string literals, lvalue casts, labels at the end of compound statements etc. More and more patches I'm applying to the GCC in TIGCC are to readd things GCC removed. TIGCC also enables -fms- extensions by default (in C, this allows at least unnamed struct/union fields). I haven't been updating TIGCC's GCC in a while, but I'm sure that if I do, there'll be another bunch of such patches to add. And that's only for C, g++ is much worse.)

Kevin Kofler

Björn Persson wrote:

And what happens the day you need to compile that code with another compiler?

What compiler? A proprietary compiler? The BSD-style-licensed LLVM/clang which any proprietary vendor can embrace&extend into a non-Free version? It's not in the interest of the GNU project (which GCC is supposed to be a part of) to make it easy to compile code with other compilers!

Do you consider vendor lock-in through embrace-and-extend tactics to be a good thing when a free software project does it?

Yes, see above.

Kevin Kofler

Ryan Rix wrote:

On Wed 10 February 2010 7:00:42 pm Kevin Kofler wrote:

...

It's not in the interest of the GNU project (which GCC is supposed to be a part of) to make it easy to compile code with other compilers!

If that is the case it is extremely short sighted of them.

That was a statement of fact, not of policy.

The GCC project's policies used to be based on that when it was controlled more directly by the FSF, but since the EGCS merge, the "follow standards" camp has basically taken over, which is IMHO quite counterproductive.

A compiler should accept as much code as possible. As long as it UNDERSTANDS standards-compliant code (real-world standards-compliant code, not some obscure corner-case designed especially to detect the absence of some extension which nobody would actually write in a real program), there's no need to ENFORCE standards compliance. All such enforcement does is: * make it easy to migrate to a proprietary compiler, * make it harder to migrate away from a proprietary compiler, as some of them do accept things like lvalue casts (M$VC supports them, for example), * gratuitously break backwards compatibility, * make real programmers (who get errors for obscure standardese reasons they don't understand when it should be clear what their code is intended to mean) angry just to make pedantic language lawyers happy.

As a general rule, it's in a compiler's best interest to accept as much code as possible so it's easy to migrate TO that compiler and hard to migrate AWAY FROM that compiler. And what's good for GCC is good for Free Software.

Kevin Kofler

Worth clarifying here: Rawhide or (and?) F13?

They are still the same thing, so both. gcc-4.4.3-5.fc13 is there right now.

Thanks, Roland

They will only be so for a fairly short time, and you gave no specific time frame for landing the change (only 'pretty soon'), so it was not entirely clear. Thanks.

Sorry, we meant to be clear that it was "now" as of the posting.

2010/2/9 Parag N(पराग़) panemade@gmail.com:

when one of my package fails to build? Should I ask upstream to hardcode required DSO names in Makefile or we need to modify CFLAGS in %build section?

Most of the time upstream (myself included) just forgets to add a library at the end of the LDADD line, so all I had to do was send a trivial patch upstream to add "-lm" or something.

See here for an example: http://bugzilla-attachments.gnome.org/attachment.cgi?id=152993

Richard.

On Tue, Feb 09, 2010 at 06:55:11AM -0500, Neal Becker wrote:

Will there we a switch to give me the old behavior? I might want this for my own legacy code.

I have not tried it, but apparently --as-needed (or from gcc use -Wl,--as-needed):

--as-needed --no-as-needed This option affects ELF DT_NEEDED tags for dynamic libraries mentioned on the command line after the --as-needed option. Normally the linker will add a DT_NEEDED tag for each dynamic library mentioned on the command line, regardless of whether the library is actually needed or not. --as-needed causes a DT_NEEDED tag to only be emitted for a library that satisfies an undefined symbol reference from a regular object file or, if the library is not found in the DT_NEEDED lists of other libraries linked up to that point, an undefined symbol reference from another dynamic library. --no-as-needed restores the default behaviour.

Rich.

On Tue, Feb 09, 2010 at 09:38:05AM -0500, Adam Jackson wrote:

No, you mean --add-needed.

Fair enough. BTW the man page seems to indicate that --add-needed is deprecated (replaced by --copy-dt-needed-entries).

Rich.

On Wed, Feb 17, 2010 at 03:04:00AM -0800, Roland McGrath wrote:

...

Is it ok to backport changes to F-12 for fixed packages in F-13 for this DSO feature?

The changes to a package's linking lines that you make for this issue are cleaning up sloppy practice to what was always the right thing to be doing. So it should always be fine to do that in builds for earlier Fedoras too.

It is much more important to find something well integrated acceptable upstream, in my opinion.

-- Pat

Hi,

On Thu, Feb 18, 2010 at 4:26 AM, Kevin Kofler kevin.kofler@chello.at wrote:

Parag N(पराग़) wrote:

...

Is it ok to backport changes to F-12 for fixed packages in F-13 for this DSO feature?

It's of course OK to apply those changes to all branches (they won't break anything for the older ld), but it doesn't make sense to push updates just for those changes! Please only push updates if you have actual user-visible changes.

Thanks all. I just want to know if its possible to apply DSO change in older active branches. No plans to update those packages in older branches but to get these changes in upstream first.

Parag.